Deep Double Descent: Where Bigger Models and More Data Hurt

Test error peaks when a model can only just fit its training set, so near that point a bigger model or more data can make it worse.

As a network grows, its test error first falls, then rises to a peak at the size where training error first reaches zero, then falls again as the network keeps growing. The same peak appears when one large network is trained for more epochs, and when a fixed network is given more training data.

A 2019 experimental paper from Harvard and OpenAI: ResNets, CNNs and Transformers trained at dozens of widths, label-noise levels, training lengths and dataset sizes, and one definition that predicts where their test error peaks.

The paper introduces no architecture and no training method; it reports a large set of controlled experiments and a definition for reading them. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak (all at Harvard) and Ilya Sutskever (OpenAI) trained families of networks that differ in one setting at a time and plotted test error against that setting. The curves have a peak in a predictable place, and the paper names the quantity that locates it. The page covers, in order:

1. the two rules of thumb the experiments reconcile;
2. double descent in a least-squares model small enough to draw, and why its error peaks;
3. effective model complexity, the paper's definition, and its hypothesis;
4. the three experiments: varying model size, dataset size and training time;
5. what changes the picture: early stopping, regularization, label noise.

Two rules of thumb that disagree

Classical statistics teaches the bias-variance trade-off. A model's expected test error splits into bias (how far the average fitted model is from the truth, high when the model family is too simple) and variance (how much the fitted model changes from one training sample to another, high when the model can bend to fit noise). Adding capacity lowers bias and raises variance, so test error plotted against model size is U-shaped: too small underfits, too large overfits, and somewhere in between is a best size, past which, the rule says, bigger is worse.

Deep learning practice says the opposite. Networks with millions of parameters can fit the training set exactly, even with the labels replaced by random noise (the result covered in the Rethinking Generalization explainer), yet larger networks keep getting better test scores, and practitioners scale up by default. Both camps also agree that more data never hurts; Section 7 of the paper finds settings where it does.

Belkin, Hsu, Ma and Mandal proposed in 2018 that both rules describe the same curve, which they called double descent. Up to the interpolation threshold (the smallest model that can fit the training data exactly, also called interpolating it), test error follows the classical U. At the threshold it peaks. Past it, test error falls again, and in their experiments with random features and two-layer networks trained with squared loss it fell below the bottom of the U. Similar curves had appeared earlier in statistical physics (Opper, 1995) and in work on the "jamming" transition of neural networks (Spigler et al., 2018). Nakkiran and colleagues set out to test whether the curve holds for modern networks trained the usual way, with cross-entropy loss, data augmentation, Adam or SGD, and whether model size is the only variable that produces it.

How double descent works in a model you can draw

Figure 1 is a least-squares problem small enough to compute in the browser and simple enough to see. There are $n = 24$ training points with inputs on $[-1, 1]$, labels from a triangle wave plus Gaussian noise of standard deviation $\sigma$, and a model that is a weighted sum of the first $d$ Fourier features (a constant, then $\cos(\pi x)$, $\sin(\pi x)$, $\tfrac12\cos(2\pi x)$, and so on, each frequency-$k$ pair scaled by $1/k$). Its $d$ weights are the model's parameters, so $d$ is the model size. Write $F$ for the $n \times d$ matrix whose row $i$ holds the features of training input $x_i$, and $y$ for the vector of labels.

For $d < n$ there are fewer weights than points, and least squares picks the single best compromise. For $d > n$ there are infinitely many weight vectors that fit all 24 points exactly, and the figure picks the one with the smallest Euclidean norm:

That choice is the one gradient descent makes. Started from $w = 0$, every gradient step on the squared loss adds a combination of the rows of $F$:

so the weights never leave the span of the rows, and the only interpolating solution in that span is the minimum-norm one (Hastie et al., 2019, Proposition 1; the Rethinking Generalization page has a two-weight picture of the same fact). When $d \ge n$ and the rows are independent, (1) has the closed form $\hat w = F^\top (F F^\top)^{-1} y$. The $1/k$ scaling matters here: a weight on a high-frequency feature has to be $k$ times larger to have the same effect, so the minimum-norm solution puts as little as it can on high frequencies and stays smooth.

Drag $d$ from 1 upward in Figure 1 and watch both panels. The bottom curve is the median test error over 15 random training sets, on a log scale; the hollow ring is the error of the training set drawn in the top panel.

At the default noise level, $\sigma = 0.15$, the median test error falls from 0.21 at $d = 1$ to 0.038 at $d = 13$, the classical sweet spot. It then rises to 0.073 at $d = 23$ and jumps to 0.62 at $d = 24$, sixteen times the sweet spot, while training error is zero to machine precision. One feature later it is back to 0.077, and it keeps falling: 0.034 by $d = 120$, lower than any model below the threshold. With $\sigma = 0$ the peak is 0.111 against 0.026 and 0.019 on either side; it does not vanish, because no finite set of sine waves reproduces the corners of a triangle wave, and the part of the target the model cannot represent behaves like noise. At $\sigma = 0.3$ the peak is 2.07 and the over-parameterized end (0.082 at $d = 120$) no longer beats the sweet spot (0.062 at $d = 7$), so "falls again" does not always mean "falls below the U".

This toy's peak is one feature wide. In the paper's networks the peak spans a range of widths, because training error approaches zero gradually rather than at a single size; the next sections come back to that.

Why the error peaks at d = n

At $d = n$ the matrix $F$ is square, and exactly one weight vector fits the data: $\hat w = F^{-1} y$. Split the labels into the clean target and the noise, $y = f + \varepsilon$. The fitted weights carry the noise term $F^{-1}\varepsilon$. A square matrix built from random inputs almost always has some direction it nearly flattens: its smallest singular value (the smallest factor by which it scales the length of any unit vector) is close to zero, and $F^{-1}$ multiplies the noise along that direction by one over that singular value. The weights become very large to thread the curve through every noisy point, and between the points the curve swings wildly. For the training set in Figure 1's first draw, $\lVert \hat w\rVert$ is 2.1 at $d = 23$, 12.2 at $d = 24$, and 2.3 at $d = 25$.

With one extra feature the matrix is wide, the solutions form a one-dimensional family, and the minimum-norm rule can choose a member that does not need huge weights. With many extra features the noise is spread across many directions, each absorbing a little of it with small weights. The paper states the same intuition for networks: "at the interpolation threshold, there is effectively only one model that fits the train data and this interpolating model is very sensitive to noise", while far past it there are many interpolating models and the optimizer finds one that absorbs the noise without wrecking its global structure.

For linear regression this is a theorem. Hastie, Montanari, Rosset and Tibshirani (2019) take $p$ features with independent, unit-variance entries, a true coefficient vector of squared length $r^2$, label noise of variance $\sigma^2$, and let $n$ and $p$ grow together with $p/n \to \gamma$. The test risk of the minimum-norm least-squares fit converges to

Below $\gamma = 1$ the fit has no bias (the model contains the truth) and its variance $\sigma^2\gamma/(1-\gamma)$ grows with $\gamma$. Above it, the variance $\sigma^2/(\gamma-1)$ shrinks with $\gamma$ and a bias $r^2(1-1/\gamma)$ appears, because the minimum-norm solution lies in the $n$-dimensional span of the training inputs and misses the part of the true coefficients outside it, on average a fraction $1-1/\gamma$ of $r^2$. Both variance terms diverge as $\gamma \to 1$. With $\sigma^2 = 1$ the variance is 1 at $\gamma = 0.5$, 9 at $\gamma = 0.9$, 99 at $\gamma = 0.99$, then 1 again at $\gamma = 2$ and 0.11 at $\gamma = 10$. In this well-specified setting the best risk is always on the under-parameterized side, where $R \to 0$ as $\gamma \to 0$. Hastie et al. show that when the features miss part of the signal (mis-specification, as in the triangle-wave toy) and the signal-to-noise ratio $r^2/\sigma^2$ exceeds 1, the global minimum can sit in the over-parameterized regime. Nakkiran and colleagues cite this line of work and say they believe similar mechanisms operate in deep networks; they do not prove it.

Effective model complexity

In least squares the threshold is easy to locate: $d = n$. In a network trained with SGD for a fixed number of steps, whether the training set gets fit exactly depends on far more than the parameter count: the optimizer, the learning rate, how long training runs, data augmentation, label noise. The paper replaces parameter count with a property of the whole training procedure. A training procedure $\mathcal{T}$ is anything that takes a labeled training set $S$ of $n$ samples and returns a classifier $\mathcal{T}(S)$: architecture, initialization, optimizer and schedule together. Its effective model complexity (EMC) with respect to a data distribution $\mathcal{D}$ is

where $\mathrm{Error}_S(M)$ is the mean error of model $M$ on its own training samples. In words: the largest number of training samples the procedure can still fit, on average, to within training error $\epsilon$. The paper uses $\epsilon = 0.1$ and says it has no principled way to choose it. For the least-squares toy with $d$ features and $\epsilon$ close to zero, EMC is $d$: any $d$ generic points can be fit exactly, and $d + 1$ noisy points cannot. Appendix D states the same for random Fourier features.

EMC differs from classical capacity measures in two ways the paper relies on. It depends on the true labels, so adding label noise lowers it (noisy labels are harder to fit), whereas VC dimension and Rademacher complexity are defined by fitting arbitrary or random labels and do not move when the label distribution changes. And it depends on the training procedure, so training longer raises it, whereas VC dimension and Rademacher complexity are properties of the model family alone and cannot express "the same network, trained for more epochs".

The paper's Hypothesis 1, which it labels informal, splits training procedures into three regimes by comparing EMC with the number of training samples $n$:

• Under-parameterized, EMC sufficiently smaller than $n$: any change that raises EMC lowers test error.
• Over-parameterized, EMC sufficiently larger than $n$: any change that raises EMC also lowers test error.
• Critically parameterized, EMC $\approx n$: a change that raises EMC can lower or raise test error.

"Sufficiently" is left undefined; the paper reports a critical interval around EMC $= n$ whose width depends on the data and the procedure "in ways we do not yet completely understand". The hypothesis makes three predictions, one per quantity that moves EMC or $n$. Making the model bigger raises EMC, so test error against model size should show a peak (model-wise double descent). Training longer raises EMC, so test error against epochs should show a peak (epoch-wise). Adding training samples raises $n$, which moves the critical regime to larger models, so for a model near the new threshold more data should hurt (sample-wise non-monotonicity).

Model-wise double descent in real networks

Three model families make up the experiments, each scaled by one width hyperparameter. ResNet18s (the pre-activation variant of the ResNet design, four stages of two residual blocks) use widths $[k, 2k, 4k, 8k]$; $k = 64$ is the standard ResNet18 with 11.2 million parameters. Five-layer CNNs have four convolutional layers of widths $[k, 2k, 4k, 8k]$, each followed by batch normalization and ReLU, with max-pooling that shrinks the 32×32 image to 1×1 by the end, then one linear layer. Their parameter count, from the official code, is

which gives 1,558,026 at $k = 64$, the number in Appendix B, and 25,418 at $k = 8$. The third family is the encoder-decoder Transformer (6 layers, 8 attention heads), scaled by its embedding dimension $d_{\text{model}}$ with the feed-forward width set to $4\,d_{\text{model}}$. Image models train on CIFAR-10 and CIFAR-100 (50,000 training images each) either with Adam at a constant learning rate of 0.0001 for 4,000 epochs, or with SGD (no momentum) for 500,000 gradient steps on a decaying schedule, batch size 128 in both cases. Transformers train for 80,000 gradient steps on IWSLT'14 German-to-English (160,000 sentence pairs) and a 200,000-sentence subsample of WMT'14 English-to-French, with 10% label smoothing (the training target puts 0.9 of its probability on the correct token and spreads 0.1 over the vocabulary) and no dropout. No run uses weight decay or dropout unless stated, and every run trains for a fixed long budget instead of stopping when test error is lowest, because the question is what happens when training continues to the end.

Most experiments add label noise. With probability $p$ a training image's label is replaced by a uniformly random wrong label, drawn once before training, so the network sees the same wrong label every epoch. Figure 1 of the paper reports test error on equally noisy test labels; the other plots use clean test labels. The two are related exactly. With $K$ classes, a correct prediction is scored wrong when its test label was flipped (probability $p$), and a wrong prediction is scored right only when the flipped label happens to equal it (probability $p/(K-1)$). Summing the two cases:

For CIFAR-10 at 15% noise this is $0.15 + 0.833\,\mathrm{err}_{\text{clean}}$, so a clean test error of 0.20 reads as 0.317 on noisy labels. The map is linear and increasing, which is why the paper calls the two curves "just linear rescaling of one another": peaks sit at the same widths in both.

The results, read off the paper's Figure 4 for ResNet18 on CIFAR-10 with Adam and data augmentation: with no label noise the test error falls almost monotonically, with only a slight plateau around the interpolation point. With 15% noise it falls to about 0.22 near width 4, climbs to about 0.31 near width 10, then declines to about 0.165 at width 64. At 20% noise the peak is about 0.345, at width 10 to 12. The training-error panel shows why the peak sits there: noisier labels need wider networks to fit, and training error reaches roughly zero near width 8 with clean labels and near width 15 with 20% noise. A width-10 ResNet18 has 274,540 parameters, about 5.5 per training image, and yet it is the model that can barely fit the training set. On CIFAR-100 the peak appears even with clean labels.

The other changes to the training setup move the peak the way EMC predicts. Data augmentation (random crops and flips, each copy keeping its image's possibly-wrong label) makes the training set harder to fit and shifts the peak to wider CNNs (paper Figure 5). More label noise shifts it to wider models and makes it taller. Transformers on both translation tasks show peaks with no label noise at all (Figure 8). The paper also reports a peak in robust test error for adversarially trained ResNets with clean labels (Figure 26).

When more data hurts

More training samples have two effects on the curve of test error against model size. They lower it, as expected. And they move the peak to larger models, because fitting more samples takes a bigger model. Where the peak moves onto a model size that used to sit past the old peak, the two effects collide.

Figure 2 shows the collision in the least-squares toy. Each cell is the median test error for $d$ features and $n$ training points at $\sigma = 0.15$. A row (fixed $n$) is the curve of Figure 1. A column (fixed $d$) is what happens to one model as data grows. Click a column right of the diagonal and read its plot.

The bright ridge lies on $d = n$. Take the column $d = 36$: with 24 training points the median test error is 0.036; with 34 points it is 0.091; with 36 points, the one sample count this model can just fit, it is 3.3. Fifty percent more data made this model about ninety times worse, and the model with 48 points is better again (0.027). A model in the over-parameterized regime gets pushed back toward the critical regime as data arrives. Switch the ridge penalty from 0 to 0.01 and the diagonal ridge disappears: from 12 points upward, the column $d = 36$ stays between 0.02 and 0.06, with 0.041 at $n = 36$.

The paper finds the same collision in Transformers trained to completion on IWSLT'14 German-to-English with either 4,000 or 18,000 sentence pairs (its Figure 3, re-plotted as Figure 3 here). The 4k curve peaks at $d_{\text{model}} = 24$; the 18k curve peaks at $d_{\text{model}} = 56$, because 4.5 times as many pairs need a bigger model to fit. The loss is per-token perplexity, the exponential of the average cross-entropy per target token; a perplexity of 12 means the model is, on average, as uncertain as a uniform choice among 12 tokens.

For the smallest model, $d_{\text{model}} = 8$, more data helps as usual (about 8.4 against 9.8). Between roughly 52 and 100 the 18k model is worse; at $d_{\text{model}} = 64$ it is about 15.6 against 13.6, a gap of two points. Past 100 more data helps again. The paper draws its arrows from 56 to 88. In the CNN experiments the collision shows up as a plateau rather than a reversal: with 20% label noise there is a band of widths, roughly 30 to 42, where four times as many training images (12,500 against 50,000) give no lower test error (paper Figure 11a). The paper's Appendix D shows the same peak along $n = d$ for random Fourier features on Fashion-MNIST, where EMC equals the number of features exactly.

Epoch-wise double descent

Training longer raises EMC: a network that cannot fit the training set after 10 epochs may fit it after 1,000. If EMC is the right axis, a model large enough to eventually interpolate should pass through the critical regime during training, and its test error over epochs should show a peak and a second descent. Epoch-wise double descent is the paper's new prediction; to the authors' knowledge it had not been proposed before.

The paper's Figure 9 trains ResNet18s on CIFAR-10 with 20% label noise, Adam and data augmentation, and plots test error against epochs for three widths. The width-3 model never gets near interpolation, and its test error falls monotonically to about 0.25. The width-12 model, which only just reaches near-zero training error by the end, follows the classical U: lowest at about 0.23 near 100 epochs, then rising to about 0.35. The standard ResNet18 (width 64) reaches about 0.18 by epoch 20, climbs to about 0.3 near epoch 60, and comes back down to about 0.20 by epoch 4,000. In that run the second descent ends slightly above the early minimum; with 10% noise (paper Figure 10) the final error ends below it. The effect survived every optimizer and schedule the authors tried: Adam, SGD and SGD with momentum 0.9, each with a constant, an inverse-square-root, and a "dynamic drop" learning rate (cut by a factor whenever training loss stops improving) (Appendix E.1).

The paper offers EMC as the account of this curve and stops there. Heckel and Yilmaz (2020) later gave a mechanism that can be solved exactly in a linear model: a network learns some parts of its function much faster than others, and each part has its own bias-variance trade-off over time. Figure 4 is that model. There are 100 training samples and 70 features. Sixty features have inputs of scale 1 and carry half the signal; ten have inputs 20 times smaller and carry the other half, so gradient descent learns them about 400 times more slowly (the gradient on a weight scales with its input's variance). The labels have noise of standard deviation 0.4. Gradient descent from zero is solved in closed form at every step and averaged over 12 random draws.

At a multiplier of 1, test error starts at 2.23, drops to 1.83 within two steps as the fast features pick up their half of the signal, then climbs to 3.04 by step 108 as the same features fit the label noise (their 60 weights against 100 noisy samples overfit). From a few hundred steps on, the slow features learn their half of the signal and the error falls to 0.40 by step 16,000, well under the early minimum. Stopping at the first minimum would have given 1.83. Raise the multiplier to 400 and both groups learn at the same rate: the curve becomes a single U with its minimum, 0.30, at step 37, and the same final value of 0.40. Heckel and Yilmaz report the same in networks: adjusting per-layer step sizes removes the epoch-wise bump and improves early-stopping performance.

The toy does not interpolate (70 parameters for 100 samples, final training error 0.054), so it illustrates the two-speed mechanism, not the paper's EMC crossing.

Early stopping, regularization and label noise

Early stopping. Most of the paper's peaks disappear if each model is stopped at its best test error. This fits the hypothesis: a model stopped before it fits the noisy labels never has EMC near $n$. The paper reports one exception, ResNet18 on CIFAR-100 with clean labels, where model-wise double descent survives optimal early stopping (Figure 19). It also states that it never observed more data hurting under optimal early stopping, while knowing of no reason it could not. The translation experiments (Figure 23) show this directly: with optimal early stopping, 18k samples beat 4k at every model size.

Regularization. The paper notes that varying weight decay for a fixed model gives a similar double-descent curve (Figure 22, flagged as preliminary). A follow-up by Nakkiran, Venkat, Kakade and Ma (2020) proves that for certain linear regression models with isotropic data, an optimally tuned L2 penalty makes test error monotone in both model size and sample size, and shows empirically that it mitigates double descent in networks. Figure 2's ridge buttons are a small instance: a penalty of 0.01 removes the ridge along $d = n$ because it stops the fit from inverting near-zero singular values. With Adam, an L2 penalty added to the loss behaves differently from shrinking the weights directly; the AdamW explainer covers why.

Label noise. The peaks are strongest with label noise, but the paper argues noise stands in for mis-specification, the part of the labeling function the model family cannot represent. Its thought experiment: replace random label noise with pseudorandom noise generated by a fixed function. The best possible classifier is unchanged (the corruption is deterministic and invertible), but to the network the labels look just as unfittable, so the peak should look the same. The toy in Figure 1 shows the clean-label version: a triangle wave is outside every finite Fourier model, and the peak at $d = n$ survives $\sigma = 0$.

Ensembling. Averaging five independently trained models by plurality vote helps most near the peak (paper Figure 28, ResNet18s with 15% noise, each trained on its own independently corrupted labels). That matches the explanation in the section on the toy: at the threshold each model's prediction depends heavily on which noise it happened to fit, and different runs fit it differently, so their errors partly cancel.

Reproducing the setup

The authors released the model definitions and their raw results. The CNN parameter count in (5), the label-noise protocol, and the SGD learning-rate schedule fit in a few lines:

# Width-k 5-layer CNN from the official repo (models/mcnn.py), CIFAR-10
# conv widths [k, 2k, 4k, 8k], 3x3 kernels with bias, BatchNorm, ReLU, max-pool
def mcnn_params(k, classes=10):
    convs = [(3, k), (k, 2*k), (2*k, 4*k), (4*k, 8*k)]
    p = sum(9*cin*cout + cout for cin, cout in convs)   # weights + biases
    p += sum(2*cout for _, cout in convs)               # BatchNorm scale, shift
    return p + 8*k*classes + classes                    # final linear layer

mcnn_params(64)   # 1,558,026, the count in Appendix B.1

# Label noise with probability p, drawn ONCE before training (not per epoch)
def corrupt(labels, p, K, rng):
    out = labels.copy()
    for i in range(len(out)):
        if rng.random() < p:                    # replace with a WRONG label,
            wrong = rng.integers(K - 1)         # uniform over the other K-1
            out[i] = wrong if wrong < labels[i] else wrong + 1
    return out
# every random crop / flip of image i keeps the label corrupt() gave it

# SGD recipe: batch 128, no momentum, no weight decay, 500K gradient steps
def lr(t, g0=0.1):
    return g0 / (1 + t // 512) ** 0.5           # equation (7)

The schedule is defined in gradient steps rather than epochs so that runs with different training-set sizes are comparable: with batch size 128, one epoch over 50,000 images is 391 steps, and the learning rate halves each time $1 + \lfloor t/512\rfloor$ quadruples. For the Adam runs the recipe is a constant learning rate of 0.0001 and PyTorch defaults for everything else. The model-wise experiments train one network per width to the full budget, so a single curve in the paper costs dozens of 4,000-epoch runs; the authors report means over 5 random initializations for Figures 1 and 7.