Decoupled Weight Decay Regularization

Weight decay and L2 regularization are the same for SGD, but not for Adam.

Most deep-learning libraries implement L2 regularization and call it weight decay. For adaptive optimizers like Adam the two are not the same, and separating them is a one-line change that lets Adam generalize as well as SGD.

Open any training script and you will find a knob labeled weight decay. For Adam, that knob usually does something subtly different from weight decay, and the difference is why Adam earned a reputation for generalizing worse than plain SGD.

Two techniques in machine learning are treated as the same thing so routinely that the libraries themselves conflate them. One is L2 regularization: add a penalty for large weights to the loss. The other is weight decay: pull every weight a little toward zero on every step. PyTorch, TensorFlow, and their predecessors all expose a weight_decay argument that, under the hood, does L2. For decades nobody worried about the difference, because for ordinary stochastic gradient descent there isn't one.

This paper, by Ilya Loshchilov and Frank Hutter, makes a narrow and surprisingly consequential point: the equivalence holds for SGD and fails for adaptive optimizers like Adam. Because the popular libraries only implemented the L2 version, Adam was getting a weaker, lopsided form of regularization, and that is a large part of why it generalized worse than SGD with momentum on image classification. Applying weight decay the original way, separate from the gradient step, repairs it. The authors call the result AdamW, and it has since become the default optimizer for training large models.

A handful of ideas explain all of it: what L2 regularization is, what the original weight decay is, why they coincide for SGD, where a decay term enters the Adam update, and why that one placement decision breaks the equivalence. Then two practical refinements and the empirical payoff. None of the pieces is hard.

L2 regularization, and the one-half

Start with the version everyone knows. To discourage a network from leaning on large weights, you add a penalty to the loss that grows with the size of the weights:

Here $f$ is the ordinary loss, $\boldsymbol{\theta}$ is the weight vector, $\lVert\boldsymbol{\theta}\rVert_2^2 = \sum_i \theta_i^2$ is the squared length of that vector, and $\lambda'$ sets how hard the penalty pushes. Minimizing this modified loss trades a little extra training error for smaller weights. Smaller weights usually mean a smoother, less brittle function, the kind that tends to do better on data it has not seen, which is why we regularize in the first place.

The $\tfrac12$ in front is not decoration. The gradient of the penalty is what actually enters training, and differentiating $\tfrac{\lambda'}{2}\lVert\boldsymbol{\theta}\rVert_2^2$ brings the square's exponent down to cancel the two:

Without the one-half you would carry a stray factor of two through every formula below. So the penalty contributes a clean term $\lambda'\boldsymbol{\theta}$ to the gradient, pointing back toward the origin in proportion to each weight. Run one step of gradient descent with learning rate $\alpha$ and the update is $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \alpha\nabla f - \alpha\lambda'\boldsymbol{\theta}$: the ordinary step, plus a small tug toward zero. That tug, scaled by $\alpha\lambda'$, is the only thing the penalty ever does. Hold onto that scaling by $\alpha$; it is where the trouble starts.

Weight decay: shrink the weights directly

There is an older way to keep weights small that never touches the loss function. Before (or after) each gradient step, multiply every weight by a number slightly less than one. That is weight decay, introduced in this form by Hanson and Pratt in 1988:

Read it left to right: take the current weights, shave off a fixed fraction $\lambda$ of every one of them, then take the usual gradient step. The decay is a flat haircut. A weight at $3.0$ with $\lambda = 0.01$ loses $0.03$; a weight at $0.2$ loses $0.002$; both lose the same one percent. Left alone, with no gradient pushing back, a weight would shrink geometrically toward zero, compound interest run in reverse.

The contrast with L2 that matters later is small and easy to miss. In equation (1) the decay term $\lambda\boldsymbol{\theta}$ is applied directly to the weights. It is not part of the loss, not part of the gradient, and crucially not multiplied by the learning rate. It is its own separate operation, sitting beside the gradient step rather than inside it. Compare that with L2, where the penalty entered as a gradient term $\lambda'\boldsymbol{\theta}$ and then got scaled by $\alpha$ along with everything else. The two recipes look almost identical. The next section shows they are, for SGD, and then the section after that shows why the resemblance breaks for Adam.

For plain SGD, the two coincide

Set the two updates side by side. L2 regularization, expanded, gives $\boldsymbol{\theta} - \alpha\nabla f - \alpha\lambda'\boldsymbol{\theta}$. Weight decay, from equation (1), gives $(1-\lambda)\boldsymbol{\theta} - \alpha\nabla f$, which is the same as $\boldsymbol{\theta} - \lambda\boldsymbol{\theta} - \alpha\nabla f$. Line them up:

They are identical except for the shrink term. L2 shrinks by $\alpha\lambda'$; weight decay shrinks by $\lambda$. Set those equal and you get the paper's first proposition: the two are exactly the same optimizer, provided you choose

That asymmetry drives everything that follows. Weight decay's shrink term is not multiplied by the learning rate; L2's is, because it enters through the gradient. To produce the same shrink, L2's coefficient has to be larger by a factor of $1/\alpha$, and for a typical $\alpha < 1$ that makes $\lambda'$ larger than $\lambda$, sometimes much larger. It also couples the two knobs: the effective amount of regularization applied to a weight is $\alpha\lambda'$, so if you find a good $\lambda'$ and then change the learning rate, the regularization strength changes with it, and the setting that was optimal is now wrong. You cannot tune one knob and leave the other alone. The figure below shows what this does to the search you run when you train a model: a grid over the learning rate $\alpha$ and the regularization factor, colored by test error, brightest where the model does best.

Watch the bright region rotate as you toggle. In the coupled panel the good settings lie along a diagonal, so optimizing the two hyperparameters means searching a tilted valley, and a one-at-a-time sweep keeps walking off the ridge. Decoupling, which the rest of this piece works out, swings that valley flat: pick a learning rate, then tune the decay independently, and one decay value keeps working as you change the rate. For SGD the coupling is a nuisance you can rescale away with equation (2). This paper exists because for Adam no single rescale works.

Adam, in one line

To see why, recall what Adam does differently from SGD. Plain SGD multiplies the gradient by one global learning rate. Adam gives every parameter its own effective rate, set by how big that parameter's gradients have recently been. It keeps two running averages: $\boldsymbol{m}_t$, a smoothed gradient (momentum), and $\boldsymbol{v}_t$, a smoothed average of the squared gradient. After a small bias correction (writing $\hat{\boldsymbol{m}}_t$ and $\hat{\boldsymbol{v}}_t$), the update is:

Everything adaptive about Adam lives in that denominator. $\sqrt{\hat{\boldsymbol{v}}_t}$ is a per-parameter root-mean-square of recent gradient sizes (a magnitude, not a standard deviation, since $\boldsymbol{v}_t$ averages the raw square). Dividing by it means a parameter whose gradients have been large takes smaller steps, and one whose gradients have been small takes larger steps, so every parameter moves at a comparable pace regardless of how steep its slice of the loss is. That per-parameter rescaling is what makes Adam robust to badly scaled problems and easy to use out of the box. You can write it compactly as a preconditioner, a diagonal matrix $\boldsymbol{M}_t = \text{diag}\big(1/(\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon)\big)$ that reshapes the step, so the update is $\boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \alpha\,\boldsymbol{M}_t\hat{\boldsymbol{m}}_t$.

Now the question this paper turns on. We want to add regularization, a pull toward zero, to this update. That pull has to enter somewhere. Does it go in before the $\sqrt{\hat{\boldsymbol{v}}_t}$ division, or after it?

Where the decay enters

There are exactly two places to put it, and they are the two techniques from the start of this piece.

L2 in Adam (coupled). Fold the penalty into the gradient, exactly as L2 prescribes: $\boldsymbol{g}_t = \nabla f_t + \lambda\boldsymbol{\theta}_{t-1}$. Now $\lambda\boldsymbol{\theta}$ is part of $\boldsymbol{g}_t$, so it flows into both running averages and gets divided by $\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon$ right along with the real gradient. This is what torch.optim.Adam(weight_decay=...) does.

AdamW (decoupled). Do what the original weight decay did: leave the gradient alone, and subtract $\lambda\boldsymbol{\theta}$ from the weights directly, after the normalized step, where the $\sqrt{\hat{\boldsymbol{v}}_t}$ division can no longer touch it:

Equation (4) is the complete algorithm: the $\lambda\boldsymbol{\theta}_{t-1}$ term sits outside the fraction, added at the end. The $\eta_t$ in front is a schedule multiplier (a global factor you can ramp up or down over training); it scales both terms together, and you can ignore it for now by reading it as one. The figure traces a single decay packet through the pipeline so you can see where it joins.

The two differ by where one term is written, nothing else. The moments, the bias correction, the defaults ($\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$) are all unchanged. Move $\lambda\boldsymbol{\theta}$ from inside the $\sqrt{\hat{\boldsymbol{v}}_t}$ division to outside it and you have turned Adam into AdamW. The rest of this piece is about why so small a change matters so much.

# one Adam-family step. The only difference is the last line.
g  = grad(f, theta)                    # loss gradient only
g  = g + lam * theta                   # Adam+L2: fold decay into g  (A)
m  = b1 * m + (1 - b1) * g             # 1st moment EMA
v  = b2 * v + (1 - b2) * g * g         # 2nd raw moment EMA
mh = m / (1 - b1**t)                   # bias-corrected
vh = v / (1 - b2**t)
step = alpha * mh / (sqrt(vh) + eps)   # the normalized Adam step
theta = theta - eta * step             # Adam+L2 stops here (decay already folded in at (A))
theta = theta - eta * lam * theta      # AdamW: decay theta directly  (B)

Why Adam breaks the match

For SGD, the coupling was annoying but curable: one rescale, $\lambda' = \lambda/\alpha$, made L2 and weight decay identical. The paper's second proposition says that for Adam no such rescale exists. The argument is short: when you fold L2 into the gradient, the penalty term $\lambda'\boldsymbol{\theta}$ is multiplied by the same preconditioner as the real gradient, so the L2 step shrinks the weights by $\alpha\lambda'\boldsymbol{M}_t\boldsymbol{\theta}$, while decoupled weight decay shrinks them by $\lambda\boldsymbol{\theta}$. For those to be equal at every weight you would need

and a single scalar $\lambda'$ can pull that off only if $\boldsymbol{M}_t$ is a scalar multiple of the identity, the same number on every parameter. For SGD it is (the preconditioner equals $1$), so the rescale works. For Adam, $\boldsymbol{M}_t$ is deliberately non-uniform, so no scalar reproduces a uniform shrink. L2 and weight decay have genuinely come apart.

Make the gap concrete, because it has a direction. Under L2, the decay a weight actually receives is its intended decay times $\alpha/(\sqrt{\hat{v}}+\epsilon)$, the same factor the normalization applies to its gradient. A weight whose gradients have been large, say $\sqrt{\hat{v}} = 0.1$ at the default $\alpha = 0.001$, gets only about one percent of the intended decay; a weight with a tiny gradient history gets several times too much. Decoupled weight decay gives every weight exactly the intended amount. Drag a weight along its gradient history below and watch the two diverge.

There is a partial way to describe what decoupled weight decay does instead, and it points at why the change might help. The paper's third proposition freezes the preconditioner at a fixed matrix $\text{diag}(\boldsymbol{s})^{-1}$ and shows that, in that idealized case, decoupled weight decay is equivalent to a scale-adjusted L2 penalty:

The $\sqrt{\boldsymbol{s}}$ weighting is the interesting part. A weight with a large $s_i$ (a large historic gradient) is penalized more here, in proportion to $\sqrt{s_i}$, the exact opposite of what L2 inside Adam does to it. The motivation the authors offer is the flat-versus-sharp-minimum idea: weights whose gradients are large are the ones where small changes move the loss a lot, the brittle directions, and leaning on them harder may push the optimizer toward flatter regions that generalize better. That last step is a hypothesis, not something proposition 3 proves; the proposition only establishes the single-step equivalence.

A second, independent argument arrives at the same place. Aitchison (2018) showed that adaptive optimizers can be read as Bayesian filtering, tracking a distribution over the best weights as the other weights shift underneath them. In that reading Adam's preconditioner is the posterior uncertainty (you take bigger steps on weights you are less sure of), and the slow drift of the estimate between steps is modeled by a prior $\mathcal{N}\big((\boldsymbol{I}-\boldsymbol{A})\boldsymbol{\theta}_t, \boldsymbol{Q}\big)$. Set $\boldsymbol{A}=\lambda\boldsymbol{I}$ and that prior multiplies the mean by $(1-\lambda)$ each step, which is exactly decoupled weight decay. The pull comes from the data-independent prior and does not depend on how certain you are about any weight; an L2 penalty would have made the pull scale with certainty. So under this theory it is weight decay, not L2, that emerges naturally. Both arguments land on weight decay, not L2.

Two practical add-ons

With the core idea in hand, the paper adds two refinements that make AdamW pleasant to use. Both are independent of the decoupling; they just remove sharp edges.

Normalized weight decay. The best raw $\lambda$ depends on how long you train. Every update applies another small shrink, so a longer run accumulates more total decay, and the per-step value that was right for a short run is too strong for a long one. The authors absorb this by writing the raw decay in terms of a budget-free number $\lambda_{\text{norm}}$:

where $b$ is the batch size, $B$ the dataset size, and $T$ the number of epochs, so $I$ is the total number of weight updates. The raw $\lambda$ falls off as one over the square root of the number of updates; $\lambda_{\text{norm}}$, the decay you would use for a single batch pass, stays put. You tune it once and reuse it across budgets and datasets.

That factor of five is not hypothetical. Train on ImageNet32×32, whose epochs are about 24 times longer than CIFAR-10's, with the raw $\lambda$ that was optimal for CIFAR-10, and you would be decaying roughly five times too hard. Normalizing makes the optimal $\lambda_{\text{norm}}$ land near the same value ($0.025$ to $0.05$) across both datasets and across SGDW and AdamW alike, which is most of the convenience.

Cosine annealing with warm restarts. The second add-on schedules the multiplier $\eta_t$ from equation (4). Within a run it cools the multiplier from one down to zero along a cosine, then restarts, snapping it back to one and beginning a new, longer run:

where $T_{\text{cur}}$ counts epochs since the last restart and $T_i$ is the length of the current run, each run a factor of $T_{\text{mult}}$ longer than the last. Because $\eta_t$ scales both the gradient step and the decay, a restart relaxes the step and the decay together. The low points, where $\eta_t = 0$, are good moments to snapshot the model.

Combining AdamW with this schedule gives AdamWR. Restarts do not change the final answer much, but they reach a good answer far sooner, up to ten times faster to a usable model than running the cosine straight through. Normalized weight decay is what made the combination practical: it let the authors keep one decay setting even as each restart stretched the run length.

AdamW closes the gap

The headline number is a 15% relative reduction in test error over Adam with L2, at the default learning rate of $0.001$, and it holds on both CIFAR-10 and ImageNet32×32. Relative is the operative word: if Adam was at a 6% error, AdamW lands near 5.1%, enough to close most of the gap to SGD with momentum. The improvement comes from better generalization, not faster convergence alone; matched at equal training loss, AdamW still tests better.

Figure 1's basin shows the other half of the result. With decoupled weight decay the good region of the $(\alpha, \lambda)$ grid straightens out, so the two knobs can be tuned almost independently. That alone helps a lot: it turns a search over a tilted diagonal into two nearly separate one-dimensional searches.

Be precise about the claim, because it is easy to inflate. AdamW does not set a new state of the art; the $2.86\%$ CIFAR-10 figure floating around the experiments belongs to the Shake-Shake network trained with SGD that the authors borrow as their setup, not to AdamW. What AdamW does is make Adam competitive with SGD plus momentum on problems where Adam used to trail it, so practitioners no longer have to switch optimizers to get good generalization. This mattered: the decoupled form is now the default in PyTorch's torch.optim.AdamW and the optimizer behind most large language models. (The older torch.optim.Adam(weight_decay=...) still gives you coupled L2, and its default decay is zero where AdamW's is $0.01$, so the two are genuinely different calls.)

Everything traces back to one relocation. L2 regularization writes a decay term into the gradient, where Adam's per-parameter normalization then shrinks it unevenly, hardest on exactly the weights you most want to regularize. Move that term out of the gradient and apply it straight to the weights, the way weight decay was originally defined, and the unevenness disappears: every weight decays by the same fraction, the learning rate and the decay no longer interact, and Adam catches up to SGD. One line of code, in a place nobody had thought to look.