VerifiedarXiv:1711.0510123 min
Optimization · Regularization

AdamW: Decoupled Weight Decay Regularization

Weight decay and L2 regularization are the same for SGD, but not for Adam.

Deep-learning libraries implemented L2 regularization and called it weight decay. Inside Adam, the L2 penalty is divided by each weight's gradient size, so weights with large gradients get much less of the decay than weights with small ones. AdamW subtracts the decay from the weights directly, after Adam's division.

Explaining the paperDecoupled Weight Decay RegularizationLoshchilov & Hutter · University of Freiburg · ICLR 2019 · arXiv:1711.05101 ↗

A 2017 paper that moves the weight-decay term in Adam's update out of the gradient (Algorithm 2, line 6) and into the weight update (line 12). On CIFAR-10, Adam with L2 regularization did no better than Adam without it; with the term moved, test error fell 15% (relative) below Adam with L2.

Two techniques for keeping a network's weights small are treated as the same thing so routinely that the libraries merged them. L2 regularization adds a penalty for large weights to the loss. Weight decay multiplies every weight by a number slightly below one on every step. When this paper appeared, the common deep-learning libraries implemented only the first and exposed it as an argument named weight_decay. For plain stochastic gradient descent (SGD) that naming is harmless: the two produce identical updates once one coefficient is rescaled.

Ilya Loshchilov and Frank Hutter show that the equivalence breaks for adaptive optimizers such as Adam, which divide each weight's step by the size of its recent gradients. Fed through that division, the L2 penalty shrinks weights unevenly, and least of all the weights with large gradients. They propose applying the decay the original way, to the weights and outside the division, and call the result AdamW (and SGDW for the SGD-with-momentum version). AdamW is widely used for large models; LLaMA, for example, was trained with it.

The page builds up in this order: what L2 regularization and weight decay each do to a weight, why they coincide for SGD, how Adam rescales steps, where the decay term enters the Adam update, why no L2 coefficient can match decoupled decay, and what decoupled decay penalizes instead. Then the paper's two practical additions (normalized weight decay and warm restarts), the experiments, and how this maps onto PyTorch.

L2 regularization and weight decay

L2 regularization changes the loss. To ff, the ordinary training loss of the weight vector θ\boldsymbol{\theta}, it adds a penalty proportional to the squared length of that vector:

freg(θ)=f(θ)+λ′2 ∥θ∥22f^{\text{reg}}(\boldsymbol{\theta}) = f(\boldsymbol{\theta}) + \frac{\lambda'}{2}\,\lVert\boldsymbol{\theta}\rVert_2^2

The norm ∥θ∥22\lVert\boldsymbol{\theta}\rVert_2^2 is the sum of the squared weights and λ′\lambda' sets how hard the penalty pushes. Smaller weights usually mean a smoother function that does better on data it has not seen, which is the reason to regularize. Training only ever uses the penalty through its gradient. Differentiating θi2\theta_i^2 gives 2θi2\theta_i, and the one-half in front cancels that 2:

∇freg(θ)=∇f(θ)+λ′θ\nabla f^{\text{reg}}(\boldsymbol{\theta}) = \nabla f(\boldsymbol{\theta}) + \lambda'\boldsymbol{\theta}

One gradient step with learning rate α\alpha is then

θ←θ−α∇f−αλ′θ\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \alpha\nabla f - \alpha\lambda'\boldsymbol{\theta}

the ordinary step plus a pull toward zero of size αλ′\alpha\lambda' times each weight. The pull is multiplied by the learning rate because it travels inside the gradient.

Weight decay never touches the loss. Hanson and Pratt (1988) described it as an exponential decay of the weights, applied alongside the gradient step:

θt+1=(1−λ) θt−α ∇ft(θt)\boldsymbol{\theta}_{t+1} = (1-\lambda)\,\boldsymbol{\theta}_t - \alpha\,\nabla f_t(\boldsymbol{\theta}_t)
(1)

Every weight loses the same fraction λ\lambda of itself each step, then the gradient step happens as usual. With λ=0.01\lambda = 0.01, a weight at 3.0 loses 0.03 and a weight at 0.2 loses 0.002, one percent each. A weight with no gradient pushing back shrinks geometrically: after 100 steps it keeps 0.99100≈0.370.99^{100} \approx 0.37 of its value. In (1), λ\lambda stands alone and α\alpha multiplies only the gradient.

Why weight decay and L2 match for SGD

Expand both updates for plain SGD and line them up:

θ−α∇f−αλ′θ⏟L2 regularizationvsθ−α∇f−λθ⏟weight decay\underbrace{\boldsymbol{\theta} - \alpha\nabla f - \alpha\lambda'\boldsymbol{\theta}}_{\text{L2 regularization}} \qquad\text{vs}\qquad \underbrace{\boldsymbol{\theta} - \alpha\nabla f - \lambda\boldsymbol{\theta}}_{\text{weight decay}}θ−α∇f−αλ′θ⏟L2 regularizationvsθ−α∇f−λθ⏟weight decay\begin{gathered}\underbrace{\boldsymbol{\theta} - \alpha\nabla f - \alpha\lambda'\boldsymbol{\theta}}_{\text{L2 regularization}} \\ \text{vs} \\ \underbrace{\boldsymbol{\theta} - \alpha\nabla f - \lambda\boldsymbol{\theta}}_{\text{weight decay}}\end{gathered}

They differ only in the shrink term: αλ′\alpha\lambda' against λ\lambda. Setting those equal is the paper's Proposition 1, which the authors call a well-known fact: SGD with weight decay λ\lambda takes exactly the same steps as SGD on the L2-regularized loss when

λ′=λα\lambda' = \frac{\lambda}{\alpha}
(2)

With a learning rate of 0.1 and a weight decay of 5×10−45\times10^{-4}, the matching L2 coefficient is 5×10−35\times10^{-3}, ten times larger, because L2's pull gets multiplied by α\alpha on the way in and has to start out bigger to end up the same size.

Equation (2) also couples the two hyperparameters. The shrink a weight receives under L2 is αλ′\alpha\lambda', so if you find a good λ′\lambda' and then halve the learning rate, you have also halved the regularization, and the old λ′\lambda' is no longer the good one. A grid search over learning rate and L2 coefficient therefore finds its good settings along a diagonal. In the paper's SGD grid (a 26-layer ResNet on CIFAR-10), one of the ten best settings sits at α=1/2\alpha = 1/2, L2 coefficient λ′=18×0.001\lambda' = \tfrac18 \times 0.001 (the paper writes it as λ\lambda): changing either value alone makes the result worse, while changing both together can still improve it. With SGDW, the same search is close to separable. Fixing a badly tuned α=1/1024\alpha = 1/1024 and tuning only the decay still reaches a good value, 14×0.001\tfrac14\times 0.001.

The figure shows the shape of that search. Look for the orientation of the bright basin as you toggle.

Figure 1 · the hyperparameter basin
A stylized test-error landscape over learning rate α (x) and decay factor λ (y), brighter is better; rings mark the ten best grid points, as the black circles do in the paper's Figure 2, and the amber dot the best one. Toggle between L2 (coupled), where the basin runs along a diagonal, and decoupled decay, where it lies flat. Hover to read a grid point. The colors are a model, not data; the paper's grids come from a 26 2×64d ResNet (two residual branches, first block 64 channels wide) on CIFAR-10.

In the coupled panel, a one-at-a-time sweep (fix α\alpha, tune λ\lambda, then fix λ\lambda, tune α\alpha) walks off the diagonal ridge after every move. In the decoupled panel the same sweep converges in one pass. The paper suggests this coupling may be part of why SGD has a reputation for being sensitive to its hyperparameters. For SGD, though, equation (2) is an exact fix in principle. For Adam no rescale works, for the reason the next two sections walk through.

How Adam sizes each weight's step

SGD multiplies every gradient by the same learning rate. Adam keeps two running averages per weight and uses them to give each weight its own step size: mt\boldsymbol{m}_t, an exponential moving average of the gradient (momentum), and vt\boldsymbol{v}_t, an exponential moving average of the squared gradient. After dividing each by a bias-correction factor (written m^t\hat{\boldsymbol{m}}_t, v^t\hat{\boldsymbol{v}}_t; the Adam explainer derives it), the update is

θt=θt−1−α m^tv^t+ϵ\boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \alpha\,\frac{\hat{\boldsymbol{m}}_t}{\sqrt{\hat{\boldsymbol{v}}_t} + \epsilon}
(3)

with all operations elementwise and ϵ=10−8\epsilon = 10^{-8} outside the square root. v^\sqrt{\hat{v}} is the root-mean-square of a weight's recent gradients. It is a magnitude, not a standard deviation: vv averages the raw square g2g^2, not the squared deviation from a mean.

Dividing by it makes the step roughly the same size for every weight. Take two weights whose gradients have been steady: weight A at about 0.1 per step, weight B at about 0.0001. For a steady gradient, m^\hat{m} approaches the gradient and v^\sqrt{\hat{v}} its absolute value, so the ratio approaches ±1 and both weights move by about α=0.001\alpha = 0.001 per step, even though their gradients differ by a factor of 1000. SGD would have moved A a thousand times further than B.

The paper writes this per-weight rescaling as a preconditioner, a diagonal matrix that reshapes the gradient before the step:

Mt=diag⁡ ⁣(1v^t+ϵ),θt=θt−1−αMtm^t\boldsymbol{M}_t = \operatorname{diag}\!\Big(\frac{1}{\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon}\Big),\qquad \boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \alpha\boldsymbol{M}_t\hat{\boldsymbol{m}}_tMt=diag⁡ ⁣(1v^t+ϵ),θt=θt−1−αMtm^t\begin{gathered}\boldsymbol{M}_t = \operatorname{diag}\!\Big(\frac{1}{\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon}\Big), \\ \boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \alpha\boldsymbol{M}_t\hat{\boldsymbol{m}}_t\end{gathered}

For plain SGD the preconditioner is the identity. A regularizer's pull toward zero has to be added somewhere in this update, and there are two candidate places: before the division by v^t\sqrt{\hat{\boldsymbol{v}}_t}, or after it.

How AdamW works: where the decay term enters

Adam with L2 (coupled). Add the penalty's gradient to the loss gradient, which is what L2 prescribes. The paper's Algorithm 2, line 6, where the paper uses λ\lambda for the L2 coefficient too:

gt←∇ft(θt−1)+λθt−1\boldsymbol{g}_t \leftarrow \nabla f_t(\boldsymbol{\theta}_{t-1}) + \lambda\boldsymbol{\theta}_{t-1}

The term λθ\lambda\boldsymbol{\theta} is now part of gt\boldsymbol{g}_t, so it enters both moving averages and is divided by v^t+ϵ\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon together with the real gradient. This is what torch.optim.Adam(weight_decay=...) computes.

AdamW (decoupled). Leave the gradient alone and subtract λθ\lambda\boldsymbol{\theta} from the weights in the update itself, where the division cannot reach it. Algorithm 2, line 12:

θt=θt−1−ηt(α m^tv^t+ϵ+λ θt−1)\boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \eta_t\Big(\alpha\,\frac{\hat{\boldsymbol{m}}_t}{\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon} + \lambda\,\boldsymbol{\theta}_{t-1}\Big)θt=θt−1−ηt(α m^tv^t+ϵ+λ θt−1)\begin{aligned}\boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \eta_t\Big(&\alpha\,\frac{\hat{\boldsymbol{m}}_t}{\sqrt{\hat{\boldsymbol{v}}_t}+\epsilon} \\ &+ \lambda\,\boldsymbol{\theta}_{t-1}\Big)\end{aligned}
(4)

ηt\eta_t is a schedule multiplier, a global factor the user can ramp over training (the warm-restart section below uses it). It multiplies both terms. The base learning rate α\alpha multiplies only the Adam step, so λ\lambda keeps the meaning it has in (1): the fraction of each weight removed per step when ηt=1\eta_t = 1. The moments, the bias correction and the defaults (α=0.001\alpha = 0.001, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}) are the same as in Adam. SGDW, the paper's Algorithm 1, makes the same move for SGD with momentum: the decay leaves the momentum buffer and is subtracted from the weights directly.

The figure traces the decay term through both pipelines. Watch the amber packet at the normalization box.

Figure 2 · where λθ joins the update
inside ÷√v̂
The Adam pipeline is the same either way: gradient, moment averages, divide by v^+ϵ\sqrt{\hat{\boldsymbol{v}}}+\epsilon, update θ. Toggle where the decay term λθ joins. In Adam + L2 it enters before the averages and shrinks as it passes through the normalization. In AdamW it joins after the normalization and reaches the update at full size.

Put numbers on one step. Let λ=10−3\lambda = 10^{-3} and ηt=1\eta_t = 1, and take a weight at θ=0.5\theta = 0.5 whose gradients fluctuate with a typical size v^≈0.1\sqrt{\hat{v}} \approx 0.1. AdamW removes λθ=5×10−4\lambda\theta = 5\times10^{-4} from it this step. Under Adam with L2, the same 5×10−45\times10^{-4} is added to every gradient. It shifts m^\hat{m} by 5×10−45\times10^{-4}, barely changes v^\sqrt{\hat{v}}, and after the division moves the weight by about α⋅5×10−4/0.1=5×10−6\alpha \cdot 5\times10^{-4} / 0.1 = 5\times10^{-6}, one percent of what AdamW removes. Weights with very small gradients are the opposite case, covered with Figure 4 below. The listing shows the two variants side by side:

# One Adam-family step (all ops elementwise). Only two lines differ.
g  = grad(f, theta)                    # loss gradient
g  = g + lam * theta                   # Adam+L2 only: decay folded into g  (line 6)
m  = b1 * m + (1 - b1) * g             # 1st moment EMA
v  = b2 * v + (1 - b2) * g * g         # 2nd raw moment EMA
mh = m / (1 - b1**t)                   # bias correction
vh = v / (1 - b2**t)
theta = theta - eta * alpha * mh / (sqrt(vh) + eps)
theta = theta - eta * lam * theta      # AdamW only: decay applied to theta  (line 12)
# defaults: alpha = 1e-3, b1 = 0.9, b2 = 0.999, eps = 1e-8; eta = schedule multiplier

Why no L2 coefficient can match AdamW

For SGD, rescaling the L2 coefficient made the two methods identical. Proposition 2 says that for any optimizer whose preconditioner Mt\boldsymbol{M}_t is not a multiple of the identity, no rescale exists. Write out one step of each. L2 shrinks the weights by αλ′Mtθ\alpha\lambda'\boldsymbol{M}_t\boldsymbol{\theta}, because the penalty gradient is preconditioned along with everything else; decoupled weight decay shrinks them by λθ\lambda\boldsymbol{\theta}. Equal steps for every θ\boldsymbol{\theta} would need

λ θ=αλ′ Mt θfor all θ\lambda\,\boldsymbol{\theta} = \alpha\lambda'\,\boldsymbol{M}_t\,\boldsymbol{\theta}\quad\text{for all }\boldsymbol{\theta}

and a single number λ′\lambda' can satisfy that only if Mt\boldsymbol{M}_t has the same entry for every weight, Mt=kI\boldsymbol{M}_t = k\boldsymbol{I}. SGD's preconditioner is the identity, so (2) works. Adam's entries are 1/(v^i+ϵ)1/(\sqrt{\hat{v}_i}+\epsilon), different for every weight by design.

Per weight, with the same coefficient on both lines of Algorithm 2, L2 delivers α/(v^+ϵ)\alpha/(\sqrt{\hat{v}}+\epsilon) times the decay AdamW delivers in one step. That factor is 1 at v^=α\sqrt{\hat{v}} = \alpha, 0.01 at v^=0.1\sqrt{\hat{v}} = 0.1, and above 1 for weights whose gradients are smaller than α\alpha. The figure plots it across gradient sizes. Its second slider rescales the L2 coefficient, which is the only freedom an L2 implementation has; watch what that does to the slope.

Figure 3 · how much decay each weight gets
√v̂=1.0e-2
λ′=1.0·λ
The fraction of AdamW's decay that Adam + L2 applies in one step, against a weight's typical gradient v^\sqrt{\hat{v}}. AdamW is flat at 1×. Adam + L2 applies kα/(v^+ϵ)k\alpha/(\sqrt{\hat{v}}+\epsilon) for an L2 coefficient λ′=kλ\lambda' = k\lambda; violet shading marks over-decay, teal under-decay. Drag √v̂ to move the marked weight. Drag L2 coef to rescale: the line slides up or down but keeps its slope, so it meets 1× at one gradient size only. At either end of the coefficient slider it misses every weight on the plot.

That factor is a one-step estimate. It ignores the momentum average and the fact that λθ\lambda\theta also enters vv. The next figure drops both simplifications and runs the optimizers. It uses four weights that the loss does not depend on: each receives only random gradient noise with mean zero and size σ\sigma, from 10−410^{-4} to 10−110^{-1}, and each starts at 1. A regularizer should shrink all four toward zero, since nothing in the loss holds them up. Both panels see the same noise; only the placement of λθ\lambda\theta differs. The run uses the Adam defaults, λ=10−3\lambda = 10^{-3} and 3000 steps, so pure decay would leave 0.9993000≈0.050.999^{3000} \approx 0.05.

Figure 4 · four weights, two placements of the decay
step 3000
Each line is one weight, colored by the size σ\sigma of its noise-only gradient; the dashed curve is pure decay (1−λ)t(1-\lambda)^t. The row under each plot reads every weight's value at the current step. Press Play to run 3000 steps from the start, or drag the slider. Switch to plain SGD, with L2 coefficient λ′=λ/α\lambda' = \lambda/\alpha, and the two panels become identical.

In the AdamW panel all four weights follow the dashed curve and end between about 0.04 and 0.09. The spread is a small random walk of the same size for every weight, because Adam's noise step does not depend on σ\sigma. In the Adam + L2 panel the σ=10−1\sigma = 10^{-1} weight ends near 1.09 (the decay had almost no effect on it), the 10−210^{-2} weight near 0.73, the 10−310^{-3} weight, whose gradient size is close to α\alpha, near 0.09, and the 10−410^{-4} weight at 0.002. That last weight falls fastest at the start: its noise is smaller than λθ\lambda\theta, so the penalty dominates vv, the normalized step saturates near α\alpha, and the weight loses close to 0.001 per step until λθ\lambda\theta shrinks to the size of its noise, around θ=0.1\theta = 0.1. Real networks contain both kinds of weight, and one L2 coefficient has to serve all of them.

The paper's experiments show the same effect. Adam with L2 on CIFAR-10 "did not benefit from it at all": its best results with a nonzero L2 coefficient were comparable to its best results with none (§4.2). The same hyperparameter grid with AdamW produced results substantially better than Adam with L2, rivaling SGD and SGDW.

What decoupled decay penalizes instead

Proposition 3 identifies the loss that decoupled decay does regularize, in an idealized case. Hold Adam's preconditioner fixed at diag⁡(s)−1\operatorname{diag}(\boldsymbol{s})^{-1}, with sis_i playing the role of v^i\sqrt{\hat{v}_i}. Then decoupled decay λ\lambda takes the same steps as running the optimizer, with no decay, on a scale-adjusted L2 loss:

fsreg(θ)=f(θ)+λ′2 ∥θ⊙s ∥22,λ′=λαf^{\text{sreg}}(\boldsymbol{\theta}) = f(\boldsymbol{\theta}) + \frac{\lambda'}{2}\,\big\lVert\boldsymbol{\theta}\odot\sqrt{\boldsymbol{s}}\,\big\rVert_2^2,\qquad \lambda'=\frac{\lambda}{\alpha}fsreg(θ)=f(θ)+λ′2 ∥θ⊙s ∥22,λ′=λα\begin{gathered}f^{\text{sreg}}(\boldsymbol{\theta}) = f(\boldsymbol{\theta}) + \frac{\lambda'}{2}\,\big\lVert\boldsymbol{\theta}\odot\sqrt{\boldsymbol{s}}\,\big\rVert_2^2, \\ \lambda'=\frac{\lambda}{\alpha}\end{gathered}
(5)

The check takes three lines. The penalty's gradient is λ′(θ⊙s)\lambda'(\boldsymbol{\theta}\odot\boldsymbol{s}). The preconditioner divides it by s\boldsymbol{s}, leaving λ′θ\lambda'\boldsymbol{\theta}, and the learning rate multiplies it by α\alpha, giving αλ′θ=λθ\alpha\lambda'\boldsymbol{\theta} = \lambda\boldsymbol{\theta}, which is the decoupled decay step. The coefficient is λ′/2=λ/(2α)\lambda'/2 = \lambda/(2\alpha); the proposition's printed statement has λ′/(2α)\lambda'/(2\alpha), which the Provenance panel below addresses.

Because of the s\sqrt{\boldsymbol{s}} inside the norm, a weight with a large sis_i, meaning historically large gradients, is penalized more, in proportion to si\sqrt{s_i}, which is the opposite of what L2 inside Adam does to it. The authors offer a motivation rather than a proof for why that helps: weights with large gradients are directions where small changes move the loss a lot, and leaning on them harder may favor flatter minima, which earlier work (Keskar et al., 2016) linked to better generalization. Proposition 3 establishes only the single-step equivalence under a fixed preconditioner, and the paper notes it does not apply directly to Adam, whose preconditioner changes every step.

A second argument comes from Aitchison (2018), who reads adaptive optimizers as Bayesian filtering: tracking a probability distribution over each weight's best value while the other weights keep changing. The mean of that distribution updates as

μpost=μprior+Σpost g\boldsymbol{\mu}_{\text{post}} = \boldsymbol{\mu}_{\text{prior}} + \boldsymbol{\Sigma}_{\text{post}}\,\boldsymbol{g}

where g\boldsymbol{g} is the gradient of the minibatch log-likelihood, so the posterior uncertainty Σpost\boldsymbol{\Sigma}_{\text{post}} plays the preconditioner's role: larger steps for weights the filter is less sure about. Between steps the weights drift according to a prior,

P(θt+1∣θt)=N((I−A)θt,  Q)P(\boldsymbol{\theta}_{t+1}\mid\boldsymbol{\theta}_t) = \mathcal{N}\big((\boldsymbol{I}-\boldsymbol{A})\boldsymbol{\theta}_t,\;\boldsymbol{Q}\big)

and with A=λI\boldsymbol{A} = \lambda\boldsymbol{I} that prior multiplies the mean by (1−λ)(1-\lambda) every step, which is equation (1). The shrink comes from a prior that is independent of the data and of each weight's uncertainty. An L2 penalty, in this framework, would have to scale with uncertainty. Aitchison's summary, which the paper quotes, is that weight decay rather than L2 regularization "emerges through the straightforward application of Bayesian filtering."

Normalized weight decay

The paper adds two refinements that are separate from the decoupling. The first addresses a dependence on training length. Each update applies another shrink of 1−λ1-\lambda, so a run with more updates accumulates more total decay, and the authors found that the best λ\lambda gets smaller as the number of epochs grows. They reparameterize the decay as

λ=λnormbB T=λnormI,I=TBb\lambda = \lambda_{\text{norm}}\sqrt{\frac{b}{B\,T}} = \frac{\lambda_{\text{norm}}}{\sqrt{I}},\qquad I = \frac{T B}{b}λ=λnormbB T=λnormI,I=TBb\begin{gathered}\lambda = \lambda_{\text{norm}}\sqrt{\frac{b}{B\,T}} = \frac{\lambda_{\text{norm}}}{\sqrt{I}}, \\ I = \frac{T B}{b}\end{gathered}
(6)

where bb is the batch size, BB the number of training points and TT the number of epochs, so II is the total number of weight updates. The raw λ\lambda falls as one over the square root of the update count. λnorm\lambda_{\text{norm}} is the decay you would use if training were a single pass of one batch (I=1I = 1).

A worked case: CIFAR-10 has 50,000 training images. At batch size 128 for 100 epochs, I=100×50,000/128≈39,000I = 100 \times 50{,}000 / 128 \approx 39{,}000, I≈198\sqrt{I} \approx 198, and λnorm=0.05\lambda_{\text{norm}} = 0.05 gives a raw λ≈2.5×10−4\lambda \approx 2.5\times10^{-4}. ImageNet32×32, a downsampled ImageNet with 1.2 million 32×32 images, has epochs 24 times longer, so at the same 100 epochs II is 24 times larger and the raw λ\lambda is 24≈4.9\sqrt{24} \approx 4.9 times smaller. The paper reports that the raw CIFAR-10 value, transferred unchanged, would have been roughly 5 times too large.

Figure 5 · normalized weight decay
I=39k · λ=2.5e-4
The raw λ falls along 1/I1/\sqrt{I} as the run gets longer (more weight updates II), while the normalized λ_norm stays fixed. Drag the budget slider to move along the curve; the two dots mark CIFAR-10 and ImageNet32×32 at 100 epochs, about 4.9× apart in raw λ.

With the normalization, the best λnorm\lambda_{\text{norm}} landed in the same range, about 0.025 to 0.05, for SGDW and AdamW alike, and the best CIFAR-10 values were nearly optimal on ImageNet32×32. The authors say the square-root rule was suggested by their CIFAR-10 runs, checked on ImageNet32×32, and that the experiments "were not very comprehensive"; better scaling rules may exist.

Cosine annealing with warm restarts: AdamWR

The second refinement sets the schedule multiplier ηt\eta_t from (4). It comes from the authors' earlier SGDR paper: within a run, ηt\eta_t falls from 1 to 0 along half a cosine, then a warm restart resets it to 1 and starts a longer run from the current weights.

ηt=0.5+0.5cos⁡ ⁣(π TcurTi)\eta_t = 0.5 + 0.5\cos\!\left(\pi\,\frac{T_{\text{cur}}}{T_i}\right)
(7)

TcurT_{\text{cur}} counts epochs since the last restart and is updated every batch, so it takes fractional values; TiT_i is the length of the current run, and each run is TmultT_{\text{mult}} times longer than the one before. With T0=100T_0 = 100 and Tmult=2T_{\text{mult}} = 2, the runs last 100, 200, 400 and 800 epochs, so ηt\eta_t reaches zero at epochs 100, 300, 700 and 1500, each a natural point to save the model. Because ηt\eta_t multiplies both terms of (4), a restart raises the step size and the decay together. Normalized decay let the authors keep one decay setting across restarts: inside each run, TT in (6) is the length of that run, so one λnorm\lambda_{\text{norm}} served short and long runs alike.

Figure 6 · cosine annealing with warm restarts
T₀=100
T_mult=2
The multiplier η falls from 1 to 0 along a cosine within each run, then jumps back to 1; amber dots mark the η=0\eta = 0 points. Drag T0T_0 to set the first run length and TmultT_{\text{mult}} to set how much each run grows. At Tmult=1T_{\text{mult}} = 1 every run has the same length.

AdamW with this schedule is AdamWR. Its final error matched AdamW's, but it got to good models much sooner: up to 10 times faster on CIFAR-10 and ImageNet32×32, judged at the first restart. The appendix curves also show the restart variants, AdamWR and SGDWR, generalizing better than AdamW and SGDW.

What the experiments show

All experiments use the setup of Gastaldi (2017): 26-layer ResNets with Shake-Shake regularization, the 2×64d variant (11.6M parameters) and the 2×96d variant (25.6M), batch size 128, on CIFAR-10 and ImageNet32×32.

AdamW did not set a new state of the art. The 2.86% CIFAR-10 error that appears in the experimental setup is Gastaldi's result for Shake-Shake trained with SGD, which the authors adopted as their baseline. The paper's claim is that decoupling lets Adam compete with SGD with momentum on image classification, where it had trailed, so that practitioners would not need to switch optimizers per task.

Section 4.5 collects early outside results. On intracranial EEG classification, Völker et al. ran AdamW with the same hyperparameters as Adam and got 73.68% against 71.37% test accuracy on Deep4Net, and 72.04% against 61.34% on a ResNet. Zhang et al. (2018) compared L2 and decoupled decay for SGD, Adam and K-FAC on CIFAR with ResNet and VGG networks and found decoupled decay consistently better wherever the two differ. The authors state the main limitation: their own results are on image classification and "must be verified on a wider range of tasks." They expect the same effect for other adaptive methods such as AdaGrad and AMSGrad but did not test it.

AdamW in PyTorch and in LLM training

In PyTorch, torch.optim.Adam(weight_decay=...) is coupled L2: it adds λθ\lambda\boldsymbol{\theta} to the gradient before updating the moments, as in line 6 of Algorithm 2. torch.optim.AdamW decouples. Recent versions of Adam also accept decoupled_weight_decay=True, documented as equivalent to AdamW. The defaults differ: Adam uses weight_decay=0, AdamW uses 0.01.

import torch
# coupled L2: lam * theta is added to the gradient before m and v
opt = torch.optim.Adam(params, lr=1e-3, weight_decay=1e-2)
# decoupled: theta <- theta - lr * weight_decay * theta, outside sqrt(v_hat)
opt = torch.optim.AdamW(params, lr=1e-3, weight_decay=1e-2)
# per-step shrink here is lr * weight_decay = 1e-5 (times any LR schedule factor)

PyTorch's AdamW decays by γλptθ\gamma\lambda_{\text{pt}}\boldsymbol{\theta}, where γ\gamma is the current learning rate and λpt\lambda_{\text{pt}} the weight_decay argument. The paper's decay in (4) is ηtλθ\eta_t\lambda\boldsymbol{\theta}, with no α\alpha. Since PyTorch's γ\gamma is the paper's ηtα\eta_t\alpha, the two agree when λ=αλpt\lambda = \alpha\lambda_{\text{pt}}. Both keep the decay out of the v^\sqrt{\hat{v}} division, so Proposition 2's inequivalence does not apply to either. PyTorch's form does tie the decay to the base learning rate again: halving lr halves the per-step shrink. The paper's CIFAR-10 example, raw λ=2.5×10−4\lambda = 2.5\times10^{-4} at α=0.001\alpha = 0.001, corresponds to weight_decay=0.25 in PyTorch.

LLaMA (Touvron et al., 2023), as one large-language-model example, reports AdamW with β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95, weight decay 0.1 and a cosine schedule that ends at 10% of the peak learning rate.

Provenance Verified against primary literatureHow we verify
Hanson & Pratt (1988)The multiplicative weight decay of equation (1): a fixed fraction of each weight removed per step, outside the loss. Krogh & Hertz (1992) instead define weight decay as an added L2 cost term, the conflation this paper separates.
Kingma & Ba (2015)Adam: momentum over a gradient divided by the root of a moving average of raw squared gradients. AdamW keeps its moments, bias correction and defaults.
Loshchilov & Hutter (2017), SGDRCosine annealing with warm restarts, equation (7), reused to build AdamWR and SGDWR.
Aitchison (2018)Adaptive optimizers as Bayesian filtering; decoupled decay is the state-transition prior with A = lambda I. The paper cites the arXiv id as 1507.02030; the correct id is 1807.07540.
Gastaldi (2017)Shake-Shake ResNets and the training code all experiments reuse. The 2.86% CIFAR-10 error is this SGD baseline, not an AdamW result.
PyTorch docs (torch.optim)Adam(weight_decay) adds lambda theta to the gradient (coupled L2, default 0); AdamW subtracts lr * weight_decay * theta (default 0.01).
correctionProposition 3's statement prints the scale-adjusted regularizer as λ′/(2α)·‖θ⊙√s‖² with λ′ = λ/α, which counts the 1/α twice: the coefficient becomes λ/(2α²), 1000 times too strong at α = 0.001. The proof in Appendix A uses λ′/2 = λ/(2α), and only that coefficient makes the proof's own derivation lines agree. This page teaches λ′/2.

Questions you might still have

?

Is the weight_decay argument of torch.optim.Adam the same as AdamW?
No. torch.optim.Adam(weight_decay=...) adds the decay to the gradient before the moment averages, so it is divided by sqrt(v-hat): coupled L2. torch.optim.AdamW, or Adam with decoupled_weight_decay=True, subtracts the decay from the weights directly. The defaults differ too: 0 for Adam, 0.01 for AdamW.

?

How do I convert the paper's lambda to PyTorch's weight_decay?
PyTorch multiplies its weight_decay by the learning rate, while the paper's lambda is not multiplied by the base rate alpha. Divide: weight_decay = lambda / alpha. The paper's CIFAR-10 setting (lambda_norm 0.05, 100 epochs, batch 128, so lambda of about 2.5e-4) at lr = 0.001 becomes weight_decay = 0.25.

?

If decoupled decay is better, why did libraries implement L2 for so long?
For plain SGD the two are identical once the coefficient is rescaled by 1/alpha, a fact the paper calls well known, so implementing the loss penalty and naming it weight decay looked harmless. The difference appears only once a per-weight preconditioner divides the gradient, as Adam, RMSProp and AdaGrad do.

?

Does AdamW still need a learning-rate schedule?
The paper recommends one. Because Adam already adapts per-weight step sizes, schedules were less common with it than with SGD, but in Section 4.1 cosine annealing clearly beat both step-drop and a fixed rate, and decoupled decay beat L2 under all three, by more under the better schedules. The schedule multiplier eta_t scales the decay as well as the step.

?

Does AdamW change the model or inference?
No. AdamW is an optimizer; the architecture and the forward pass are unchanged. A model trained with AdamW and one trained with Adam plus L2 run identically at inference. They differ only in the weights training converged to.

?

Why would penalizing large-gradient weights more improve generalization?
The paper offers this as a hypothesis. Weights with large gradients are directions where small changes move the loss a lot; shrinking them harder may steer training toward flatter minima, which earlier work linked to generalization. Proposition 3 proves only a single-step equivalence under a fixed preconditioner. The evidence that it helps is empirical: equal training loss, lower test error.

?

Does AdamW help outside image classification?
The paper's own experiments are image classification only, and the authors say the result needs verifying on other tasks. They cite early outside results on EEG classification and face detection, and AdamW was later used to train large language models such as LLaMA, whose explainer covers that recipe.

Footnotes & further reading

  1. The paper: Loshchilov & Hutter, Decoupled Weight Decay Regularization (University of Freiburg, ICLR 2019). Code.
  2. The base optimizer: Kingma & Ba, Adam: A Method for Stochastic Optimization (2015). Walkthrough: Adam, explained.
  3. The original multiplicative weight decay: Hanson & Pratt, "Comparing biases for minimal network construction with back-propagation" (NeurIPS 1988). Krogh & Hertz, A Simple Weight Decay Can Improve Generalization (1992), defines weight decay as an added L2 cost, an example of the conflation this paper separates.
  4. Cosine annealing with warm restarts: Loshchilov & Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts (2017).
  5. The Bayesian-filtering view: Aitchison, A Unified Theory of Adaptive Stochastic Gradient Descent as Bayesian Filtering (2018). The paper's reference list gives the arXiv id as 1507.02030; the correct id is 1807.07540.
  6. Implementations: PyTorch's torch.optim.AdamW and torch.optim.Adam. The experiments use 26-layer ResNets (2×64d and 2×96d, 11.6M and 25.6M parameters) with Shake-Shake regularization from Gastaldi, Shake-Shake regularization (2017).
  7. LLaMA's optimizer settings: Touvron et al., LLaMA: Open and Efficient Foundation Language Models (2023), Section 2.3. Explainer: LLaMA.