AdamW: Decoupled Weight Decay Regularization
Weight decay and L2 regularization are the same for SGD, but not for Adam.
Deep-learning libraries implemented L2 regularization and called it weight decay. Inside Adam, the L2 penalty is divided by each weight's gradient size, so weights with large gradients get much less of the decay than weights with small ones. AdamW subtracts the decay from the weights directly, after Adam's division.
Explaining the paperDecoupled Weight Decay RegularizationA 2017 paper that moves the weight-decay term in Adam's update out of the gradient (Algorithm 2, line 6) and into the weight update (line 12). On CIFAR-10, Adam with L2 regularization did no better than Adam without it; with the term moved, test error fell 15% (relative) below Adam with L2.
Two techniques for keeping a network's weights small are treated as the same thing so routinely that the libraries merged them. L2 regularization adds a penalty for large weights to the loss. Weight decay multiplies every weight by a number slightly below one on every step. When this paper appeared, the common deep-learning libraries implemented only the first and exposed it as an argument named weight_decay. For plain stochastic gradient descent (SGD) that naming is harmless: the two produce identical updates once one coefficient is rescaled.
Ilya Loshchilov and Frank Hutter show that the equivalence breaks for adaptive optimizers such as Adam, which divide each weight's step by the size of its recent gradients. Fed through that division, the L2 penalty shrinks weights unevenly, and least of all the weights with large gradients. They propose applying the decay the original way, to the weights and outside the division, and call the result AdamW (and SGDW for the SGD-with-momentum version). AdamW is widely used for large models; LLaMA, for example, was trained with it.
The page builds up in this order: what L2 regularization and weight decay each do to a weight, why they coincide for SGD, how Adam rescales steps, where the decay term enters the Adam update, why no L2 coefficient can match decoupled decay, and what decoupled decay penalizes instead. Then the paper's two practical additions (normalized weight decay and warm restarts), the experiments, and how this maps onto PyTorch.
L2 regularization and weight decay
L2 regularization changes the loss. To , the ordinary training loss of the weight vector , it adds a penalty proportional to the squared length of that vector:
The norm is the sum of the squared weights and sets how hard the penalty pushes. Smaller weights usually mean a smoother function that does better on data it has not seen, which is the reason to regularize. Training only ever uses the penalty through its gradient. Differentiating gives , and the one-half in front cancels that 2:
One gradient step with learning rate is then
the ordinary step plus a pull toward zero of size times each weight. The pull is multiplied by the learning rate because it travels inside the gradient.
Weight decay never touches the loss. Hanson and Pratt (1988) described it as an exponential decay of the weights, applied alongside the gradient step:
Every weight loses the same fraction of itself each step, then the gradient step happens as usual. With , a weight at 3.0 loses 0.03 and a weight at 0.2 loses 0.002, one percent each. A weight with no gradient pushing back shrinks geometrically: after 100 steps it keeps of its value. In (1), stands alone and multiplies only the gradient.
Why weight decay and L2 match for SGD
Expand both updates for plain SGD and line them up:
They differ only in the shrink term: against . Setting those equal is the paper's Proposition 1, which the authors call a well-known fact: SGD with weight decay takes exactly the same steps as SGD on the L2-regularized loss when
With a learning rate of 0.1 and a weight decay of , the matching L2 coefficient is , ten times larger, because L2's pull gets multiplied by on the way in and has to start out bigger to end up the same size.
Equation (2) also couples the two hyperparameters. The shrink a weight receives under L2 is , so if you find a good and then halve the learning rate, you have also halved the regularization, and the old is no longer the good one. A grid search over learning rate and L2 coefficient therefore finds its good settings along a diagonal. In the paper's SGD grid (a 26-layer ResNet on CIFAR-10), one of the ten best settings sits at , L2 coefficient (the paper writes it as ): changing either value alone makes the result worse, while changing both together can still improve it. With SGDW, the same search is close to separable. Fixing a badly tuned and tuning only the decay still reaches a good value, .
The figure shows the shape of that search. Look for the orientation of the bright basin as you toggle.
In the coupled panel, a one-at-a-time sweep (fix , tune , then fix , tune ) walks off the diagonal ridge after every move. In the decoupled panel the same sweep converges in one pass. The paper suggests this coupling may be part of why SGD has a reputation for being sensitive to its hyperparameters. For SGD, though, equation (2) is an exact fix in principle. For Adam no rescale works, for the reason the next two sections walk through.
How Adam sizes each weight's step
SGD multiplies every gradient by the same learning rate. Adam keeps two running averages per weight and uses them to give each weight its own step size: , an exponential moving average of the gradient (momentum), and , an exponential moving average of the squared gradient. After dividing each by a bias-correction factor (written , ; the Adam explainer derives it), the update is
with all operations elementwise and outside the square root. is the root-mean-square of a weight's recent gradients. It is a magnitude, not a standard deviation: averages the raw square , not the squared deviation from a mean.
Dividing by it makes the step roughly the same size for every weight. Take two weights whose gradients have been steady: weight A at about 0.1 per step, weight B at about 0.0001. For a steady gradient, approaches the gradient and its absolute value, so the ratio approaches ±1 and both weights move by about per step, even though their gradients differ by a factor of 1000. SGD would have moved A a thousand times further than B.
The paper writes this per-weight rescaling as a preconditioner, a diagonal matrix that reshapes the gradient before the step:
For plain SGD the preconditioner is the identity. A regularizer's pull toward zero has to be added somewhere in this update, and there are two candidate places: before the division by , or after it.
How AdamW works: where the decay term enters
Adam with L2 (coupled). Add the penalty's gradient to the loss gradient, which is what L2 prescribes. The paper's Algorithm 2, line 6, where the paper uses for the L2 coefficient too:
The term is now part of , so it enters both moving averages and is divided by together with the real gradient. This is what torch.optim.Adam(weight_decay=...) computes.
AdamW (decoupled). Leave the gradient alone and subtract from the weights in the update itself, where the division cannot reach it. Algorithm 2, line 12:
is a schedule multiplier, a global factor the user can ramp over training (the warm-restart section below uses it). It multiplies both terms. The base learning rate multiplies only the Adam step, so keeps the meaning it has in (1): the fraction of each weight removed per step when . The moments, the bias correction and the defaults (, , , ) are the same as in Adam. SGDW, the paper's Algorithm 1, makes the same move for SGD with momentum: the decay leaves the momentum buffer and is subtracted from the weights directly.
The figure traces the decay term through both pipelines. Watch the amber packet at the normalization box.
Put numbers on one step. Let and , and take a weight at whose gradients fluctuate with a typical size . AdamW removes from it this step. Under Adam with L2, the same is added to every gradient. It shifts by , barely changes , and after the division moves the weight by about , one percent of what AdamW removes. Weights with very small gradients are the opposite case, covered with Figure 4 below. The listing shows the two variants side by side:
# One Adam-family step (all ops elementwise). Only two lines differ.
g = grad(f, theta) # loss gradient
g = g + lam * theta # Adam+L2 only: decay folded into g (line 6)
m = b1 * m + (1 - b1) * g # 1st moment EMA
v = b2 * v + (1 - b2) * g * g # 2nd raw moment EMA
mh = m / (1 - b1**t) # bias correction
vh = v / (1 - b2**t)
theta = theta - eta * alpha * mh / (sqrt(vh) + eps)
theta = theta - eta * lam * theta # AdamW only: decay applied to theta (line 12)
# defaults: alpha = 1e-3, b1 = 0.9, b2 = 0.999, eps = 1e-8; eta = schedule multiplierWhy no L2 coefficient can match AdamW
For SGD, rescaling the L2 coefficient made the two methods identical. Proposition 2 says that for any optimizer whose preconditioner is not a multiple of the identity, no rescale exists. Write out one step of each. L2 shrinks the weights by , because the penalty gradient is preconditioned along with everything else; decoupled weight decay shrinks them by . Equal steps for every would need
and a single number can satisfy that only if has the same entry for every weight, . SGD's preconditioner is the identity, so (2) works. Adam's entries are , different for every weight by design.
Per weight, with the same coefficient on both lines of Algorithm 2, L2 delivers times the decay AdamW delivers in one step. That factor is 1 at , 0.01 at , and above 1 for weights whose gradients are smaller than . The figure plots it across gradient sizes. Its second slider rescales the L2 coefficient, which is the only freedom an L2 implementation has; watch what that does to the slope.
That factor is a one-step estimate. It ignores the momentum average and the fact that also enters . The next figure drops both simplifications and runs the optimizers. It uses four weights that the loss does not depend on: each receives only random gradient noise with mean zero and size , from to , and each starts at 1. A regularizer should shrink all four toward zero, since nothing in the loss holds them up. Both panels see the same noise; only the placement of differs. The run uses the Adam defaults, and 3000 steps, so pure decay would leave .
In the AdamW panel all four weights follow the dashed curve and end between about 0.04 and 0.09. The spread is a small random walk of the same size for every weight, because Adam's noise step does not depend on . In the Adam + L2 panel the weight ends near 1.09 (the decay had almost no effect on it), the weight near 0.73, the weight, whose gradient size is close to , near 0.09, and the weight at 0.002. That last weight falls fastest at the start: its noise is smaller than , so the penalty dominates , the normalized step saturates near , and the weight loses close to 0.001 per step until shrinks to the size of its noise, around . Real networks contain both kinds of weight, and one L2 coefficient has to serve all of them.
The paper's experiments show the same effect. Adam with L2 on CIFAR-10 "did not benefit from it at all": its best results with a nonzero L2 coefficient were comparable to its best results with none (§4.2). The same hyperparameter grid with AdamW produced results substantially better than Adam with L2, rivaling SGD and SGDW.
What decoupled decay penalizes instead
Proposition 3 identifies the loss that decoupled decay does regularize, in an idealized case. Hold Adam's preconditioner fixed at , with playing the role of . Then decoupled decay takes the same steps as running the optimizer, with no decay, on a scale-adjusted L2 loss:
The check takes three lines. The penalty's gradient is . The preconditioner divides it by , leaving , and the learning rate multiplies it by , giving , which is the decoupled decay step. The coefficient is ; the proposition's printed statement has , which the Provenance panel below addresses.
Because of the inside the norm, a weight with a large , meaning historically large gradients, is penalized more, in proportion to , which is the opposite of what L2 inside Adam does to it. The authors offer a motivation rather than a proof for why that helps: weights with large gradients are directions where small changes move the loss a lot, and leaning on them harder may favor flatter minima, which earlier work (Keskar et al., 2016) linked to better generalization. Proposition 3 establishes only the single-step equivalence under a fixed preconditioner, and the paper notes it does not apply directly to Adam, whose preconditioner changes every step.
A second argument comes from Aitchison (2018), who reads adaptive optimizers as Bayesian filtering: tracking a probability distribution over each weight's best value while the other weights keep changing. The mean of that distribution updates as
where is the gradient of the minibatch log-likelihood, so the posterior uncertainty plays the preconditioner's role: larger steps for weights the filter is less sure about. Between steps the weights drift according to a prior,
and with that prior multiplies the mean by every step, which is equation (1). The shrink comes from a prior that is independent of the data and of each weight's uncertainty. An L2 penalty, in this framework, would have to scale with uncertainty. Aitchison's summary, which the paper quotes, is that weight decay rather than L2 regularization "emerges through the straightforward application of Bayesian filtering."
Normalized weight decay
The paper adds two refinements that are separate from the decoupling. The first addresses a dependence on training length. Each update applies another shrink of , so a run with more updates accumulates more total decay, and the authors found that the best gets smaller as the number of epochs grows. They reparameterize the decay as
where is the batch size, the number of training points and the number of epochs, so is the total number of weight updates. The raw falls as one over the square root of the update count. is the decay you would use if training were a single pass of one batch ().
A worked case: CIFAR-10 has 50,000 training images. At batch size 128 for 100 epochs, , , and gives a raw . ImageNet32×32, a downsampled ImageNet with 1.2 million 32×32 images, has epochs 24 times longer, so at the same 100 epochs is 24 times larger and the raw is times smaller. The paper reports that the raw CIFAR-10 value, transferred unchanged, would have been roughly 5 times too large.
With the normalization, the best landed in the same range, about 0.025 to 0.05, for SGDW and AdamW alike, and the best CIFAR-10 values were nearly optimal on ImageNet32×32. The authors say the square-root rule was suggested by their CIFAR-10 runs, checked on ImageNet32×32, and that the experiments "were not very comprehensive"; better scaling rules may exist.
Cosine annealing with warm restarts: AdamWR
The second refinement sets the schedule multiplier from (4). It comes from the authors' earlier SGDR paper: within a run, falls from 1 to 0 along half a cosine, then a warm restart resets it to 1 and starts a longer run from the current weights.
counts epochs since the last restart and is updated every batch, so it takes fractional values; is the length of the current run, and each run is times longer than the one before. With and , the runs last 100, 200, 400 and 800 epochs, so reaches zero at epochs 100, 300, 700 and 1500, each a natural point to save the model. Because multiplies both terms of (4), a restart raises the step size and the decay together. Normalized decay let the authors keep one decay setting across restarts: inside each run, in (6) is the length of that run, so one served short and long runs alike.
AdamW with this schedule is AdamWR. Its final error matched AdamW's, but it got to good models much sooner: up to 10 times faster on CIFAR-10 and ImageNet32×32, judged at the first restart. The appendix curves also show the restart variants, AdamWR and SGDWR, generalizing better than AdamW and SGDW.
What the experiments show
All experiments use the setup of Gastaldi (2017): 26-layer ResNets with Shake-Shake regularization, the 2×64d variant (11.6M parameters) and the 2×96d variant (25.6M), batch size 128, on CIFAR-10 and ImageNet32×32.
- Schedules. On 100-epoch runs with the 2×64d network, cosine annealing beat step-drop, which beat a fixed learning rate. Decoupled decay beat L2 under all three schedules, and the gap was larger for the better schedules.
- Separability. The best hyperparameter regions for SGDW and AdamW were closer to axis-aligned than for SGD and Adam with L2 (Figure 1's point, on real grids).
- Generalization. On 1800-epoch runs at , comparing 12 L2 settings for Adam against 7 normalized decay settings for AdamW, the learning curves often coincided for the first half of training before AdamW pulled ahead. At equal training loss AdamW still had lower test error, so the gain is in generalization as well as in optimization.
- Headline. At the default , AdamW reduced test error by 15% relative to Adam with L2, on both CIFAR-10 and ImageNet32×32. On an illustrative 6% error that would be about 5.1%. With AdamWR, the improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and gave comparable performance on ImageNet32×32.
AdamW did not set a new state of the art. The 2.86% CIFAR-10 error that appears in the experimental setup is Gastaldi's result for Shake-Shake trained with SGD, which the authors adopted as their baseline. The paper's claim is that decoupling lets Adam compete with SGD with momentum on image classification, where it had trailed, so that practitioners would not need to switch optimizers per task.
Section 4.5 collects early outside results. On intracranial EEG classification, Völker et al. ran AdamW with the same hyperparameters as Adam and got 73.68% against 71.37% test accuracy on Deep4Net, and 72.04% against 61.34% on a ResNet. Zhang et al. (2018) compared L2 and decoupled decay for SGD, Adam and K-FAC on CIFAR with ResNet and VGG networks and found decoupled decay consistently better wherever the two differ. The authors state the main limitation: their own results are on image classification and "must be verified on a wider range of tasks." They expect the same effect for other adaptive methods such as AdaGrad and AMSGrad but did not test it.
AdamW in PyTorch and in LLM training
In PyTorch, torch.optim.Adam(weight_decay=...) is coupled L2: it adds to the gradient before updating the moments, as in line 6 of Algorithm 2. torch.optim.AdamW decouples. Recent versions of Adam also accept decoupled_weight_decay=True, documented as equivalent to AdamW. The defaults differ: Adam uses weight_decay=0, AdamW uses 0.01.
import torch
# coupled L2: lam * theta is added to the gradient before m and v
opt = torch.optim.Adam(params, lr=1e-3, weight_decay=1e-2)
# decoupled: theta <- theta - lr * weight_decay * theta, outside sqrt(v_hat)
opt = torch.optim.AdamW(params, lr=1e-3, weight_decay=1e-2)
# per-step shrink here is lr * weight_decay = 1e-5 (times any LR schedule factor)PyTorch's AdamW decays by , where is the current learning rate and the weight_decay argument. The paper's decay in (4) is , with no . Since PyTorch's is the paper's , the two agree when . Both keep the decay out of the division, so Proposition 2's inequivalence does not apply to either. PyTorch's form does tie the decay to the base learning rate again: halving lr halves the per-step shrink. The paper's CIFAR-10 example, raw at , corresponds to weight_decay=0.25 in PyTorch.
LLaMA (Touvron et al., 2023), as one large-language-model example, reports AdamW with , , weight decay 0.1 and a cosine schedule that ends at 10% of the peak learning rate.
Questions you might still have
Is the weight_decay argument of torch.optim.Adam the same as AdamW?
No. torch.optim.Adam(weight_decay=...) adds the decay to the gradient before the moment averages, so it is divided by sqrt(v-hat): coupled L2. torch.optim.AdamW, or Adam with decoupled_weight_decay=True, subtracts the decay from the weights directly. The defaults differ too: 0 for Adam, 0.01 for AdamW.
How do I convert the paper's lambda to PyTorch's weight_decay?
PyTorch multiplies its weight_decay by the learning rate, while the paper's lambda is not multiplied by the base rate alpha. Divide: weight_decay = lambda / alpha. The paper's CIFAR-10 setting (lambda_norm 0.05, 100 epochs, batch 128, so lambda of about 2.5e-4) at lr = 0.001 becomes weight_decay = 0.25.
If decoupled decay is better, why did libraries implement L2 for so long?
For plain SGD the two are identical once the coefficient is rescaled by 1/alpha, a fact the paper calls well known, so implementing the loss penalty and naming it weight decay looked harmless. The difference appears only once a per-weight preconditioner divides the gradient, as Adam, RMSProp and AdaGrad do.
Does AdamW still need a learning-rate schedule?
The paper recommends one. Because Adam already adapts per-weight step sizes, schedules were less common with it than with SGD, but in Section 4.1 cosine annealing clearly beat both step-drop and a fixed rate, and decoupled decay beat L2 under all three, by more under the better schedules. The schedule multiplier eta_t scales the decay as well as the step.
Does AdamW change the model or inference?
No. AdamW is an optimizer; the architecture and the forward pass are unchanged. A model trained with AdamW and one trained with Adam plus L2 run identically at inference. They differ only in the weights training converged to.
Why would penalizing large-gradient weights more improve generalization?
The paper offers this as a hypothesis. Weights with large gradients are directions where small changes move the loss a lot; shrinking them harder may steer training toward flatter minima, which earlier work linked to generalization. Proposition 3 proves only a single-step equivalence under a fixed preconditioner. The evidence that it helps is empirical: equal training loss, lower test error.
Does AdamW help outside image classification?
The paper's own experiments are image classification only, and the authors say the result needs verifying on other tasks. They cite early outside results on EEG classification and face detection, and AdamW was later used to train large language models such as LLaMA, whose explainer covers that recipe.
Footnotes & further reading
- The paper: Loshchilov & Hutter, Decoupled Weight Decay Regularization (University of Freiburg, ICLR 2019). Code.
- The base optimizer: Kingma & Ba, Adam: A Method for Stochastic Optimization (2015). Walkthrough: Adam, explained.
- The original multiplicative weight decay: Hanson & Pratt, "Comparing biases for minimal network construction with back-propagation" (NeurIPS 1988). Krogh & Hertz, A Simple Weight Decay Can Improve Generalization (1992), defines weight decay as an added L2 cost, an example of the conflation this paper separates.
- Cosine annealing with warm restarts: Loshchilov & Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts (2017).
- The Bayesian-filtering view: Aitchison, A Unified Theory of Adaptive Stochastic Gradient Descent as Bayesian Filtering (2018). The paper's reference list gives the arXiv id as 1507.02030; the correct id is 1807.07540.
- Implementations: PyTorch's torch.optim.AdamW and torch.optim.Adam. The experiments use 26-layer ResNets (2×64d and 2×96d, 11.6M and 25.6M parameters) with Shake-Shake regularization from Gastaldi, Shake-Shake regularization (2017).
- LLaMA's optimizer settings: Touvron et al., LLaMA: Open and Efficient Foundation Language Models (2023), Section 2.3. Explainer: LLaMA.
How could this explainer be improved? Found an error, or something unclear? I read every message.