Layer Normalization

Normalize each example across its own features, not across the batch.

Batch normalization standardizes each neuron's input with a mean and variance taken over the mini-batch. Layer normalization takes the mean and variance over all the neurons of one layer, for one example at a time, so it works at batch size 1, computes the same thing at training and test time, and applies unchanged at every step of a recurrent network.

Jimmy Ba, Jamie Kiros and Geoffrey Hinton posted it in July 2016 as a fix for recurrent networks. Eleven months later the Transformer put a layer norm after every sub-layer.

A layer norm takes a vector, subtracts its mean, divides by its standard deviation, and then multiplies by a learned gain and adds a learned bias, one of each per element. PyTorch ships it as nn.LayerNorm. BERT, GPT-2 and GPT-3 use it, and LLaMA and Mistral use a simplified version called RMSNorm. The 2016 paper that introduced it defines the operation in one equation, derives a table of transformations the operation ignores, analyses how normalization changes the effective learning rate, and tests it on six tasks, most of them recurrent networks.

The method is a response to batch normalization, which a year earlier had made deep feedforward networks train much faster. Batch norm computes its statistics over the training cases in a mini-batch. Layer norm computes the same statistics over the units of a layer instead, which the paper calls a transpose of batch norm. The page follows that change through: what batch norm needs and where it breaks, the layer-norm formula with a worked example, the paper's invariance table (which you can test in a figure), what normalization does to the learning rate, why recurrent networks gain the most, the experiments, and what Transformers and RMSNorm later changed.

What batch normalization needs

Start with an ordinary layer. Neuron $i$ takes the previous layer's output $\mathbf{h}$, forms a summed input (its pre-activation), and applies a nonlinearity $f$ such as tanh or ReLU:

Both normalization methods in this paper operate on the summed inputs $a_i$, before $f$. Batch normalization rescales each one using that neuron's own mean and standard deviation over the data:

The expectations are over the whole training distribution, which no training step can afford to compute, so in practice $\mu_i$ and $\sigma_i$ are estimated from the current mini-batch, and $g_i$ is a learned gain. Each neuron gets one mean and one standard deviation per step, computed from the $N$ cases in the batch.

With a small batch the estimate is poor. Take a batch of two cases in which neuron $i$ receives $0.3$ and $1.1$. The batch mean is $0.7$, both cases sit $0.4$ from it, so the (biased) standard deviation is $0.4$ and the normalized values are exactly $-1$ and $+1$. Any two different values give $-1$ and $+1$, so at $N=2$ the layer passes on only which of the two cases was larger. At $N=1$ the variance is zero and the formula divides zero by zero.

The paper names three settings where this dependence on the batch causes problems:

• Small batches. Online learning (one case at a time) has no batch, and the paper names very large distributed models whose mini-batches have to be small.
• Training versus test. A single test example has no batch to average over, so batch norm stores running averages of $\mu_i$ and $\sigma_i$ during training and uses those at test time. The two phases compute different functions.
• Recurrent networks. An RNN applies the same weights at every time-step, but the summed inputs at step 3 and step 30 have different statistics. Applied in the obvious way, batch norm keeps separate statistics for every time-step, and a test sequence longer than every training sequence reaches steps that have no statistics at all.

How layer normalization works

Lay out a layer's summed inputs for one mini-batch as a grid with one row per neuron ($H$ rows) and one column per training case ($N$ columns). Batch normalization standardizes each row: one neuron, its values across the batch. Layer normalization standardizes each column: one case, its values across all $H$ neurons of the layer. The statistics become

Every neuron in the layer shares this one $\mu$ and $\sigma$, and every case gets its own pair. No other example enters the computation, which settles all three problems at once. A batch of one is enough. Training and test run the same arithmetic, because there are no batch statistics to estimate during training and freeze for later. And an RNN can apply the same normalization at step 3, step 30 or step 3,000, because each step's statistics come from that step's own summed inputs.

Batch norm is like grading each exam question on a curve against the whole class: one adjustment per question, pooled over students. Layer norm curves each student against the spread of that student's own scores: one adjustment per student, pooled over questions. The analogy holds for the axis and stops there; it says nothing about why either choice helps training. In code the two differ only in the axis of the reduction:

# A: summed inputs for a mini-batch, shape [N cases, H units]
# batch norm: one mean/std per UNIT, taken down the batch     (axis 0)
bn = (A - A.mean(0)) / A.std(0)                    # needs N >= 2
# layer norm: one mean/std per CASE, taken across its units   (axis 1)
ln = (A - A.mean(1, keepdims=True)) / A.std(1, keepdims=True)  # any N

Figure 1 draws the grid. Switch between the two modes and drag the batch size down to 1. The outlined group in layer-norm mode is a column of $H = 6$ values whatever $N$ is. In batch-norm mode the group is a row of $N$ values, and at $N = 1$ it holds a single number with zero variance.

Gain, bias and a worked example

Standardizing forces every layer's summed inputs to mean 0 and variance 1. A layer whose next nonlinearity works best with inputs centered at 2, or spread over a range of 5, would lose that. So, as in batch norm, each unit gets a learned gain $g_i$ and bias $b_i$, applied after standardizing and before the nonlinearity. The paper writes the whole thing in vector form for a recurrent layer, whose summed inputs at step $t$ combine the previous hidden state and the current input:

The $\odot$ is elementwise multiplication and $\mu^t, \sigma^t$ are Eq. (3) applied to $\mathbf{a}^t$. The gain and bias put back the two degrees of freedom that standardizing removed, per unit, so the layer can still represent any scale and offset it needs. They are initialized to $\mathbf{g} = 1$ and $\mathbf{b} = 0$, so a freshly initialized layer norm is a pure standardizer. (The appendix states this initialization with the names of the two parameters swapped, in a way that would erase the input. Section 6 and every implementation use gain 1, bias 0; the Provenance panel below has the details.)

The paper also writes all three methods it compares in one template:

The methods differ only in where $\mu_i$ and $\sigma_i$ come from: batch norm takes them per unit over the batch, layer norm per case over the units, and weight normalization (Salimans and Kingma, 2016) sets $\mu_i = 0$ and $\sigma_i = \lVert \mathbf{w}_i \rVert_2$, the length of the neuron's incoming weight vector, which involves no data.

A concrete pass, with $H = 4$. One case's summed inputs are $\mathbf{a} = [\,2.0,\ -1.0,\ 0.5,\ 1.5\,]$. The mean is $\mu = 3/4 = 0.75$. The deviations are $[\,1.25,\ -1.75,\ -0.25,\ 0.75\,]$, their squares average to $5.25/4 = 1.3125$, and $\sigma = \sqrt{1.3125} \approx 1.146$. Dividing gives

which has mean 0 and variance 1. At the initial $g = 1, b = 0$ this vector goes straight into $f$. If training later sets the first unit's gain to 2 and its bias to 0.5, that unit outputs $f(2 \times 1.09 + 0.5) = f(2.68)$. The forward pass for one case:

# layer norm on one case's summed inputs a, shape [H]
mu    = a.mean()                    # average over the H units
var   = ((a - mu) ** 2).mean()      # biased: divide by H, not H - 1
a_hat = (a - mu) / sqrt(var + eps)  # mean 0, variance 1 (eps: code only)
h     = f(g * a_hat + b)            # per-unit gain g and bias b, then f
# parameters: g, b (H each), init g = 1, b = 0. mu and var are not
# parameters; autograd differentiates through them like any other op.

The variance in the listing divides by $H$, the biased estimator, as Eq. (3) does. The listing also has an $\epsilon$ inside the square root that the paper's equations do not; every implementation adds a small one so that a case whose $H$ inputs are all equal does not divide by zero. PyTorch's module does the same thing on the last dimension of a tensor and stores the gain and bias as weight and bias:

import torch, torch.nn as nn
ln = nn.LayerNorm(512)            # weight = ones(512), bias = zeros(512)
x  = torch.randn(1, 512)          # a batch of ONE case is fine
y  = ln(x)                        # stats over the last dim: 512 values
print(y.mean().item(), y.std(unbiased=False).item())   # ~0.0, ~1.0
print(sum(p.numel() for p in ln.parameters()))         # 1024 = 2 * 512

Figure 2 runs the standardize-then-restore steps on one 8-unit case. At the default gain and bias the bottom row matches the middle row. Moving the gain changes the output's spread and moving the bias shifts its center, while the standardized middle row stays fixed, because it depends only on $\mathbf{a}$.

What layer norm is invariant to

Section 5.1 of the paper asks, for each method, which changes to the weights or the data leave the normalized output exactly the same. If a change has no effect on the output, the loss is flat along it, and training cannot be disrupted by drift in that direction. Table 1 of the paper:

“Weight matrix” means all of the layer's incoming weights $W$ at once, “weight vector” means one row $\mathbf{w}_i$ (the incoming weights of a single neuron), “dataset” means every training case, and “single case” means one training case alone. Figure 3 computes every cell of the table. A toy layer maps 4 training cases through a $5 \times 4$ weight matrix, and the same summed inputs are normalized three ways. Pick a transformation and a strength, and each row reports how far its 20 outputs moved. At the default, scaling case 1 by $\delta = 1.93$ moves the batch-norm outputs by up to 0.40 and the weight-norm outputs by up to 1.18, and leaves the layer-norm outputs unchanged.

Two columns are layer norm's alone. The first is re-scaling a single case. Multiply one input $\mathbf{x}$ by $\delta > 0$ and every summed input of that case, $a_i = \mathbf{w}_i^\top \mathbf{x}$, is multiplied by $\delta$. The mean of those summed inputs is multiplied by $\delta$, so is their standard deviation, and the factor cancels in the ratio:

With the worked example, tripling the input gives $\mathbf{a} = [\,6,\ -3,\ 1.5,\ 4.5\,]$, $\mu = 2.25$ and $\sigma \approx 3.437$, and the same $\hat{\mathbf{a}}$ as before. Batch norm has no such property: its statistics pool the whole batch, so scaling one case changes the mean and variance that every case in the batch is normalized with. Layer norm behaves like a camera that sets the exposure separately for each photo: a dim shot and a bright copy of the same scene come out the same. (The analogy breaks for a negative $\delta$, which flips the sign of every $\hat a_i$.) Figure 4 shows the cancellation on one case.

The second is re-centering the weight matrix: add the same vector $\boldsymbol{\gamma}$ to every row of $W$, and optionally scale the matrix by $\delta$, so $W' = \delta W + \mathbf{1}\boldsymbol{\gamma}^\top$. For a case $\mathbf{x}$, the added term contributes the same scalar $\boldsymbol{\gamma}^\top\mathbf{x}$ to every unit's summed input. That raises the layer mean by exactly $\boldsymbol{\gamma}^\top\mathbf{x}$, the subtraction removes it, the deviations from the mean do not change, and $\delta$ cancels against $\sigma$ as before:

Batch norm fails this one because $\boldsymbol{\gamma}^\top\mathbf{x}$ differs from case to case, so it changes the spread of each unit across the batch (at the default strength in Figure 3, by up to 1.64).

The table also has a “no” where batch norm and weight norm have “yes”: layer norm is invariant to scaling the whole matrix, but not one weight vector. Doubling only $\mathbf{w}_1$ doubles only $a_1$, which changes the $\mu$ and $\sigma$ that all $H$ units share, so every output of the layer moves (by up to 0.54 in Figure 3 at $\delta = 1.93$). Batch norm and weight norm compute each unit's statistics from that unit alone, so scaling its weights cancels. Likewise, shifting every input by a constant vector $\mathbf{c}$ adds $\mathbf{w}_i^\top\mathbf{c}$ to unit $i$, a different amount for each unit, which layer norm's single mean cannot remove; batch norm removes it because the shift is the same for every case. (Section 5.1's prose says all three methods are invariant to re-scaling the dataset, which contradicts the paper's own table for weight norm. The table is right, and Figure 3 shows weight norm's outputs scaling with the data.)

Why the weight norm acts as a learning rate

Invariance is a statement about outputs. Section 5.2 asks what normalization does to learning, when two settings of the weights compute the same function but sit at different points of parameter space. It borrows a measure of step size from Amari's natural-gradient work (1998): instead of the Euclidean length of a parameter change $\delta$, measure how much it changes the model's predictions, using the KL divergence (a standard measure of how different two probability distributions are). To second order that is a quadratic form in the Fisher information matrix $F$:

$F$ is the curvature of that KL distance in parameter coordinates: a large entry means a small step in that direction changes the predictions a lot. (The same matrix bounds the policy step in TRPO.) The paper computes $F$ for a normalized generalized linear model, meaning one linear layer, an elementwise nonlinearity, and an exponential-family output distribution. It extends the result to deep networks only through a block-diagonal approximation with one block per neuron. Two conclusions follow.

The first is an implicit learning-rate decay. The Fisher block along $\mathbf{w}_i$ is multiplied by the gain and divided by the normalization scalar $\sigma_i$, so it shrinks as $\sigma_i$ grows. If the norm of $\mathbf{w}_i$ doubles, the output stays the same, but $\sigma_i$ doubles, and the paper states that the curvature along $\mathbf{w}_i$ changes by a factor of $\tfrac{1}{2}$. The same gradient step then changes the function less, so a weight vector with a large norm is harder to rotate, which the paper calls an implicit “early stopping” effect. Read the factor of one half as an illustration. It assumes $\sigma_i$ grows in exact proportion to $\lVert\mathbf{w}_i\rVert$ with the rest of the layer fixed. That holds for weight norm, where $\sigma_i$ is the norm, and for batch norm, where $\sigma_i$ belongs to unit $i$ alone. It holds only partly for layer norm, whose $\sigma$ pools all $H$ units.

For layer norm an exact version follows from the first column of Table 1. The output does not change when the whole matrix is multiplied by $\delta > 0$, so any loss computed from it satisfies $L(\delta W) = L(W)$. Differentiating that identity gives two facts. Differentiating with respect to $W$ gives $\nabla L(\delta W) = \nabla L(W)/\delta$: the gradient shrinks in proportion to the norm. Differentiating with respect to $\delta$ at $\delta = 1$ gives $\langle \nabla L(W), W \rangle = 0$: the gradient is perpendicular to $W$. A plain gradient step $W' = W - \eta\nabla L$ is therefore perpendicular to $W$, and by Pythagoras

so the norm grows at every step. The angle a step turns $W$ is about $\eta \lVert\nabla L\rVert / \lVert W\rVert$, and with the gradient itself scaling as $1/\lVert W\rVert$, the direction effectively learns at rate $\eta / \lVert W \rVert^2$, which falls as the norm grows. Salimans and Kingma derive the same norm growth for weight normalization (Section 2.1 of their paper) and observe that a learning rate that is too large inflates the norm until the effective rate is reasonable. They also say the argument is exact only for plain gradient descent; momentum and Adam (every layer-norm experiment that names an optimizer used Adam) follow it only qualitatively.

Figure 5 runs this on the smallest example that has it: a 2-D weight vector and a loss that depends only on the vector's angle to a target direction, so it is invariant to scaling just like layer norm's loss is to scaling $W$. The learning rate is fixed at 0.3 and every run starts 140° from the target. Watch the norm readout and the angle turned per step as you change the starting norm.

Starting at norm 1, the vector is 9.4° from the target after 15 steps, with its norm grown to 1.24. Starting at norm 3, it is still 42.6° away after all 60 steps, because its effective rate is about nine times smaller. Starting at norm 0.1, the first step is so large relative to the vector that it turns it 87° and inflates the norm to 1.93 in one step, after which the run is slower than one started at norm 0.25 (18° away at step 15, against 0.2°).

The second conclusion concerns the gain. Under the original parameterization, the paper shows, the metric along the magnitude of a neuron's incoming weights scales with the norm of the input; for the normalized model, learning the gain $g_i$ depends only on the magnitude of the prediction error. So learning a neuron's output scale is more robust to the scale of its input. This result is derived inside the generalized-linear-model setting and is not an exact statement about a deep stack of nonlinear layers.

Layer norm in RNNs and LSTMs

A recurrent network applies the same recurrent matrix to its hidden state at every step. If that tends to multiply the size of the summed inputs by a factor $\rho$ slightly above 1, the size grows like $\rho^t$: at $\rho = 1.15$, a factor of 268 after 40 steps. Slightly below 1, it decays toward zero or, when each step adds fresh input, toward the level that input sustains. The paper links this drift in the summed inputs to exploding and vanishing gradients. That link is heuristic. Gradient explosion is governed by the products of the recurrent Jacobians (for a linear RNN, the singular values of $W_{hh}$), and normalizing the forward pass controls the forward scale directly and the gradients only indirectly.

Eq. (4) re-standardizes the summed inputs at every step, so the input to the nonlinearity has the same size whatever size the previous step produced: with gain 1 and bias 0 its root-mean-square is exactly 1. In the paper's words, the network becomes invariant to re-scaling all of the summed inputs to a layer, which gives “much more stable hidden-to-hidden dynamics”. One gain vector and one bias vector are shared by all time-steps, and the statistics at step $t$ depend only on $\mathbf{a}^t$, so nothing is stored per step and a test sequence of any length is handled the same way. Recurrent batch normalization (Cooijmans et al., 2016) keeps separate statistics per step instead.

Figure 6 simulates a 24-unit linear recurrent layer:

The matrix $Q$ is a fixed random rotation (so $\rho$ is exactly the per-step factor) and $\mathbf{u}^t$ is a fixed random input of RMS 1 scaled by the input drive $s$. Drag $\rho$ across 1 and the input drive to 0. Without layer norm the state's RMS climbs off the top or sinks toward $s/\sqrt{1-\rho^2}$, and to the floor when $s = 0$. With layer norm the state sits at 1. The dashed line shows the summed input arriving at the normalization each step, which stays near $\sqrt{\rho^2 + s^2}$ (one step's growth of a unit-size state plus one step's input) instead of compounding.

For LSTMs, which most of the experiments use, the appendix normalizes the recurrent and input projections separately, each with its own gain $\boldsymbol{\alpha}$ and bias $\boldsymbol{\beta}$, and adds a third layer norm on the cell state before the output tanh:

In Eqs. (20)–(22), $\sigma$ is the logistic sigmoid, not a standard deviation. Normalizing the two projections separately, instead of their sum, standardizes each before they are added, so a projection with large raw values cannot dominate the sum: after normalization each contributes at the scale set by its own gain. The appendix defines the notation as

with $\mu$ and $\sigma$ as in Eq. (3), so $\boldsymbol{\alpha}$ is the gain and starts at 1, and $\boldsymbol{\beta}$ is the bias and starts at 0. The appendix gives a matching GRU variant with four layer norms (Eqs. 26–28), and for the DRAW experiment applies a single layer norm to the LSTM cell state only (Eq. 31).

What the experiments show

The paper tests layer norm on six tasks, five of them recurrent. Unless noted, the gains start at 1 and the biases at 0. Layer norm trains faster on every task. On four the paper also reports a better final result, on handwriting a comparable one, and on MNIST it reports robustness to batch size and faster convergence.

• Image–sentence ranking (order embeddings on MSCOCO, layer norm added to the GRU sentence encoder). Layer norm reaches its best validation model in 60% of the time the baseline takes. Averaged over 5 test splits, caption retrieval R@1 rises from 46.6 to 48.5 and image retrieval R@1 from 37.8 to 38.9 (R@1 is the fraction of queries whose correct match is ranked first).
• Question answering (a unidirectional attentive reader on the CNN corpus). Layer norm trains faster and reaches a better validation result than both the baseline and recurrent batch norm. Recurrent batch norm needed its gain initialized to 0.1; layer norm was tried at 1.0 and 0.1 and did significantly better at 1.0.
• Skip-thought sentence vectors (a 2400-dimensional sentence encoder trained on BookCorpus). After 1M iterations, layer norm improves every downstream score in Table 3 over the authors' own baseline, for example MR accuracy 77.3 to 79.5 and SICK relatedness r 0.842 to 0.854. Training the layer-norm model for a month improved all but one task further.
• Generative modeling (DRAW, 64 glimpses and 256 LSTM units, on binarized MNIST). Layer norm converges almost twice as fast. After 200 epochs the test variational bound is 82.09 nats against the baseline's 82.36; this bounds the negative log-likelihood, so lower is better.
• Handwriting generation (IAM-OnDB, three layers of 400 LSTM cells, about 3.7M weights). Lines average around 700 steps and the mini-batch is 8, the combination where stable dynamics matter most. Layer norm reaches a comparable log-likelihood to the baseline, much faster.
• Permutation-invariant MNIST (a 784-1000-1000-10 feedforward net). Layer norm stays robust as the batch shrinks from 128 to 4 and converges faster than batch norm applied to all layers. (At small batch sizes the batch norm baseline used the unbiased variance estimator.) Layer norm is applied to the hidden layers only, not the softmax layer, because the scale of the logits sets the prediction's confidence and layer norm would remove it.

Convolutional networks are the exception, reported from preliminary experiments: layer norm beats no normalization but batch norm beats layer norm. The authors' explanation is that in a fully connected layer the units tend to contribute similarly to the prediction, so one shared mean and variance fits them, while in a convolutional layer the many units whose receptive fields lie near the image boundary are rarely on and have very different statistics. They write that “further research is needed” to make layer norm work well in ConvNets.

The paper motivates normalization, as the batch norm paper did, by “covariate shift”: the inputs to a layer keep changing distribution as the layers below it learn, and fixing their mean and variance should reduce that. Santurkar et al. (2018) tested the explanation for batch norm. They injected noise with non-zero mean and non-unit variance after every batch-norm layer, found almost no difference in training, and argued that batch norm helps by making the loss landscape smoother. Their experiments are on batch norm only. The layer-norm paper proves the invariance table and measures faster training, mostly in recurrent networks; neither result depends on the covariate-shift explanation.

From 2016 layer norm to Transformers and RMSNorm

The Transformer (June 2017) made layer norm standard. It applies a layer norm after every residual addition, $\mathrm{LN}(x + \mathrm{Sublayer}(x))$, an arrangement later called Post-LN. Moving the norm inside each residual branch instead, $x + \mathrm{Sublayer}(\mathrm{LN}(x))$ (Pre-LN), trains more stably; Xiong et al. (2020) analysed it and showed its gradients are well-behaved at initialization, while calling it “recently proposed”, so the practice predates their paper. In both arrangements the vector being normalized is a token's residual-stream vector of width $d_{\text{model}}$ (512 in the original Transformer), and the normalization is over that vector, one token at a time.

The 2016 paper applies it at a different spot. There, layer norm standardizes a layer's pre-activation summed inputs and the nonlinearity follows, $\mathbf{h} = f(\mathrm{LN}(\mathbf{a}))$. In a Transformer it standardizes an activation vector with no nonlinearity right after it, using the same statistics.

RMSNorm (Zhang and Sennrich, 2019) drops the mean subtraction and usually the bias, and divides by the root mean square $\sqrt{\tfrac{1}{H}\sum_i a_i^2}$ alone. Its authors hypothesized that the re-centering invariance is dispensable, and LLaMA and Mistral use RMSNorm. Without the mean subtraction, RMSNorm loses one of layer norm's invariances: re-centering the weight matrix (Eq. 6) is no longer cancelled. Figure 7 isolates that difference. Add the same offset to every component of a vector and compare the two outputs.