VerifiedarXiv:1502.0185234 min
Training · Initialization

He Initialization and PReLU: Delving Deep into Rectifiers

A ReLU zeroes the negative half of its input, so weights with variance 1/n, where n is the number of inputs to each unit, halve the signal's variance at every layer. Variance 2/n keeps the variance level, and a 30-layer network trains from scratch.

He, Zhang, Ren and Sun derive the 2/n initialization from the variance of each layer's output, give the ReLU a learned slope for negative inputs (PReLU), and combine the two into ImageNet models with 4.94% top-5 test error.

Explaining the paperDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationHe, Zhang, Ren, Sun · ICCV 2015 · arXiv:1502.01852 ↗

A February 2015 paper from Microsoft Research. Its initialization rule ships in every major framework as “He” or “Kaiming” initialization, and its 4.94% error was the first published ImageNet result below the 5.1% measured for a trained human annotator.

Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun posted the paper on 6 February 2015. At the time the best ImageNet classifiers were 16 to 22 layers deep, and the deepest of them needed workarounds to train from a random start: VGG trained an 11-layer network first and copied its weights into the deeper ones, and GoogLeNet attached extra classifiers to intermediate layers to inject gradient there. Every one of these networks used the rectified linear unit, ReLU, which outputs its input when positive and zero otherwise. The paper studies that zero from two sides: the information it throws away, and what it does to the scale of the signal in a deep network.

The page covers, in order (equation numbers follow the paper):

  1. PReLU, a ReLU whose slope for negative inputs is learned, and how that slope is trained, equations (1) to (4);
  2. why a rectifier network's signal shrinks or grows by a constant factor per layer, and the initialization that makes the factor 1, equations (5) to (10);
  3. the same analysis for gradients flowing backward, equations (11) to (14), and the PReLU version, equation (15);
  4. why one fixed standard deviation fails on VGG, and what happened at 22 and 30 layers;
  5. the ImageNet models, the results in Tables 2 to 7, and what the comparison with human error does and does not show.

What PReLU changes in the ReLU

A convolutional layer computes, at every pixel, a weighted sum of the pixels around it in every input channel, and then applies an activation function to each sum. Call the sum for channel ii at one position yiy_i. ReLU returns max⁡(0,yi)\max(0, y_i). It is cheap, and its gradient is exactly 1 for every positive input, one reason it replaced the sigmoid and tanh, whose gradients shrink toward zero for large inputs. A negative input gets output 0 and gradient 0: no signal passes forward and no gradient passes back through that unit at that position.

PReLU, the parametric rectified linear unit, gives the negative side a slope:

f(yi)={yi,if yi>0aiyi,if yi≤0f(y_{i})=\begin{cases}y_{i},&\text{if }y_{i}>0\\a_{i}y_{i},&\text{if }y_{i}\leq 0\end{cases}
(1)

The same function in one line:

f(yi)=max⁡(0,yi)+aimin⁡(0,yi)f(y_i) = \max(0, y_i) + a_i \min(0, y_i)

With ai=0a_i = 0 this is ReLU. With aia_i fixed at 0.01 it is the Leaky ReLU of Maas, Hannun and Ng (2013), designed so that negative inputs still pass a small gradient; their experiments found it changed accuracy very little. PReLU makes aia_i a parameter of the network, trained by backpropagation with everything else, so training sets how much of the negative half each channel passes. The subscript ii means one slope per channel: a layer with 256 channels gets 256 slopes, shared across every position of each channel's feature map. The paper also tests a channel-shared variant with a single aa per layer.

The extra parameters are few. In the 14-layer model the paper uses for its comparison, the channel-wise version adds one slope per channel of the 11 conv layers and two hidden fully connected layers: 64 + 4 × 128 + 6 × 256 + 2 × 4,096 = 10,304 slopes, against about 75 million weights, or 0.014%. The channel-shared version adds 13. Figure 1 shows the shape for any slope; the strip beneath it holds the values the 14-layer model learned.

Figure 1 · PReLU and the slopes it learned
a = 0.25
Drag the slider to set a: 0 gives the ReLU, 1 the identity, below 0 a function that rises again for negative inputs. The header gives E[f(y)2]E[f(y)^2] for a zero-mean symmetric input, used again in equation (15). Click a layer in the strip to load the channel-shared slope it learned.

How the slope is learned

The gradient of the loss E\mathcal{E} with respect to a slope comes from the chain rule. Every position of channel ii's feature map uses the same aia_i, so its gradient is a sum over those positions:

∂E∂ai=∑yi∂E∂f(yi)∂f(yi)∂ai\frac{\partial\mathcal{E}}{\partial a_{i}}=\sum_{y_{i}}\frac{\partial\mathcal{E}}{\partial f(y_{i})}\frac{\partial f(y_{i})}{\partial a_{i}}
(2)

The first factor is the gradient arriving from the layer above. The second is the derivative of (1) with respect to the slope, which is zero wherever the input was positive:

∂f(yi)∂ai={0,if yi>0yi,if yi≤0\frac{\partial f(y_{i})}{\partial a_{i}}=\begin{cases}0,&\text{if }y_{i}>0\\y_{i},&\text{if }y_{i}\leq 0\end{cases}
(3)

So the slope learns only from positions where the input was negative, weighted by how negative it was. In the channel-shared variant the sum in (2) also runs over every channel of the layer. The slope is updated with momentum μ\mu, which keeps a decaying running sum of past steps so that successive updates smooth each other, and learning rate ϵ\epsilon:

Δai:=μΔai+ϵ∂E∂ai\Delta a_{i}:=\mu\Delta a_{i}+\epsilon\frac{\partial\mathcal{E}}{\partial a_{i}}
(4)

Written this way the buffer accumulates the gradient itself, so the parameter is then decreased by it, ai←ai−Δaia_i \leftarrow a_i - \Delta a_i, which is how Caffe's SGD solver applies the update. Here are (1) to (4) on four positions of one channel:

# Equations (1)-(4) on four positions of one channel's feature map, a = 0.25.
y    = [ 1.2, -0.5,  0.3, -2.0]       # inputs y_i to the activation
f    = [ 1.2, -0.125, 0.3, -0.5]      # eq. (1): y if y > 0, else a * y
g    = [ 0.1,  0.4, -0.2,  0.05]      # dE/df(y_i), arriving from the next layer

# eq. (3): df/da = 0 where y > 0 and y where y <= 0, so only two terms survive
dE_da = 0.4 * (-0.5) + 0.05 * (-2.0)  # = -0.3, eq. (2) summed over positions
dE_dy = [0.1, 0.4 * 0.25, -0.2, 0.05 * 0.25]   # slope 1 or a: passed down

# eq. (4) with momentum mu = 0.9 and learning rate eps = 0.01, no weight decay
delta = 0.9 * 0.0 + 0.01 * dE_da      # = -0.003
a = 0.25 - delta                      # = 0.253 (Caffe applies a <- a - delta)

The gradient sent down to yy is the upstream gradient times the local slope, 1 or aa, so with a=0.25a = 0.25 a negative input passes a quarter of its gradient where a ReLU would pass none. In a real layer the sum in (2) runs over every position in every image of the mini-batch: 56 × 56 positions times 128 images is about 400,000 terms per channel, roughly half of them negative.

Section 2.1 makes three further choices. The slopes start at ai=0.25a_i = 0.25. They get no weight decay, because weight decay adds a pull toward zero that would bias every PReLU back toward a ReLU. And they are not constrained to [0,1][0, 1], so a negative slope, and with it a non-monotonic activation, is allowed; the paper reports that the learned values rarely exceed 1 in magnitude. In PyTorch this is nn.PReLU(C, init=0.25) for a layer with C channels, whose documentation also says not to apply weight decay to aa.

What the learned slopes look like

The comparison uses model E of He and Sun (2014), 11 conv layers with mostly 2×2 filters followed by spatial pyramid pooling and three fully connected layers, trained for 75 epochs. With ReLU it reaches 33.82% top-1 and 13.34% top-5 error on the ImageNet 2012 validation set, using 10-view testing (the four corners and the center of the image plus their mirror images, scores averaged). Replacing every ReLU with PReLU and retraining from scratch gives:

14-layer model, 10-viewtop-1top-5extra parameters
ReLU33.8213.340
PReLU, channel-shared32.7112.8713
PReLU, channel-wise32.6412.7510,304

That is 1.18 points of top-1 error for the channel-wise version and 1.11 points for the channel-shared one, so almost all of the gain comes from the 13 per-layer slopes. The strip in Figure 1 shows two patterns in the learned values. The first conv layer keeps a large slope on its negative side, 0.681 shared and 0.596 averaged over channels. Its 64 filters are mostly edge and texture detectors, and the paper's reading is that the layer uses both the positive and the negative response of each filter (an edge and the same edge with opposite polarity), which gets more out of a small number of filters. Deeper layers learn smaller slopes, down to 0.008 for conv35 in the shared version and 0.031 to 0.075 in the fully connected layers. The paper reads this, without testing it, as the network keeping more of each signal early and becoming more nonlinear, closer to a ReLU, near the classifier.

Why deep rectifier networks were hard to start

Before training, every weight is set to a random number, usually drawn from a zero-mean Gaussian. In a deep network the standard deviation of that Gaussian decides whether training starts. AlexNet used a fixed 0.01 for every layer. The VGG team reported that networks deeper than about 8 conv layers had difficulty converging from such a start, which is why they pre-trained the shallow configuration and copied it into the deep ones; the paper notes that this costs extra training time and may land in a worse local optimum. GoogLeNet added auxiliary classifiers in the middle of the network to help it converge.

The scale problem compounds. Suppose each layer multiplies the size of the signal passing through it by a factor β\beta. After LL layers the signal has been multiplied by βL\beta^L. With β=0.9\beta = 0.9 and 30 layers that is 0.04; with β=1.1\beta = 1.1 it is 17. The same holds for the gradient going backward. If the factor is below 1 the early layers receive almost no gradient and learning stalls; if it is above 1 the output grows until it overflows. Section 2.2 describes these two failure modes: with β>1\beta > 1 the algorithm diverges, with β<1\beta < 1 it stalls.

Glorot and Bengio (2010) had derived an initialization with β=1\beta = 1 by tracking the variance of each layer's output. Their derivation assumes the activation function is linear around zero, which is roughly true for tanh, one of the activations they studied, and it arrives at n Var[w]=1n\,\mathrm{Var}[w] = 1, where nn is the number of inputs to a unit. Caffe calls this “Xavier” initialization, after Glorot's first name, and the paper uses that name. A ReLU is not linear around zero. It is linear on one side and zero on the other, and the next section shows that this changes the factor by exactly 2.

How He initialization works

Take one conv layer ll and one output position. The response is a matrix-vector product:

yl=Wlxl+bl\mathbf{y}_{l}=\mathrm{W}_{l}\mathbf{x}_{l}+\mathbf{b}_{l}
(5)

Here xl\mathbf{x}_l stacks the k×kk \times k pixels around the position in all cc input channels into one vector of length n=k2cn = k^2 c, the number of inputs each output sees (its fan-in). Each of the dd rows of Wl\mathrm{W}_l is one filter, so Wl\mathrm{W}_l is d×nd \times n and yl\mathbf{y}_l holds the dd filter responses at that pixel. The input to layer ll is the previous layer's output after the activation, xl=f(yl−1)\mathbf{x}_l = f(\mathbf{y}_{l-1}), and the number of input channels is the previous layer's number of filters, cl=dl−1c_l = d_{l-1}. A 3×3 conv reading 256 channels has n=9×256=2,304n = 9 \times 256 = 2{,}304.

The derivation treats the weights and inputs as random variables: the weights independent of each other and of the inputs, all drawn from one distribution with mean zero, and the inputs also independent and identically distributed. Each response is then a sum of nn independent products wxw x, and the variance of a sum of independent terms is the sum of their variances:

Var[yl]=nlVar[wlxl]\mathrm{Var}[y_{l}]=n_{l}\mathrm{Var}[w_{l}x_{l}]
(6)

For independent ww and xx with E[w]=0E[w] = 0, the variance of the product is

E[w2]E[x2]−(E[w]E[x])2=Var[w] E[x2]E[w^2]E[x^2] - (E[w]E[x])^2 = \mathrm{Var}[w]\,E[x^2]

so (6) becomes

Var[yl]=nlVar[wl]E[xl2]\mathrm{Var}[y_{l}]=n_{l}\mathrm{Var}[w_{l}]E[x^{2}_{l}]
(7)

The factor is E[x2]E[x^2], the mean of the square, and it equals Var[x]\mathrm{Var}[x] only when xx has mean zero. Glorot and Bengio's derivation could use the variance because a tanh output has mean zero. A ReLU output is never negative, so its mean is positive and its variance is smaller than its mean square.

If the previous layer's weights are symmetric around zero and its biases are zero, then yl−1y_{l-1} is symmetric around zero too: a value and its negative are equally likely. The ReLU keeps the positive half and replaces the negative half with zeros, so the square of its output is y2y^2 on half the distribution and 0 on the other half. For any distribution symmetric around zero that gives

E[xl2]=12Var[yl−1]E[x_l^2] = \tfrac{1}{2}\mathrm{Var}[y_{l-1}]

A million standard normal samples show the half, and that the variance of the ReLU output is smaller still:

y = torch.randn(1_000_000)             # zero mean, symmetric, Var[y] = 1
x = y.clamp(min=0)                     # ReLU
print(x.pow(2).mean())                 # 0.500  E[x^2] = Var[y] / 2
print(x.mean(), x.var())               # 0.399, 0.341  x is not zero-mean

Substituting the half into (7) gives the per-layer recursion:

Var[yl]=12nlVar[wl]Var[yl−1]\mathrm{Var}[y_{l}]=\frac{1}{2}n_{l}\mathrm{Var}[w_{l}]\mathrm{Var}[y_{l-1}]
(8)

and chaining it from layer 2 to layer LL:

Var[yL]=Var[y1](∏l=2L12nlVar[wl])\mathrm{Var}[y_{L}]=\mathrm{Var}[y_{1}]\left(\prod_{l=2}^{L}\frac{1}{2}n_{l}\mathrm{Var}[w_{l}]\right)
(9)

Each factor in the product is the β2\beta^2 of the previous section for one layer. If every factor is 1, the variance of the last layer equals the variance of the first, which is the condition the paper adopts:

12nlVar[wl]=1,∀l\frac{1}{2}n_{l}\mathrm{Var}[w_{l}]=1,\quad\forall l
(10)

That is a zero-mean Gaussian with standard deviation 2/nl\sqrt{2/n_l}, and biases set to zero. For the 3×3 conv over 256 channels, 2/2304=0.0295\sqrt{2/2304} = 0.0295, where the linear-case rule n Var[w]=1n\,\mathrm{Var}[w] = 1 gives 1/2304=0.0208\sqrt{1/2304} = 0.0208. The first layer reads the image, which has not been through a ReLU, so strictly it should use n1Var[w1]=1n_1 \mathrm{Var}[w_1] = 1; the paper uses (10) there too, because a factor of 2 at a single layer does not compound.

Figure 2 runs the recursion on actual random networks. Each is 30 fully connected layers of width n=128n = 128 (a fully connected layer is the k=1k = 1 case of (5)), with weights of standard deviation g/ng/\sqrt{n} for a gain gg. Since a rectifier network without biases satisfies f(gy)=gf(y)f(g y) = g f(y), changing gg rescales layer ll by exactly glg^l, so the figure computes each network once and the slider is exact. Look at where the teal line sits at layer 30 for g=1g = 1 and for g=2g = \sqrt{2}.

Figure 2 · Signal and gradient through 30 rectifier layers
g = 1.000
Slide g or press a preset. Thick lines are the RMS over eight random networks, thin lines the individual networks, dashed lines equations (9) and (13). The tabs switch between ReLU and PReLU with a=0.25a = 0.25.

At g=1g = 1, the Xavier setting, every ReLU layer halves the variance, so the standard deviation falls by 1/21/\sqrt{2} per layer: at layer 30 it is 4.3×10−54.3 \times 10^{-5} of the input's, 2−152^{-15} of what g=2g = \sqrt{2} gives, and the gradient reaching layer 1 has shrunk by 2−152^{-15} as well. At g=2g = \sqrt{2} both lines stay level. The individual networks scatter around the average by up to about a factor of 2 after 30 layers, because a width of 128 is a small sample of the distribution the derivation averages over; a wider network scatters less. The slider's ends, 0.5 and 2, show the stall and the explosion: a factor of about 4×10−144 \times 10^{-14} or 5×1045 \times 10^{4} across 30 layers.

For PReLU, the negative half is scaled by aa instead of zeroed, so its square contributes a2a^2 times as much, and the mean square of the output is the number in the header of Figure 1:

E[f(y)2]=12(1+a2) Var[y]E[f(y)^2] = \tfrac{1}{2}(1 + a^2)\,\mathrm{Var}[y]

The condition becomes

12(1+a2)nlVar[wl]=1,∀l\frac{1}{2}(1+a^{2})n_{l}\mathrm{Var}[w_{l}]=1,\quad\forall l
(15)

with aa the initial slope. At a=0a = 0 this is (10), and at a=1a = 1, where the activation is the identity, it is the linear rule n Var[w]=1n\,\mathrm{Var}[w] = 1. At the paper's initial a=0.25a = 0.25 the correction is small, 1+a2=1.06251 + a^2 = 1.0625: using the ReLU rule for a PReLU network grows the variance by 1.0625 per layer, about 6 times over 30 layers (2.4 times in standard deviation), which the PReLU tab of Figure 2 shows at g=2g = \sqrt{2}.

In PyTorch, kaiming_normal_ with a=0.25 and nonlinearity='leaky_relu' draws standard deviation 2/(1+a2)/n\sqrt{2/(1+a^2)}/\sqrt{n}, which is (15); with a=0 or nonlinearity='relu' it is (10):

import torch, torch.nn as nn

conv = nn.Conv2d(256, 256, kernel_size=3, padding=1)   # fan-in n = 9 * 256 = 2304
act = nn.PReLU(num_parameters=256, init=0.25)          # one slope per channel

# eq. (15) with a = 0.25: std = sqrt(2 / ((1 + 0.25**2) * 2304)) = 0.0286
nn.init.kaiming_normal_(conv.weight, a=0.25, mode='fan_in',
                        nonlinearity='leaky_relu')
nn.init.zeros_(conv.bias)                               # b = 0, as in the paper
# the paper's own runs used eq. (14): mode='fan_out' divides by 9 * 256 too here,
# because this layer has as many filters (d) as input channels (c)

opt = torch.optim.SGD([
    {'params': conv.parameters(), 'weight_decay': 5e-4},
    {'params': act.parameters(), 'weight_decay': 0.0},  # no decay on the slopes
], lr=1e-2, momentum=0.9)

x = torch.randn(32, 256, 56, 56)       # batch 32, 256 channels, 56 x 56
y = act(conv(x))                       # same shape; act.weight has shape (256,)

The backward pass: fan-in or fan-out

Gradients flow through the same weights in the opposite direction. Write Δx\Delta\mathbf{x} and Δy\Delta\mathbf{y} for the gradients of the loss with respect to x\mathbf{x} and y\mathbf{y}. At one pixel,

Δxl=W^lΔyl\Delta\mathbf{x}_{l}=\hat{\mathrm{W}}_{l}\Delta\mathbf{y}_{l}
(11)

Δyl\Delta\mathbf{y}_l now stacks k×kk \times k pixels in all dd output channels, a vector of length n^=k2d\hat{n} = k^2 d, because each input pixel was read by k2k^2 positions of each of the dd filters. W^l\hat{\mathrm{W}}_l is the same set of weights rearranged into a c×n^c \times \hat{n} matrix, and Δxl\Delta\mathbf{x}_l has one entry per input channel. So the backward pass sums over n^\hat{n} terms, the fan-out, where the forward pass summed over n=k2cn = k^2 c, the fan-in.

The activation enters through its derivative: Δyl=f′(yl) Δxl+1\Delta y_l = f'(y_l)\,\Delta x_{l+1}. For a ReLU, f′f' is 1 or 0 with equal probability, because yly_l is symmetric around zero. Assuming f′(yl)f'(y_l) independent of Δxl+1\Delta x_{l+1}, the mean square of Δyl\Delta y_l is half that of Δxl+1\Delta x_{l+1}, and since gradients have mean zero (the weights are symmetric), mean square and variance agree. The variance of (11) then follows the same steps as (6) to (8):

Var[Δxl]=n^lVar[wl]Var[Δyl]=12n^lVar[wl]Var[Δxl+1]\begin{aligned}\mathrm{Var}[\Delta x_{l}]&=\hat{n}_{l}\mathrm{Var}[w_{l}]\mathrm{Var}[\Delta y_{l}]\\&=\frac{1}{2}\hat{n}_{l}\mathrm{Var}[w_{l}]\mathrm{Var}[\Delta x_{l+1}]\end{aligned}
(12)

The half has a different origin here, the probability that the ReLU passes gradient rather than the mean square of its output, and the same value. Chaining from the top of the network down to layer 2:

Var[Δx2]=Var[ΔxL+1](∏l=2L12n^lVar[wl])\mathrm{Var}[\Delta x_{2}]=\mathrm{Var}[\Delta x_{L+1}]\left(\prod_{l=2}^{L}\frac{1}{2}\hat{n}_{l}\mathrm{Var}[w_{l}]\right)Var[Δx2]= Var[ΔxL+1](∏l=2L12n^lVar[wl])\begin{aligned}\mathrm{Var}[\Delta x_{2}]=&\,\mathrm{Var}[\Delta x_{L+1}]\\&\left(\prod_{l=2}^{L}\frac{1}{2}\hat{n}_{l}\mathrm{Var}[w_{l}]\right)\end{aligned}
(13)

and the condition that keeps gradients from shrinking or growing is

12n^lVar[wl]=1,∀l\frac{1}{2}\hat{n}_{l}\mathrm{Var}[w_{l}]=1,\quad\forall l
(14)

a Gaussian with standard deviation 2/n^l\sqrt{2/\hat{n}_l}. Equations (10) and (14) differ only in which count divides: the number of input channels clc_l or the number of filters dld_l. When a layer has as many filters as input channels they coincide.

One condition is enough. Suppose the network uses (14). Then the backward product in (13) is exactly 1, and each factor of the forward product in (9) becomes

nln^l=cldl=dl−1dl\frac{n_l}{\hat{n}_l} = \frac{c_l}{d_l} = \frac{d_{l-1}}{d_l}

Multiplying these from layer 2 to LL, each layer's filter count cancels against the next layer's input count, and the product telescopes to

∏l=2L12nlVar[wl]=∏l=2Lnln^l=c2dL\prod_{l=2}^{L}\frac{1}{2}n_{l}\mathrm{Var}[w_{l}]=\prod_{l=2}^{L}\frac{n_{l}}{\hat{n}_{l}}=\frac{c_{2}}{d_{L}}

the second layer's input channels over the last layer's filters. For a typical network that widens from 64 to 512 channels, that is 1/8 in variance, a fixed ratio that does not grow with depth. The same argument run the other way shows (10) leaves the backward product at dL/c2=8d_L/c_2 = 8. The paper trains all its models with (14). PyTorch's kaiming_normal_ and Caffe's MSRAFiller default to fan-in, (10), with fan-out available as an option, and torchvision's ResNet selects mode='fan_out'.

Why a constant std of 0.01 stalls

The derived standard deviation depends on the layer's width, so a single constant cannot be right for every layer of a network whose width changes. The paper works through VGG's model B: 10 conv layers, all 3×3, with 64 filters in layers 1 and 2, 128 in layers 3 and 4, 256 in 5 and 6, and 512 in 7 to 10. By (14), with n^=9d\hat{n} = 9d, the right standard deviations are 0.059, 0.042, 0.029 and 0.021 for 64, 128, 256 and 512 filters. A constant 0.01 is too small everywhere, by a factor of 5.9 in layer 2 and 2.1 in each of layers 7 to 10. The gradient reaching layer 2 from layer 10 passes through nine layers, and each shrinks its standard deviation by its own factor:

15.9×4.22×2.92×2.14=11.7×104\frac{1}{5.9\times 4.2^{2}\times 2.9^{2}\times 2.1^{4}}=\frac{1}{1.7\times 10^{4}}

With unrounded factors it is 1/16,729. At that scale the loss contributes almost nothing to the weight update: footnote 2 of the paper notes that with weight decay switched on, the update is then made almost entirely of the decay term, and suggests checking for exactly that as a diagnosis. Figure 3 recomputes the product for any constant and compares the two equations.

Figure 3 · One constant standard deviation on VGG's model B
0.0100
Move the slider to choose the constant, or pick equation (10) or (14). The amber line is the gradient's scale at each layer relative to the gradient arriving at conv10; the teal line is the forward signal's scale relative to conv1's output. Both axes are logarithmic in the scale.

At 0.01 the amber line falls by more than four orders of magnitude. Raising the constant to about 0.0295 brings the end-to-end product back to 1, but only by balancing layers that shrink the gradient against layers that grow it, as the per-layer row shows; at 0.1 the gradient grows about 60,000-fold. Equation (14) gives a flat amber line and a teal line that ends at 0.354, the square root of the 1/8 from the previous section; equation (10) swaps the two.

The fully connected layers are the paper's exception. It uses a standard deviation of 0.01 for the first two and 0.001 for the last, smaller than the derived 2/4096=0.022\sqrt{2/4096} = 0.022. Because the initialization preserves variance from the first layer to the last, an input image with pixel values in roughly [−128,128][-128, 128] would reach the softmax with logits large enough to overflow it. Shrinking a few layers' weights scales the output down without renormalizing the input, which would have changed other hyperparameters.

He versus Xavier at 22 and 30 layers

Xavier's n Var[w]=1n\,\mathrm{Var}[w] = 1 in a ReLU network sets each factor of (9) to 1/2. After LL layers the signal's standard deviation is (1/2)L(1/\sqrt{2})^L of what (10) gives: about 1/2,048 at 22 layers and 1/32,768 at 30. The paper trained networks of both depths with each rule.

On the 22-layer model B, described below, both initializations converge. Figure 2 of the paper plots top-1 error on 3,000 validation images over training: with (14) the error starts falling earlier. On the 14-layer model from Table 2, Xavier gives 33.90/13.44 top-1/top-5 error and (14) gives 33.82/13.34.

The 30-layer model is the 14-layer model with sixteen extra conv layers of 256 2×2 filters, 27 conv and 3 fully connected layers in all. With Xavier initialization it does not train: the error does not fall and the gradients, which the authors monitored, shrink toward zero. With (14) it converges. Its final error, though, is 38.56/16.59, worse than the 33.82/13.34 of the 14-layer model it was built from. The paper also ran model B with 3 to 9 extra layers and found both training and test error degraded in the first 20 epochs (it did not finish those runs). This is degradation, a deeper network training to a worse result, and the paper offers two guesses for it: that adding layers this way is the wrong way to add depth, or that ImageNet classification is not complex enough to need it. Ten months later He, Zhang, Ren and Sun addressed the first with residual networks, which initialize their weights with this paper's method and reach 152 layers on ImageNet.

The ImageNet models

Because extra depth was not paying off, the paper made its large models wider instead. Model A starts from VGG-19 with three changes: the first layer is a single 7×7 conv with stride 2 and 96 filters; the other three conv layers that VGG-19 runs on its two largest feature maps (224 and 112) are moved to the 56, 28 and 14 maps; and a spatial pyramid pooling (SPP) layer before the first fully connected layer pools the last feature map into 7×7, 3×3, 2×2 and 1×1 grids of bins, 63 bins in total, so the network accepts any input size. The compute is about the same as VGG-19's (1.90 against 1.96 × 10¹⁰ operations), and A runs a mini-batch of 128 in 2.6 seconds on four Nvidia K20 GPUs against 3.0 seconds for VGG-19, because the moved layers run on smaller maps. Model B adds three conv layers (22 layers). Model C is B with 384, 768 and 896 filters in its three stages instead of 256, 512 and 512, at 5.30 × 10¹⁰ operations, about 2.3 times B.

Training mostly follows AlexNet and VGG. Each training image is resized so its shorter side is a random s∈[256,512]s \in [256, 512], and a random 224×224 crop is taken, with half of the crops mirrored and the colors randomly altered. VGG applied this scale jittering only while fine-tuning; here it runs from the first epoch, and the deep models train end to end from the initialization of (14) instead of from a shallower network. The optimizer is SGD with momentum 0.9 and weight decay 0.0005, mini-batch 128, dropout of 50% in the first two fully connected layers, and a learning rate of 0.01, then 0.001, then 0.0001, each switched when the error plateaus, for about 80 epochs. That took 3 to 4 weeks for A or B on four K20s and for C on eight K40s. The convolutional layers are data-parallel across GPUs and the fully connected layers run on one GPU, giving speedups of 3.8× on 4 GPUs and 6.0× on 8.

At test time the conv layers run once on the whole resized image. Every 14×14 window of the last feature map is pooled by the SPP layer and classified, the scores of all windows are averaged, the same is done for the mirrored image, and the results at several image scales are combined.

Results on ImageNet

ImageNet 2012 has about 1.2 million training images in 1,000 classes, 50,000 validation images and 100,000 test images whose labels are held by the challenge server. The headline metric is top-5 error: the fraction of images whose true class is not among the model's five highest-scoring guesses. On model A, PReLU against ReLU, trained for the same epochs with the same learning-rate switches (Table 4):

model A, test scaleReLU top-1ReLU top-5PReLU top-1PReLU top-5
25626.258.2525.818.08
38424.777.2624.207.03
48025.467.6324.837.39
multi-scale24.026.5122.976.28

PReLU lowers top-1 error by 1.05 points and top-5 by 0.23 points in the multi-scale combination, and wins at every single scale. The best single scale, 384, is the middle of the training jitter range. On the same model family, the ReLU baseline's 6.51% is already below the best single-model result published before it, 7.1% for VGG-19; the paper attributes that gap mainly to training end to end without pre-training a shallow model. Figure 4 lines up the single-model chain and the ensembles.

Figure 4 · ImageNet 2012 results
The first tab is Table 7 of the paper, ensembles on the test set, with the human estimate and its 95% interval; the violet bar is from the Batch Normalization paper. The second tab is Table 6, single models on the validation set, in top-5 or top-1. Teal bars are this paper's models.

In the single-model chain, PReLU takes model A from 6.51% to 6.28% top-5, three more layers (B) change it to 6.27%, and the wider model C reaches 5.71%, which the paper notes is below every multi-model result published before it. Depth from 19 to 22 layers bought nothing here, and width bought 0.56 points. With 10-view testing instead of dense multi-scale testing, C scores 7.38% (Table 5). The final entry combines six models, of which only one has architecture C, and scores 4.94% top-5 on the test set. That is 1.72 points, or 26% relative, below GoogLeNet's 6.66%, the ILSVRC 2014 winner, and 17% relative below Baidu's 5.98%, the best result published in the meantime.

The per-class breakdown in the paper's Figure 6 shows zero top-5 error on 113 of the 1,000 test classes. The worst three are letter opener (49%), spotlight (38%) and restaurant (36%), where images contain several objects, small objects, or very varied examples of the class. Against the same team's ILSVRC 2014 entry (8.06%), the error fell in 824 classes, stayed the same in 127 and rose in 49.

What the human-level claim means

The 5.1% human figure comes from Russakovsky et al. (2015), the paper that describes the ImageNet challenge. One annotator trained on 500 validation images, learning what the 1,000 classes are, and then labeled 1,500 test images through an interface that showed 13 example training images next to each class name. A second annotator, trained on 100 images, labeled 258 test images with 12.0% error. The study found the two made different mistakes: on the 204 images both labeled, counting an image as correct if either got it right gives 2.4%.

So 5.1% is one careful person's error on a sample of 1,500 images, and it has sampling error of its own: a binomial proportion of 5.1% on 1,500 trials has a 95% interval from about 4.0% to 6.2%. The ensemble's 4.94% sits 0.16 points below the point estimate and inside that interval, which the band in Figure 4 shows. Five days after this paper appeared, Ioffe and Szegedy's Batch Normalization reported 4.82% with an ensemble of Inception networks.

The paper's Section 4 makes the same qualification. Russakovsky et al. found that the human errors were mostly fine-grained (telling apart 120 dog breeds, or species of birds) and from not knowing a class existed; the model is good at the first kind, and the paper's Figure 4 shows it naming a coucal, a komondor and a yellow lady's slipper. It also fails on images a person finds easy, such as spotlights, where the answer depends on context. The authors write that the result “does not indicate that machine vision outperforms human vision on object recognition in general.”

Limitations and what came next

The derivation describes the network at initialization only. It assumes independent, identically distributed weights and inputs, zero biases and symmetric pre-activations, and its conclusions are about variances averaged over random weights; any single finite-width network deviates from it, as the thin lines in Figure 2 show, and training moves the weights away from these assumptions. It does not cover pooling, residual connections or normalization layers. Getting the scale right made 30-layer plain networks trainable without making them better, and PReLU's gains, about one point of top-1 error, are smaller than those of the wider model C.

Batch normalization, five days later, made the per-layer scale far less critical by standardizing each layer's pre-activations during training. ResNet, in December 2015, combined this initialization with batch normalization and identity shortcuts and trained 152 layers. ResNet kept the plain ReLU. The variance argument itself carried forward: later work such as μP asks the same question, which scale keeps each layer's signal of order one, for the updates during training as well as for the weights at initialization.

Provenance Verified against primary literatureHow we verify
He, Zhang, Ren & Sun (arXiv 1502.01852v1, 6 February 2015; ICCV 2015)Equations (1) to (15), Tables 1 to 7, Figures 2, 3, 6 and 7, Sections 2.1 (PReLU definition, optimization, a = 0.25 initial value, no weight decay on a, the 14-layer comparison), 2.2 (forward and backward derivations, the VGG model B example, the fc-layer stds 0.01 and 0.001, the Xavier comparison at 14, 22 and 30 layers), 2.3 (models A, B, C), 3 (training and testing details, 3.8x and 6.0x multi-GPU speedups) and 4 (results, per-class analysis, the human comparison). Table and figure numbers follow the PDF; the arXiv HTML render labels the 22-layer convergence plot "Figure 3" in the text and merges Tables 5 to 7 under one number.
Glorot & Bengio, Understanding the difficulty of training deep feedforward neural networks (AISTATS 2010)The forward condition n Var[W] = 1 and the backward condition for the linear regime, and equation (16), the "normalized initialization" W ~ U[-sqrt(6)/sqrt(n_j + n_j+1), +sqrt(6)/sqrt(n_j + n_j+1)], whose variance is 2/(n_in + n_out).
Simonyan & Zisserman, Very Deep Convolutional Networks (arXiv 1409.1556v6), Section 3.1Configuration A (8 conv layers) trained first; deeper nets initialized with its first four conv layers and last three fc layers. Random initialization "from a normal distribution with the zero mean and 10^-2 variance". The note that after submission Glorot & Bengio initialization was found to work without pre-training.
Russakovsky et al., ImageNet Large Scale Visual Recognition Challenge (arXiv 1409.0575v3), Section 6.4Annotator A1 trained on 500 images and labeled 1,500 test images: 5.1% top-5 error (GoogLeNet 6.8% on the same sample). Annotator A2 trained on 100 images and labeled 258: 12.0%. On the 204 images both labeled, an image counted correct if either was right gives 2.4%.
Ioffe & Szegedy, Batch Normalization (arXiv 1502.03167, submitted 11 February 2015)Abstract: an ensemble reaches 4.9% top-5 validation error and 4.8% test error (4.82% in the paper's results table).
He, Zhang, Ren & Sun, Deep Residual Learning (arXiv 1512.03385v1), Section 3.4"We initialize the weights as in [13] and train all plain/residual nets from scratch", where [13] is this paper.
PyTorch torch/nn/init.py and torch/nn/modules/activation.py (main branch, November 2026)calculate_gain (line 173): ReLU gain sqrt(2), leaky_relu gain sqrt(2/(1 + negative_slope^2)) (line 238). kaiming_normal_ (line 619): defaults a = 0, mode = "fan_in", nonlinearity = "leaky_relu", std = gain / sqrt(fan). class PReLU (line 1576): num_parameters = 1 and init = 0.25 by default (line 1630); the docstring (line 1597) says weight decay should not be used on a. torch/nn/modules/linear.py line 124: nn.Linear's default is kaiming_uniform_ with a = sqrt(5).
BVLC Caffe include/caffe/filler.hpp and src/caffe/solvers/sgd_solver.cppfiller.hpp line 189: MSRAFiller, std = sqrt(2/n) (line 208) with n = fan_in by default (line 200), fan_in = count / num_outputs = c k k; FAN_OUT and AVERAGE are options. XavierFiller (line 150 on) also defaults to fan_in: uniform with variance 1/n. sgd_solver.cpp line 232: history = local_rate * gradient + momentum * history, then the parameter is decreased by history, the sign convention of equation (4).
This page, arithmetic and simulationsEquation (14) stds for VGG model B, 0.0589, 0.0417, 0.0295, 0.0208, and the exact gradient factor 1/16,729 for a constant 0.01 (the paper's 1/(1.7 x 10^4)); the same arithmetic at 0.1 gives x59,777; the forward product c_2/d_10 = 1/8 under equation (14). Channel counts for Table 1's model (10,304 PReLU slopes channel-wise) and its weight count (about 75.0 million, from the layer sizes in Table 1). The 95% interval 4.0% to 6.2% for 5.1% on 1,500 images. Figure 2 is a simulation of eight random 30-layer fully connected networks of width 128, built for this page; Figure 1's strip and Figure 4 are the paper's Tables 1, 6 and 7.
caveatFive notes on scope and convention; the paper's equations check out as printed. (1) "Surpassing human-level performance" compares an ensemble's 4.94% top-5 error on the 100,000-image test set with one trained annotator's 5.1% on 1,500 test images. The 0.16-point gap is well inside that estimate's own sampling error (95% interval about 4.0% to 6.2%), a second annotator in the same study scored 12.0%, and the paper itself says the result does not mean machines see better than people in general. Batch Normalization reported 4.82% five days later. (2) The paper explains VGG's difficulty with a constant std of 0.01. VGG's own text says its weights were drawn with "10^-2 variance", which is a std of 0.1; on the same arithmetic that would make the conv10-to-conv2 gradient about 60,000 times too large instead of 17,000 times too small. Either way no single constant suits every layer, which is the paper's point. (3) Equation (4) adds the gradient into the momentum buffer, Δa := μΔa + ε ∂E/∂a, so the parameter update it implies is a ← a − Δa, Caffe's convention. (4) "Xavier" in the paper means Caffe's filler, variance 1/n_in. Glorot and Bengio's own rule averages the forward and backward fan, variance 2/(n_in + n_out), from a uniform distribution; the paper's footnote 3 says so. Its sentence that Xavier's std "will be 1/√2^L of our derived std" refers to the signal after L layers; per layer the weight std is 1/√2 of He's. (5) The paper trained its models with equation (14), the fan-out form. PyTorch's kaiming_normal_ and Caffe's MSRAFiller default to fan-in, equation (10). Section 2.2 shows either one alone keeps both passes from shrinking or growing exponentially.

Questions you might still have

?

What is the difference between He initialization and Xavier initialization?
Both draw zero-mean weights with variance proportional to 1/n, where n is the number of inputs to a unit. Xavier (Glorot and Bengio, 2010) assumes the activation is linear near zero and uses variance 1/n (or 2/(n_in + n_out) in the original uniform form). He initialization accounts for the ReLU setting half of its inputs to zero, which halves the second moment of the signal, and doubles the variance to 2/n. In a ReLU network Xavier's rule therefore halves the signal's variance at every layer: after 30 layers its standard deviation is 2^-15, about 1/32,768, of what it would be under He initialization.

?

Should I use fan-in or fan-out?
Either works in a plain convolutional network. Fan-in (n = k²c, equation 10) keeps the forward activations at a constant scale; fan-out (n̂ = k²d, equation 14) keeps the backward gradients at a constant scale; the paper shows that choosing one leaves the other off by the ratio of the first layer's input channels to the last layer's filters (64/512 = 1/8 in variance for VGG's model B), which does not grow with depth. The paper trained with fan-out. PyTorch's kaiming_normal_ defaults to fan-in; torchvision's ResNet uses fan-out.

?

Does PyTorch use He initialization by default?
Not exactly. nn.Conv2d and nn.Linear default to kaiming_uniform_ with a = √5, which gives weight variance 1/(3n), a sixth of He's 2/n. Many model definitions, including torchvision's ResNet, overwrite that with kaiming_normal_. With batch normalization after every convolution the scale of the initialization matters much less, because each layer's output is renormalized.

?

Does He initialization have to be Gaussian?
No. The derivation only fixes the variance, 2/n, and any zero-mean symmetric distribution with that variance satisfies it. A uniform distribution on [-b, b] has variance b²/3, so the uniform version uses b = √(6/n); PyTorch's kaiming_uniform_ computes its bound as √3 times the Gaussian standard deviation. The paper uses a Gaussian; Glorot and Bengio's original rule is uniform.

?

Is PReLU the same as Leaky ReLU?
They have the same shape, a line of slope a for negative inputs. In Leaky ReLU (Maas et al., 2013) a is fixed at 0.01. In PReLU a is a parameter trained by backpropagation along with the weights, one per channel or one per layer, initialized to 0.25 and excluded from weight decay. The paper's learned values range from 0.008 to 0.681 (Table 1).

?

How much does PReLU help?
Modestly. On the 14-layer model, top-1 error falls from 33.82% to 32.64% (channel-wise) or 32.71% (channel-shared, 13 extra parameters). On the large model A it falls from 24.02% to 22.97% top-1 and from 6.51% to 6.28% top-5. The compute cost is one multiply on the negative inputs.

?

Did He initialization make very deep plain networks better?
It made them trainable, not better. The 30-layer model that stalls with Xavier initialization converges with He initialization, but reaches 38.56% top-1 error against 33.82% for the 14-layer model it was built from. The paper reports that training error also got worse when it added layers to model B. That degradation is the problem residual networks solved ten months later, using this paper's initialization (explained in the ResNet explainer).

?

Why does the derivation assume zero biases and symmetric weights?
With zero-mean symmetric weights and zero bias, each pre-activation y is a sum of many terms with mean zero and a distribution symmetric about zero. That symmetry gives exactly half of y's variance above zero, so a ReLU output has E[x²] = Var[y]/2. A nonzero bias would shift y and change that fraction. The analysis holds at initialization only; training moves the weights away from these assumptions.

?

Does initialization still matter with batch normalization?
Much less. Batch normalization, published five days after this paper, standardizes each layer's pre-activations over the mini-batch during training, which undoes any per-layer scale the initialization got wrong. He initialization is still the usual choice for the weights, and it matters again in networks without normalization layers.

?

What were the models, and how long did they take to train?
Model A is a 19-layer VGG-style network with a 7×7 stride-2 first layer and spatial pyramid pooling before the classifier; B adds three conv layers (22 layers); C is B with more filters per layer (384 to 896), about 2.3 times B's compute. Each trained for about 80 epochs with SGD, batch 128, weight decay 0.0005: about 3 to 4 weeks for A or B on four Nvidia K20 GPUs and for C on eight K40s.

Footnotes & further reading

  1. The paper: He, Zhang, Ren & Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (ICCV 2015). The authors did not release code; the paper implements PReLU and the initialization in its own modification of Caffe.
  2. The linear-case initialization: Glorot & Bengio, Understanding the difficulty of training deep feedforward neural networks (AISTATS 2010). Leaky ReLU: Maas, Hannun & Ng, Rectifier nonlinearities improve neural network acoustic models (ICML 2013 workshop).
  3. The networks it builds on: Simonyan & Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition (explained in VGG); Szegedy et al., Going Deeper with Convolutions (explained in GoogLeNet); He, Zhang, Ren & Sun, Spatial Pyramid Pooling in Deep Convolutional Networks (SPP-net); He & Sun, Convolutional Neural Networks at Constrained Time Cost (model E, the 14-layer comparison model).
  4. The human estimate: Russakovsky et al., ImageNet Large Scale Visual Recognition Challenge (IJCV 2015), Section 6.4.
  5. What came next: Ioffe & Szegedy, Batch Normalization (explained in Batch Normalization); He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition (explained in ResNet).
  6. Implementations: PyTorch's torch.nn.init (kaiming_normal_, kaiming_uniform_, calculate_gain) and nn.PReLU; Caffe's filler.hpp (MSRAFiller, XavierFiller).