He Initialization and PReLU: Delving Deep into Rectifiers
A ReLU zeroes the negative half of its input, so weights with variance 1/n, where n is the number of inputs to each unit, halve the signal's variance at every layer. Variance 2/n keeps the variance level, and a 30-layer network trains from scratch.
He, Zhang, Ren and Sun derive the 2/n initialization from the variance of each layer's output, give the ReLU a learned slope for negative inputs (PReLU), and combine the two into ImageNet models with 4.94% top-5 test error.
Explaining the paperDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationA February 2015 paper from Microsoft Research. Its initialization rule ships in every major framework as “He” or “Kaiming” initialization, and its 4.94% error was the first published ImageNet result below the 5.1% measured for a trained human annotator.
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun posted the paper on 6 February 2015. At the time the best ImageNet classifiers were 16 to 22 layers deep, and the deepest of them needed workarounds to train from a random start: VGG trained an 11-layer network first and copied its weights into the deeper ones, and GoogLeNet attached extra classifiers to intermediate layers to inject gradient there. Every one of these networks used the rectified linear unit, ReLU, which outputs its input when positive and zero otherwise. The paper studies that zero from two sides: the information it throws away, and what it does to the scale of the signal in a deep network.
The page covers, in order (equation numbers follow the paper):
- PReLU, a ReLU whose slope for negative inputs is learned, and how that slope is trained, equations (1) to (4);
- why a rectifier network's signal shrinks or grows by a constant factor per layer, and the initialization that makes the factor 1, equations (5) to (10);
- the same analysis for gradients flowing backward, equations (11) to (14), and the PReLU version, equation (15);
- why one fixed standard deviation fails on VGG, and what happened at 22 and 30 layers;
- the ImageNet models, the results in Tables 2 to 7, and what the comparison with human error does and does not show.
What PReLU changes in the ReLU
A convolutional layer computes, at every pixel, a weighted sum of the pixels around it in every input channel, and then applies an activation function to each sum. Call the sum for channel at one position . ReLU returns . It is cheap, and its gradient is exactly 1 for every positive input, one reason it replaced the sigmoid and tanh, whose gradients shrink toward zero for large inputs. A negative input gets output 0 and gradient 0: no signal passes forward and no gradient passes back through that unit at that position.
PReLU, the parametric rectified linear unit, gives the negative side a slope:
The same function in one line:
With this is ReLU. With fixed at 0.01 it is the Leaky ReLU of Maas, Hannun and Ng (2013), designed so that negative inputs still pass a small gradient; their experiments found it changed accuracy very little. PReLU makes a parameter of the network, trained by backpropagation with everything else, so training sets how much of the negative half each channel passes. The subscript means one slope per channel: a layer with 256 channels gets 256 slopes, shared across every position of each channel's feature map. The paper also tests a channel-shared variant with a single per layer.
The extra parameters are few. In the 14-layer model the paper uses for its comparison, the channel-wise version adds one slope per channel of the 11 conv layers and two hidden fully connected layers: 64 + 4 × 128 + 6 × 256 + 2 × 4,096 = 10,304 slopes, against about 75 million weights, or 0.014%. The channel-shared version adds 13. Figure 1 shows the shape for any slope; the strip beneath it holds the values the 14-layer model learned.
How the slope is learned
The gradient of the loss with respect to a slope comes from the chain rule. Every position of channel 's feature map uses the same , so its gradient is a sum over those positions:
The first factor is the gradient arriving from the layer above. The second is the derivative of (1) with respect to the slope, which is zero wherever the input was positive:
So the slope learns only from positions where the input was negative, weighted by how negative it was. In the channel-shared variant the sum in (2) also runs over every channel of the layer. The slope is updated with momentum , which keeps a decaying running sum of past steps so that successive updates smooth each other, and learning rate :
Written this way the buffer accumulates the gradient itself, so the parameter is then decreased by it, , which is how Caffe's SGD solver applies the update. Here are (1) to (4) on four positions of one channel:
# Equations (1)-(4) on four positions of one channel's feature map, a = 0.25.
y = [ 1.2, -0.5, 0.3, -2.0] # inputs y_i to the activation
f = [ 1.2, -0.125, 0.3, -0.5] # eq. (1): y if y > 0, else a * y
g = [ 0.1, 0.4, -0.2, 0.05] # dE/df(y_i), arriving from the next layer
# eq. (3): df/da = 0 where y > 0 and y where y <= 0, so only two terms survive
dE_da = 0.4 * (-0.5) + 0.05 * (-2.0) # = -0.3, eq. (2) summed over positions
dE_dy = [0.1, 0.4 * 0.25, -0.2, 0.05 * 0.25] # slope 1 or a: passed down
# eq. (4) with momentum mu = 0.9 and learning rate eps = 0.01, no weight decay
delta = 0.9 * 0.0 + 0.01 * dE_da # = -0.003
a = 0.25 - delta # = 0.253 (Caffe applies a <- a - delta)The gradient sent down to is the upstream gradient times the local slope, 1 or , so with a negative input passes a quarter of its gradient where a ReLU would pass none. In a real layer the sum in (2) runs over every position in every image of the mini-batch: 56 × 56 positions times 128 images is about 400,000 terms per channel, roughly half of them negative.
Section 2.1 makes three further choices. The slopes start at . They get no weight decay, because weight decay adds a pull toward zero that would bias every PReLU back toward a ReLU. And they are not constrained to , so a negative slope, and with it a non-monotonic activation, is allowed; the paper reports that the learned values rarely exceed 1 in magnitude. In PyTorch this is nn.PReLU(C, init=0.25) for a layer with C channels, whose documentation also says not to apply weight decay to .
What the learned slopes look like
The comparison uses model E of He and Sun (2014), 11 conv layers with mostly 2×2 filters followed by spatial pyramid pooling and three fully connected layers, trained for 75 epochs. With ReLU it reaches 33.82% top-1 and 13.34% top-5 error on the ImageNet 2012 validation set, using 10-view testing (the four corners and the center of the image plus their mirror images, scores averaged). Replacing every ReLU with PReLU and retraining from scratch gives:
| 14-layer model, 10-view | top-1 | top-5 | extra parameters |
|---|---|---|---|
| ReLU | 33.82 | 13.34 | 0 |
| PReLU, channel-shared | 32.71 | 12.87 | 13 |
| PReLU, channel-wise | 32.64 | 12.75 | 10,304 |
That is 1.18 points of top-1 error for the channel-wise version and 1.11 points for the channel-shared one, so almost all of the gain comes from the 13 per-layer slopes. The strip in Figure 1 shows two patterns in the learned values. The first conv layer keeps a large slope on its negative side, 0.681 shared and 0.596 averaged over channels. Its 64 filters are mostly edge and texture detectors, and the paper's reading is that the layer uses both the positive and the negative response of each filter (an edge and the same edge with opposite polarity), which gets more out of a small number of filters. Deeper layers learn smaller slopes, down to 0.008 for conv35 in the shared version and 0.031 to 0.075 in the fully connected layers. The paper reads this, without testing it, as the network keeping more of each signal early and becoming more nonlinear, closer to a ReLU, near the classifier.
Why deep rectifier networks were hard to start
Before training, every weight is set to a random number, usually drawn from a zero-mean Gaussian. In a deep network the standard deviation of that Gaussian decides whether training starts. AlexNet used a fixed 0.01 for every layer. The VGG team reported that networks deeper than about 8 conv layers had difficulty converging from such a start, which is why they pre-trained the shallow configuration and copied it into the deep ones; the paper notes that this costs extra training time and may land in a worse local optimum. GoogLeNet added auxiliary classifiers in the middle of the network to help it converge.
The scale problem compounds. Suppose each layer multiplies the size of the signal passing through it by a factor . After layers the signal has been multiplied by . With and 30 layers that is 0.04; with it is 17. The same holds for the gradient going backward. If the factor is below 1 the early layers receive almost no gradient and learning stalls; if it is above 1 the output grows until it overflows. Section 2.2 describes these two failure modes: with the algorithm diverges, with it stalls.
Glorot and Bengio (2010) had derived an initialization with by tracking the variance of each layer's output. Their derivation assumes the activation function is linear around zero, which is roughly true for tanh, one of the activations they studied, and it arrives at , where is the number of inputs to a unit. Caffe calls this “Xavier” initialization, after Glorot's first name, and the paper uses that name. A ReLU is not linear around zero. It is linear on one side and zero on the other, and the next section shows that this changes the factor by exactly 2.
How He initialization works
Take one conv layer and one output position. The response is a matrix-vector product:
Here stacks the pixels around the position in all input channels into one vector of length , the number of inputs each output sees (its fan-in). Each of the rows of is one filter, so is and holds the filter responses at that pixel. The input to layer is the previous layer's output after the activation, , and the number of input channels is the previous layer's number of filters, . A 3×3 conv reading 256 channels has .
The derivation treats the weights and inputs as random variables: the weights independent of each other and of the inputs, all drawn from one distribution with mean zero, and the inputs also independent and identically distributed. Each response is then a sum of independent products , and the variance of a sum of independent terms is the sum of their variances:
For independent and with , the variance of the product is
so (6) becomes
The factor is , the mean of the square, and it equals only when has mean zero. Glorot and Bengio's derivation could use the variance because a tanh output has mean zero. A ReLU output is never negative, so its mean is positive and its variance is smaller than its mean square.
If the previous layer's weights are symmetric around zero and its biases are zero, then is symmetric around zero too: a value and its negative are equally likely. The ReLU keeps the positive half and replaces the negative half with zeros, so the square of its output is on half the distribution and 0 on the other half. For any distribution symmetric around zero that gives
A million standard normal samples show the half, and that the variance of the ReLU output is smaller still:
y = torch.randn(1_000_000) # zero mean, symmetric, Var[y] = 1
x = y.clamp(min=0) # ReLU
print(x.pow(2).mean()) # 0.500 E[x^2] = Var[y] / 2
print(x.mean(), x.var()) # 0.399, 0.341 x is not zero-meanSubstituting the half into (7) gives the per-layer recursion:
and chaining it from layer 2 to layer :
Each factor in the product is the of the previous section for one layer. If every factor is 1, the variance of the last layer equals the variance of the first, which is the condition the paper adopts:
That is a zero-mean Gaussian with standard deviation , and biases set to zero. For the 3×3 conv over 256 channels, , where the linear-case rule gives . The first layer reads the image, which has not been through a ReLU, so strictly it should use ; the paper uses (10) there too, because a factor of 2 at a single layer does not compound.
Figure 2 runs the recursion on actual random networks. Each is 30 fully connected layers of width (a fully connected layer is the case of (5)), with weights of standard deviation for a gain . Since a rectifier network without biases satisfies , changing rescales layer by exactly , so the figure computes each network once and the slider is exact. Look at where the teal line sits at layer 30 for and for .
At , the Xavier setting, every ReLU layer halves the variance, so the standard deviation falls by per layer: at layer 30 it is of the input's, of what gives, and the gradient reaching layer 1 has shrunk by as well. At both lines stay level. The individual networks scatter around the average by up to about a factor of 2 after 30 layers, because a width of 128 is a small sample of the distribution the derivation averages over; a wider network scatters less. The slider's ends, 0.5 and 2, show the stall and the explosion: a factor of about or across 30 layers.
For PReLU, the negative half is scaled by instead of zeroed, so its square contributes times as much, and the mean square of the output is the number in the header of Figure 1:
The condition becomes
with the initial slope. At this is (10), and at , where the activation is the identity, it is the linear rule . At the paper's initial the correction is small, : using the ReLU rule for a PReLU network grows the variance by 1.0625 per layer, about 6 times over 30 layers (2.4 times in standard deviation), which the PReLU tab of Figure 2 shows at .
In PyTorch, kaiming_normal_ with a=0.25 and nonlinearity='leaky_relu' draws standard deviation , which is (15); with a=0 or nonlinearity='relu' it is (10):
import torch, torch.nn as nn
conv = nn.Conv2d(256, 256, kernel_size=3, padding=1) # fan-in n = 9 * 256 = 2304
act = nn.PReLU(num_parameters=256, init=0.25) # one slope per channel
# eq. (15) with a = 0.25: std = sqrt(2 / ((1 + 0.25**2) * 2304)) = 0.0286
nn.init.kaiming_normal_(conv.weight, a=0.25, mode='fan_in',
nonlinearity='leaky_relu')
nn.init.zeros_(conv.bias) # b = 0, as in the paper
# the paper's own runs used eq. (14): mode='fan_out' divides by 9 * 256 too here,
# because this layer has as many filters (d) as input channels (c)
opt = torch.optim.SGD([
{'params': conv.parameters(), 'weight_decay': 5e-4},
{'params': act.parameters(), 'weight_decay': 0.0}, # no decay on the slopes
], lr=1e-2, momentum=0.9)
x = torch.randn(32, 256, 56, 56) # batch 32, 256 channels, 56 x 56
y = act(conv(x)) # same shape; act.weight has shape (256,)The backward pass: fan-in or fan-out
Gradients flow through the same weights in the opposite direction. Write and for the gradients of the loss with respect to and . At one pixel,
now stacks pixels in all output channels, a vector of length , because each input pixel was read by positions of each of the filters. is the same set of weights rearranged into a matrix, and has one entry per input channel. So the backward pass sums over terms, the fan-out, where the forward pass summed over , the fan-in.
The activation enters through its derivative: . For a ReLU, is 1 or 0 with equal probability, because is symmetric around zero. Assuming independent of , the mean square of is half that of , and since gradients have mean zero (the weights are symmetric), mean square and variance agree. The variance of (11) then follows the same steps as (6) to (8):
The half has a different origin here, the probability that the ReLU passes gradient rather than the mean square of its output, and the same value. Chaining from the top of the network down to layer 2:
and the condition that keeps gradients from shrinking or growing is
a Gaussian with standard deviation . Equations (10) and (14) differ only in which count divides: the number of input channels or the number of filters . When a layer has as many filters as input channels they coincide.
One condition is enough. Suppose the network uses (14). Then the backward product in (13) is exactly 1, and each factor of the forward product in (9) becomes
Multiplying these from layer 2 to , each layer's filter count cancels against the next layer's input count, and the product telescopes to
the second layer's input channels over the last layer's filters. For a typical network that widens from 64 to 512 channels, that is 1/8 in variance, a fixed ratio that does not grow with depth. The same argument run the other way shows (10) leaves the backward product at . The paper trains all its models with (14). PyTorch's kaiming_normal_ and Caffe's MSRAFiller default to fan-in, (10), with fan-out available as an option, and torchvision's ResNet selects mode='fan_out'.
Why a constant std of 0.01 stalls
The derived standard deviation depends on the layer's width, so a single constant cannot be right for every layer of a network whose width changes. The paper works through VGG's model B: 10 conv layers, all 3×3, with 64 filters in layers 1 and 2, 128 in layers 3 and 4, 256 in 5 and 6, and 512 in 7 to 10. By (14), with , the right standard deviations are 0.059, 0.042, 0.029 and 0.021 for 64, 128, 256 and 512 filters. A constant 0.01 is too small everywhere, by a factor of 5.9 in layer 2 and 2.1 in each of layers 7 to 10. The gradient reaching layer 2 from layer 10 passes through nine layers, and each shrinks its standard deviation by its own factor:
With unrounded factors it is 1/16,729. At that scale the loss contributes almost nothing to the weight update: footnote 2 of the paper notes that with weight decay switched on, the update is then made almost entirely of the decay term, and suggests checking for exactly that as a diagnosis. Figure 3 recomputes the product for any constant and compares the two equations.
At 0.01 the amber line falls by more than four orders of magnitude. Raising the constant to about 0.0295 brings the end-to-end product back to 1, but only by balancing layers that shrink the gradient against layers that grow it, as the per-layer row shows; at 0.1 the gradient grows about 60,000-fold. Equation (14) gives a flat amber line and a teal line that ends at 0.354, the square root of the 1/8 from the previous section; equation (10) swaps the two.
The fully connected layers are the paper's exception. It uses a standard deviation of 0.01 for the first two and 0.001 for the last, smaller than the derived . Because the initialization preserves variance from the first layer to the last, an input image with pixel values in roughly would reach the softmax with logits large enough to overflow it. Shrinking a few layers' weights scales the output down without renormalizing the input, which would have changed other hyperparameters.
He versus Xavier at 22 and 30 layers
Xavier's in a ReLU network sets each factor of (9) to 1/2. After layers the signal's standard deviation is of what (10) gives: about 1/2,048 at 22 layers and 1/32,768 at 30. The paper trained networks of both depths with each rule.
On the 22-layer model B, described below, both initializations converge. Figure 2 of the paper plots top-1 error on 3,000 validation images over training: with (14) the error starts falling earlier. On the 14-layer model from Table 2, Xavier gives 33.90/13.44 top-1/top-5 error and (14) gives 33.82/13.34.
The 30-layer model is the 14-layer model with sixteen extra conv layers of 256 2×2 filters, 27 conv and 3 fully connected layers in all. With Xavier initialization it does not train: the error does not fall and the gradients, which the authors monitored, shrink toward zero. With (14) it converges. Its final error, though, is 38.56/16.59, worse than the 33.82/13.34 of the 14-layer model it was built from. The paper also ran model B with 3 to 9 extra layers and found both training and test error degraded in the first 20 epochs (it did not finish those runs). This is degradation, a deeper network training to a worse result, and the paper offers two guesses for it: that adding layers this way is the wrong way to add depth, or that ImageNet classification is not complex enough to need it. Ten months later He, Zhang, Ren and Sun addressed the first with residual networks, which initialize their weights with this paper's method and reach 152 layers on ImageNet.
The ImageNet models
Because extra depth was not paying off, the paper made its large models wider instead. Model A starts from VGG-19 with three changes: the first layer is a single 7×7 conv with stride 2 and 96 filters; the other three conv layers that VGG-19 runs on its two largest feature maps (224 and 112) are moved to the 56, 28 and 14 maps; and a spatial pyramid pooling (SPP) layer before the first fully connected layer pools the last feature map into 7×7, 3×3, 2×2 and 1×1 grids of bins, 63 bins in total, so the network accepts any input size. The compute is about the same as VGG-19's (1.90 against 1.96 × 10¹⁰ operations), and A runs a mini-batch of 128 in 2.6 seconds on four Nvidia K20 GPUs against 3.0 seconds for VGG-19, because the moved layers run on smaller maps. Model B adds three conv layers (22 layers). Model C is B with 384, 768 and 896 filters in its three stages instead of 256, 512 and 512, at 5.30 × 10¹⁰ operations, about 2.3 times B.
Training mostly follows AlexNet and VGG. Each training image is resized so its shorter side is a random , and a random 224×224 crop is taken, with half of the crops mirrored and the colors randomly altered. VGG applied this scale jittering only while fine-tuning; here it runs from the first epoch, and the deep models train end to end from the initialization of (14) instead of from a shallower network. The optimizer is SGD with momentum 0.9 and weight decay 0.0005, mini-batch 128, dropout of 50% in the first two fully connected layers, and a learning rate of 0.01, then 0.001, then 0.0001, each switched when the error plateaus, for about 80 epochs. That took 3 to 4 weeks for A or B on four K20s and for C on eight K40s. The convolutional layers are data-parallel across GPUs and the fully connected layers run on one GPU, giving speedups of 3.8× on 4 GPUs and 6.0× on 8.
At test time the conv layers run once on the whole resized image. Every 14×14 window of the last feature map is pooled by the SPP layer and classified, the scores of all windows are averaged, the same is done for the mirrored image, and the results at several image scales are combined.
Results on ImageNet
ImageNet 2012 has about 1.2 million training images in 1,000 classes, 50,000 validation images and 100,000 test images whose labels are held by the challenge server. The headline metric is top-5 error: the fraction of images whose true class is not among the model's five highest-scoring guesses. On model A, PReLU against ReLU, trained for the same epochs with the same learning-rate switches (Table 4):
| model A, test scale | ReLU top-1 | ReLU top-5 | PReLU top-1 | PReLU top-5 |
|---|---|---|---|---|
| 256 | 26.25 | 8.25 | 25.81 | 8.08 |
| 384 | 24.77 | 7.26 | 24.20 | 7.03 |
| 480 | 25.46 | 7.63 | 24.83 | 7.39 |
| multi-scale | 24.02 | 6.51 | 22.97 | 6.28 |
PReLU lowers top-1 error by 1.05 points and top-5 by 0.23 points in the multi-scale combination, and wins at every single scale. The best single scale, 384, is the middle of the training jitter range. On the same model family, the ReLU baseline's 6.51% is already below the best single-model result published before it, 7.1% for VGG-19; the paper attributes that gap mainly to training end to end without pre-training a shallow model. Figure 4 lines up the single-model chain and the ensembles.
In the single-model chain, PReLU takes model A from 6.51% to 6.28% top-5, three more layers (B) change it to 6.27%, and the wider model C reaches 5.71%, which the paper notes is below every multi-model result published before it. Depth from 19 to 22 layers bought nothing here, and width bought 0.56 points. With 10-view testing instead of dense multi-scale testing, C scores 7.38% (Table 5). The final entry combines six models, of which only one has architecture C, and scores 4.94% top-5 on the test set. That is 1.72 points, or 26% relative, below GoogLeNet's 6.66%, the ILSVRC 2014 winner, and 17% relative below Baidu's 5.98%, the best result published in the meantime.
The per-class breakdown in the paper's Figure 6 shows zero top-5 error on 113 of the 1,000 test classes. The worst three are letter opener (49%), spotlight (38%) and restaurant (36%), where images contain several objects, small objects, or very varied examples of the class. Against the same team's ILSVRC 2014 entry (8.06%), the error fell in 824 classes, stayed the same in 127 and rose in 49.
What the human-level claim means
The 5.1% human figure comes from Russakovsky et al. (2015), the paper that describes the ImageNet challenge. One annotator trained on 500 validation images, learning what the 1,000 classes are, and then labeled 1,500 test images through an interface that showed 13 example training images next to each class name. A second annotator, trained on 100 images, labeled 258 test images with 12.0% error. The study found the two made different mistakes: on the 204 images both labeled, counting an image as correct if either got it right gives 2.4%.
So 5.1% is one careful person's error on a sample of 1,500 images, and it has sampling error of its own: a binomial proportion of 5.1% on 1,500 trials has a 95% interval from about 4.0% to 6.2%. The ensemble's 4.94% sits 0.16 points below the point estimate and inside that interval, which the band in Figure 4 shows. Five days after this paper appeared, Ioffe and Szegedy's Batch Normalization reported 4.82% with an ensemble of Inception networks.
The paper's Section 4 makes the same qualification. Russakovsky et al. found that the human errors were mostly fine-grained (telling apart 120 dog breeds, or species of birds) and from not knowing a class existed; the model is good at the first kind, and the paper's Figure 4 shows it naming a coucal, a komondor and a yellow lady's slipper. It also fails on images a person finds easy, such as spotlights, where the answer depends on context. The authors write that the result “does not indicate that machine vision outperforms human vision on object recognition in general.”
Limitations and what came next
The derivation describes the network at initialization only. It assumes independent, identically distributed weights and inputs, zero biases and symmetric pre-activations, and its conclusions are about variances averaged over random weights; any single finite-width network deviates from it, as the thin lines in Figure 2 show, and training moves the weights away from these assumptions. It does not cover pooling, residual connections or normalization layers. Getting the scale right made 30-layer plain networks trainable without making them better, and PReLU's gains, about one point of top-1 error, are smaller than those of the wider model C.
Batch normalization, five days later, made the per-layer scale far less critical by standardizing each layer's pre-activations during training. ResNet, in December 2015, combined this initialization with batch normalization and identity shortcuts and trained 152 layers. ResNet kept the plain ReLU. The variance argument itself carried forward: later work such as μP asks the same question, which scale keeps each layer's signal of order one, for the updates during training as well as for the weights at initialization.
Questions you might still have
What is the difference between He initialization and Xavier initialization?
Both draw zero-mean weights with variance proportional to 1/n, where n is the number of inputs to a unit. Xavier (Glorot and Bengio, 2010) assumes the activation is linear near zero and uses variance 1/n (or 2/(n_in + n_out) in the original uniform form). He initialization accounts for the ReLU setting half of its inputs to zero, which halves the second moment of the signal, and doubles the variance to 2/n. In a ReLU network Xavier's rule therefore halves the signal's variance at every layer: after 30 layers its standard deviation is 2^-15, about 1/32,768, of what it would be under He initialization.
Should I use fan-in or fan-out?
Either works in a plain convolutional network. Fan-in (n = k²c, equation 10) keeps the forward activations at a constant scale; fan-out (n̂ = k²d, equation 14) keeps the backward gradients at a constant scale; the paper shows that choosing one leaves the other off by the ratio of the first layer's input channels to the last layer's filters (64/512 = 1/8 in variance for VGG's model B), which does not grow with depth. The paper trained with fan-out. PyTorch's kaiming_normal_ defaults to fan-in; torchvision's ResNet uses fan-out.
Does PyTorch use He initialization by default?
Not exactly. nn.Conv2d and nn.Linear default to kaiming_uniform_ with a = √5, which gives weight variance 1/(3n), a sixth of He's 2/n. Many model definitions, including torchvision's ResNet, overwrite that with kaiming_normal_. With batch normalization after every convolution the scale of the initialization matters much less, because each layer's output is renormalized.
Does He initialization have to be Gaussian?
No. The derivation only fixes the variance, 2/n, and any zero-mean symmetric distribution with that variance satisfies it. A uniform distribution on [-b, b] has variance b²/3, so the uniform version uses b = √(6/n); PyTorch's kaiming_uniform_ computes its bound as √3 times the Gaussian standard deviation. The paper uses a Gaussian; Glorot and Bengio's original rule is uniform.
Is PReLU the same as Leaky ReLU?
They have the same shape, a line of slope a for negative inputs. In Leaky ReLU (Maas et al., 2013) a is fixed at 0.01. In PReLU a is a parameter trained by backpropagation along with the weights, one per channel or one per layer, initialized to 0.25 and excluded from weight decay. The paper's learned values range from 0.008 to 0.681 (Table 1).
How much does PReLU help?
Modestly. On the 14-layer model, top-1 error falls from 33.82% to 32.64% (channel-wise) or 32.71% (channel-shared, 13 extra parameters). On the large model A it falls from 24.02% to 22.97% top-1 and from 6.51% to 6.28% top-5. The compute cost is one multiply on the negative inputs.
Did He initialization make very deep plain networks better?
It made them trainable, not better. The 30-layer model that stalls with Xavier initialization converges with He initialization, but reaches 38.56% top-1 error against 33.82% for the 14-layer model it was built from. The paper reports that training error also got worse when it added layers to model B. That degradation is the problem residual networks solved ten months later, using this paper's initialization (explained in the ResNet explainer).
Why does the derivation assume zero biases and symmetric weights?
With zero-mean symmetric weights and zero bias, each pre-activation y is a sum of many terms with mean zero and a distribution symmetric about zero. That symmetry gives exactly half of y's variance above zero, so a ReLU output has E[x²] = Var[y]/2. A nonzero bias would shift y and change that fraction. The analysis holds at initialization only; training moves the weights away from these assumptions.
Does initialization still matter with batch normalization?
Much less. Batch normalization, published five days after this paper, standardizes each layer's pre-activations over the mini-batch during training, which undoes any per-layer scale the initialization got wrong. He initialization is still the usual choice for the weights, and it matters again in networks without normalization layers.
What were the models, and how long did they take to train?
Model A is a 19-layer VGG-style network with a 7×7 stride-2 first layer and spatial pyramid pooling before the classifier; B adds three conv layers (22 layers); C is B with more filters per layer (384 to 896), about 2.3 times B's compute. Each trained for about 80 epochs with SGD, batch 128, weight decay 0.0005: about 3 to 4 weeks for A or B on four Nvidia K20 GPUs and for C on eight K40s.
Footnotes & further reading
- The paper: He, Zhang, Ren & Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (ICCV 2015). The authors did not release code; the paper implements PReLU and the initialization in its own modification of Caffe.
- The linear-case initialization: Glorot & Bengio, Understanding the difficulty of training deep feedforward neural networks (AISTATS 2010). Leaky ReLU: Maas, Hannun & Ng, Rectifier nonlinearities improve neural network acoustic models (ICML 2013 workshop).
- The networks it builds on: Simonyan & Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition (explained in VGG); Szegedy et al., Going Deeper with Convolutions (explained in GoogLeNet); He, Zhang, Ren & Sun, Spatial Pyramid Pooling in Deep Convolutional Networks (SPP-net); He & Sun, Convolutional Neural Networks at Constrained Time Cost (model E, the 14-layer comparison model).
- The human estimate: Russakovsky et al., ImageNet Large Scale Visual Recognition Challenge (IJCV 2015), Section 6.4.
- What came next: Ioffe & Szegedy, Batch Normalization (explained in Batch Normalization); He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition (explained in ResNet).
- Implementations: PyTorch's torch.nn.init (
kaiming_normal_,kaiming_uniform_,calculate_gain) and nn.PReLU; Caffe's filler.hpp (MSRAFiller,XavierFiller).
How could this explainer be improved? Found an error, or something unclear? I read every message.