ResNet: Deep Residual Learning for Image Recognition

Add each block's input to its output, and deeper networks train better instead of worse.

In 2015 a 34-layer plain convolutional network fit its own training images worse than an 18-layer one. He, Zhang, Ren and Sun added a shortcut that carries each block's input past two or three layers and adds it to their output. With it they trained a 152-layer network that won the ImageNet 2015 classification challenge.

Adding shortcuts to that 34-layer network, with no other change, cut its ImageNet top-1 error from 28.54% to 25.03%. The layers, the weight count and the compute stayed the same.

In 2015 the best ImageNet classifiers were 16 to 30 layers deep, and the paper opens with the obvious next question: is learning a better network as easy as stacking more layers? Getting deep networks to start training was no longer the obstacle. Careful initialization and batch normalization (which rescales each layer's outputs using the statistics of the current mini-batch) let networks with tens of layers start converging under stochastic gradient descent. What the authors found instead was that a deeper plain network ended up with higher error than a shallower one, on the training set as well as the test set.

The page follows the paper's argument in order: what that failure is and what it is not; why it shows an optimization problem rather than a limit on what the network can represent; the residual block, which adds each block's input to its output; why that makes the right answer easier to reach; how the shortcut works when channel counts and map sizes change; the bottleneck block that makes 50 to 152 layers affordable; the full architecture and training recipe; and the results. A last section covers the analyses that came after the paper, including the gradient argument most people associate with ResNets.

Deeper plain networks trained worse

A "plain" network here is a straight stack of convolutional layers with no shortcuts, built in the style of VGG: mostly $3\times3$ filters, the same number of filters for every layer that outputs the same map size, and the filter count doubled whenever the map is halved. The authors trained such networks at several depths. In the paper's words, as depth grows "accuracy gets saturated (which might be unsurprising) and then degrades rapidly." The paper names this the degradation problem.

The paper gives two measurements. On CIFAR-10 (60,000 small 32x32 images in 10 classes), a 56-layer plain network has higher training error and higher test error than a 20-layer one. On ImageNet, the 34-layer plain network has higher training error than the 18-layer one throughout training (the ImageNet data is the 2012 classification set: 1.28 million training images in 1000 classes, with 50,000 validation images). Its final validation error is also higher: 28.54% top-1 against 27.94%. Top-1 error is the fraction of images for which the single highest-scoring class is wrong.

The training error rules out overfitting. An overfit model fits its training data too closely and generalizes badly, so its training error is low and its test error high. Here the deeper model is worse at fitting the very images it trains on.

The paper also argues that vanishing gradients, the older reason deep networks were hard to train, are unlikely to be the cause. Every plain baseline uses batch normalization after each convolution, which keeps the forward signal at non-zero variance. The authors also checked the backward pass and report that the gradients "exhibit healthy norms." Training longer does not close the gap either: with 3x the iterations the degradation was still there. The paper's conclusion is that the deeper plain net is harder for SGD to optimize. It offers a conjecture for why, that deep plain nets "may have exponentially low convergence rates", and says the reason "will be studied in the future."

The figure redraws the ImageNet comparison. In plain mode, watch where the two curves end: the 34-layer net finishes above the 18-layer one. Switch to residual and the order flips.

A deeper net can copy a shallower one

The paper argues that degradation should not happen, by construction. Take a trained 18-layer network. Build a 34-layer one by copying the 18 layers and inserting 16 more that each compute the identity, output equal to input. The deeper network computes exactly the same function and has exactly the same training error. So among the weight settings of the 34-layer network there is at least one as good as the 18-layer network, and a good optimizer should find something no worse.

SGD does not find it, so degradation is a failure of search, not of capacity: the deeper network can represent a solution at least as good as the shallower one, and training does not reach it. The construction also points at the layers the optimizer would have to get right, the ones computing the identity. The paper's hypothesis is that "solvers might have difficulties in approximating identity mappings by multiple nonlinear layers."

A single layer shows how specific the identity is. For a $3\times3$ convolution from 64 channels to 64 to pass its input through unchanged, its 36,864 weights must take one exact pattern: the 64 centre-tap weights that connect each channel to itself equal to 1, and the other 36,800 equal to 0. The batch norm after it must undo its own normalization, and the ReLU must not clip anything that matters. A stack of such layers has to reach that point jointly, from a random initialization.

Zero is the easier point to reach. Pushing all 36,864 weights toward 0 is a direction SGD with weight decay already moves in (the paper's decay is 0.0001). The residual block is designed so that zero weights produce the identity.

The residual block

Call the mapping a few stacked layers should compute $\mathcal{H}(\mathbf{x})$. Instead of asking the layers to produce $\mathcal{H}(\mathbf{x})$, the paper asks them for the difference between it and the input:

and adds the input back afterwards. The block's output is

where $\mathbf{x}$ is the block's input, $\mathbf{y}$ its output, and $\mathcal{F}$ the residual function computed by the block's own weight layers $W_i$. For the two-layer block in the paper's Figure 2,

with $\sigma$ the ReLU and biases left out of the notation. In the convolutional networks each $W_i$ is a $3\times3$ convolution followed by batch norm. The $+\,\mathbf{x}$ is a shortcut connection: a wire that skips the weight layers and adds the untouched input to their output, element by element, channel by channel. A second ReLU is applied after the addition, so the block returns $\sigma(\mathbf{y})$.

The residual form can compute anything the plain form can. Whatever $\mathcal{H}$ a plain block would learn, the residual block gets by learning $\mathcal{F} = \mathcal{H} - \mathbf{x}$. The two forms differ in what zero weights produce. A plain block with zero weights outputs zero. A residual block with zero weights outputs $\sigma(0 + \mathbf{x})$, and since every block's input is itself the output of a ReLU (so it has no negative entries), that is exactly $\mathbf{x}$: the block is the identity. As an analogy, a residual block stores an edit to its input rather than a rewritten copy, and a block with nothing to change stores an empty edit. The comparison is loose in one place: the ReLU after the addition means a nonzero edit is not applied purely additively.

In code the two blocks differ by one term. The shapes are those of a block in the first stage of ResNet-34, on a 56x56 map with 64 channels:

# plain block: the two conv layers produce the whole output
def plain(x):                      # x: (N, 64, 56, 56)
    out = relu(bn1(conv1(x)))      # 3x3 conv, 64 -> 64, then BN, relu
    out = bn2(conv2(out))          # 3x3 conv, 64 -> 64, then BN
    return relu(out)               # H(x): (N, 64, 56, 56)

# residual block: the same layers, plus "+ x" before the last relu
def residual(x):                   # x: (N, 64, 56, 56)
    out = relu(bn1(conv1(x)))      # identical layers ...
    out = bn2(conv2(out))          # ... now computing F(x)
    return relu(out + x)           # F(x) + x, then relu: (N, 64, 56, 56)

The shortcut adds no parameters. Its cost is the addition: 64 x 56 x 56 = 200,704 sums per image, next to 231 million multiply-adds in the two convolutions (73,728 weights, each applied at 3,136 positions). That lets the paper compare plain and residual networks with identical depth, width, parameter count and compute, which is how the 18- and 34-layer comparisons in Table 2 are set up.

Toggle between the two blocks below. The weight layers are the same in both; in residual mode the identity shortcut bows around them and meets their output at the addition.

The paper keeps $\mathcal{F}$ at two or three layers. With a single layer, (1) becomes

which is similar to a single linear layer, and the authors report no advantage from it.

Why the residual is easier to learn

The paper states the claim as a hypothesis: "it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping." It supports the claim with experiments, not a proof. The extreme case is the one from the construction above: if the identity were the optimal mapping for some block, the solver would only have to drive $\mathcal{F}$'s weights to zero.

The paper expects that in real networks the identity is unlikely to be exactly optimal, and makes a narrower claim: the reformulation "may help to precondition the problem": if the optimal function is closer to the identity than to zero, it is easier for the solver to learn a small correction relative to the identity than to learn the function from scratch. Preconditioning means rewriting a problem so the same solver converges faster. The paper points to numerical methods for partial differential equations, such as multigrid, which solve for residuals between coarse and fine scales and converge much faster than solvers that ignore that structure.

The same condition also says when residual learning does not help, and the figure shows both cases. The slider moves one block's ideal mapping from the identity (plus a small wiggle) to the zero function (plus the same wiggle). Each panel shades how far its layers must move away from zero output: the whole of $\mathcal{H}$ for the plain block, $\mathcal{H} - \mathbf{x}$ for the residual block. Start at the left end, then drag to the right end and watch the verdict line flip.

The argument therefore depends on an empirical premise: that in a deep network most blocks should compute something close to the identity, refining the representation they receive rather than replacing it. The next section shows the measurement the paper offers in support.

Residual layers respond less

If residual blocks learn small corrections, their weight layers should produce small outputs. The paper measures this on CIFAR-10. For every $3\times3$ layer, it takes the layer's output after batch norm and before the ReLU or addition, and records its standard deviation. For a ResNet this is the size of the residual function $\mathcal{F}$ at that layer. Two results come out: ResNets have "generally smaller responses than their plain counterparts", and a deeper ResNet has smaller responses still (ResNet-110 below ResNet-56 below ResNet-20). The authors read this as each layer of a deeper ResNet modifying the signal less.

The figure is traced by hand from the paper's Figure 7, so individual values are approximate. Start on 56 layers: over the first 30 layers the dashed plain-56 trace runs near 2 while ResNet-56 runs near 1. In layer order the two cross around layer 30, where plain-56 drops to about 1; switch to sorted by size, the paper's second panel, and past the first few ranks each plain trace sits above its ResNet counterpart.

The medians under the figure summarize the traces: at 20 layers the plain net's median response is about 1.9 and the ResNet's about 1.55; at 56 layers, about 1.65 and 1.1; ResNet-110 sits near 0.95. The paper takes this as support for its motivation that "the residual functions might be generally closer to zero than the non-residual functions."

When shapes change: options A, B, C

Equation (1) adds $\mathcal{F}(\mathbf{x})$ to $\mathbf{x}$ elementwise, which requires the two to have the same shape. Within a stage of ResNet-34 they do. Between stages they do not: the first block of each new stage doubles the channels and halves the map, for example from 64 channels at 56x56 to 128 channels at 28x28, using a stride-2 convolution. For those blocks the paper allows a linear projection on the shortcut:

where $W_s$ is a $1\times1$ convolution (stride 2 when the map shrinks) that maps the input to the output's channel count. A square $W_s$ could also be used where shapes already match, and the paper tests whether that helps. It compares three options on ResNet-34:

• A: every shortcut stays parameter-free. Where the shape changes, the shortcut subsamples the input with stride 2 and pads the extra channels with zeros.
• B: projections on the three shape-changing shortcuts, identity on the other 13.
• C: projections on all 16 shortcuts.

The figure draws ResNet-34's 16 blocks with each shortcut above its block, and Table 3's top-1 errors below. Click a block (or step with the block buttons) to see its shapes; the fourth block is the first one that changes shape.

B beats A by 0.51 points (24.52% against 25.03%). The paper attributes this to the zero-padded channels: in a block where 64 channels become 128, the 64 new channels receive $\mathcal{F}$ with nothing added, so no residual learning happens on them. C beats B by another 0.33 points (24.19%), which the paper attributes to the extra parameters of the thirteen additional projections. Counting weights from Table 1, those projections add 913,408 weights on top of B's 172,032, against about 21.6 million weights in the option-A network.

The paper's reading is that "projection shortcuts are not essential for addressing the degradation problem." It drops option C for the rest of the paper to save memory, time and model size, and uses option B for ResNet-50, 101 and 152. The saving matters most in the bottleneck block, covered next.

The bottleneck block

To go past 34 layers at an affordable cost, the paper changes the block. Its footnote gives the reason as practical: deeper networks built from the basic two-layer block also gain from depth (the CIFAR-10 results show it), but they are less economical. The bottleneck block replaces the two $3\times3$ convolutions with three layers: a $1\times1$ convolution that reduces the channel count, a $3\times3$ convolution at the reduced width, and a $1\times1$ convolution that restores it. A $1\times1$ convolution mixes channels at each pixel without looking at neighbours, so it is cheap: its weight count is input channels times output channels.

The paper's Figure 5 compares a basic block on a 64-channel signal with a bottleneck block on a 256-channel signal and says the two have "similar time complexity." Counting weights:

• basic: two $3\times3$ convs, 64 to 64: 2 x 36,864 = 73,728 weights;
• bottleneck: $1\times1$ from 256 to 64 (16,384), $3\times3$ from 64 to 64 (36,864), $1\times1$ from 64 to 256 (16,384): 69,632 weights.

On a 56x56 map each weight is used 3,136 times, so the two blocks cost about 231 and 218 million multiply-adds. The bottleneck carries a signal four times as wide for slightly less compute.

A bottleneck's shortcut joins the two 256-channel ends of the block, so a projection there would be a $1\times1$ convolution from 256 to 256: 65,536 weights, almost as many as the block itself. The paper says a projection here would double both time complexity and model size. Toggle the shortcut in the figure to see the count go from 69,632 to 135,168, then do the same for the basic block, where a 64-to-64 projection adds 4,096 weights, about 6%.

Replacing each two-layer block of ResNet-34 with a three-layer bottleneck gives ResNet-50, at 3.8 billion multiply-adds against ResNet-34's 3.6 billion. The paper reports that plain networks built from bottleneck blocks also show the degradation problem, so the bottleneck saves compute but does not by itself remove the need for shortcuts.

The full network and its training

All five ImageNet ResNets in the paper's Table 1 share one frame. A $224\times224$ image goes through a $7\times7$ convolution with 64 filters and stride 2 (to 112x112), then a $3\times3$ max pool with stride 2 (to 56x56). Four stages of residual blocks follow, called conv2_x to conv5_x, on 56x56, 28x28, 14x14 and 7x7 maps. The first block of stages 3, 4 and 5 downsamples with stride 2. A global average pool then reduces each channel to one number, and a 1000-way fully connected layer produces the class scores.

The depths differ in two things: the block type (basic for 18 and 34, bottleneck for 50, 101 and 152) and the number of blocks per stage. The name counts weight layers: the first convolution, two or three layers per block, and the final fully connected layer. For ResNet-152 that is 1 + 3 x (3 + 8 + 36 + 3) + 1 = 152. Step through the depths in the figure: of the 34 blocks added from ResNet-50 to ResNet-152, 30 go into the 14x14 stage, which grows from 6 blocks to 23 to 36.

ResNet-152 costs 11.3 billion multiply-adds, less than VGG-16 (15.3 billion) and VGG-19 (19.6 billion), while being 8 times deeper than the 19-layer VGG. (The paper counts multiply-add operations and calls them FLOPs; all the compute figures on this page use that unit.) The plain and residual 34-layer networks cost 3.6 billion, 18% of VGG-19; the paper notes they have fewer filters than VGG.

The code below is a sketch of ResNet-50 with tensor shapes, following the official Caffe definition. That definition puts the stride of a downsampling bottleneck on its first $1\times1$ convolution and on the projection, and follows the projection with batch norm. torchvision's implementation moves the stride to the $3\times3$ convolution and calls that variant ResNet v1.5.

# bottleneck block with an option-B shortcut, laid out as in the
# official Caffe ResNet-50 (stride on the first 1x1 conv)
def bottleneck(x, c, stride):            # x: (N, c_in, H, W)
    out = relu(bn(conv1x1(x, c, stride)))  # squeeze to c channels
    out = relu(bn(conv3x3(out, c)))        # (N, c, H/stride, W/stride)
    out = bn(conv1x1(out, 4 * c))          # restore to 4c channels
    if stride != 1 or x.shape[1] != 4 * c:
        x = bn(conv1x1(x, 4 * c, stride))  # projection W_s, only here
    return relu(out + x)                   # (N, 4c, H/stride, W/stride)

def resnet50(img):                       # img: (N, 3, 224, 224)
    x = relu(bn(conv7x7(img, 64, stride=2)))   # (N, 64, 112, 112)
    x = maxpool3x3(x, stride=2)                # (N, 64, 56, 56)
    for c, n, s in [(64, 3, 1), (128, 4, 2), (256, 6, 2), (512, 3, 2)]:
        for i in range(n):
            x = bottleneck(x, c, s if i == 0 else 1)
    x = x.mean(axis=(2, 3))                    # (N, 2048, 7, 7) -> (N, 2048)
    return fc(x, 1000)                         # logits: (N, 1000)

The ImageNet training recipe, from §3.4: each image is resized so its shorter side is a random length between 256 and 480 pixels, and a random 224x224 crop (or its mirror image) is taken, with the per-pixel mean subtracted and standard color augmentation. Batch norm follows every convolution, before the activation. Weights use He initialization (random Gaussian weights with variance 2 divided by the number of inputs to the unit, from the authors' earlier PReLU paper). Training is SGD with mini-batches of 256, momentum 0.9 and weight decay 0.0001, starting at learning rate 0.1 and dividing it by 10 whenever the error plateaus, for up to 600,000 iterations. There is no dropout. For the comparisons, each test image is scored as the average over 10 crops.

The CIFAR-10 networks are smaller versions of the same plan. After a first $3\times3$ convolution come three stages of $2n$ layers on 32x32, 16x16 and 8x8 maps with 16, 32 and 64 filters, then average pooling and a 10-way fully connected layer: $6n+2$ layers in all, so $n$ = 3, 5, 7, 9, 18 and 200 give 20, 32, 44, 56, 110 and 1202 layers. Every shortcut is option A, so each ResNet has exactly the same parameter count as its plain counterpart. Training uses mini-batches of 128 on two GPUs, learning rate 0.1 divided by 10 at 32,000 and 48,000 iterations, and stops at 64,000. ResNet-110 needed one change: it starts at learning rate 0.01 until training error drops below 80% (about 400 iterations), then continues at 0.1.

Results

Table 2 shows the reversal on ImageNet (10-crop top-1 error on the validation set). The plain network gets worse from 18 to 34 layers, 27.94% to 28.54%. The residual network with the same layers gets better, 27.88% to 25.03%. At 34 layers the residual net is 3.5 points ahead of the plain one; ResNet-34 is 2.8 points ahead of ResNet-18. At 18 layers the two are about equal, and the ResNet reaches that accuracy faster. The paper's reading is that at 18 layers SGD still finds a good solution for the plain net, and the shortcuts matter more as depth grows.

Deeper ResNets kept improving. On the same protocol (Table 3), ResNet-50, 101 and 152 reach 22.85%, 21.75% and 21.43% top-1 error, and 6.71%, 6.05% and 5.71% top-5 error (the fraction of images whose correct class is not among the five highest-scoring ones). The paper reports no degradation at these depths. With a stronger test procedure (the network run fully convolutionally at five image scales, scores averaged), a single ResNet-152 reaches 4.49% top-5 error on the validation set, lower than any previous ensemble result the paper lists. An ensemble of six ResNets of different depths, two of them 152-layer, reaches 3.57% top-5 error on the ImageNet test set. That entry won first place in the ILSVRC 2015 classification task.

The CIFAR-10 experiments reach further in depth. Plain networks there degrade too: the training error rises from 20 to 56 layers, and a plain 110-layer network has error above 60%. ResNets improve steadily: 8.75%, 7.51%, 7.17%, 6.97% and 6.43% test error at 20, 32, 44, 56 and 110 layers. ResNet-110 was trained five times and is reported as best and (mean ± std): 6.43 (6.61 ± 0.16). A 1202-layer ResNet trains without difficulty, to under 0.1% training error, but its test error is 7.93%, worse than ResNet-110's. The two networks have similar training error, so this is not degradation. The authors attribute it to overfitting: 19.4 million parameters for 50,000 training images, with no dropout or other strong regularization.

The features transferred to other tasks. Swapping VGG-16 for ResNet-101 inside the Faster R-CNN detector, with everything else fixed, raised COCO detection accuracy (mean average precision, averaged over box-overlap thresholds from 0.5 to 0.95) from 21.2 to 27.2, a 6.0-point gain, or 28% relative. ResNet-based entries also won first place in ImageNet detection, ImageNet localization, COCO detection and COCO segmentation in the 2015 competitions.

What came after

The best-known explanation of why ResNets train well is not in this paper. Three months later the same authors published Identity Mappings in Deep Residual Networks, which analyzes a stack of residual units in which nothing follows the addition, so that each unit computes

Unrolling from an early unit $l$ to a later unit $L$ gives their Eq. (4):

and differentiating a loss $E$ gives their Eq. (5):

The $1$ is the shortcut's contribution: the gradient at the deep unit reaches the shallow unit unmodified, without passing through any weight layer. The second term would have to equal exactly $-1$ for every example in a mini-batch to cancel it, which the follow-up calls unlikely. The derivation needs the function after the addition to be the identity. In the 2015 block it is a ReLU, so the follow-up moves batch norm and ReLU inside the residual branch, before the weight layers ("pre-activation"), and trains a 1001-layer network on CIFAR with the result. So the gradient argument is exact for the modified block. In the original block the ReLU after each addition zeroes the gradient wherever its output is zero, so the path is not a clean sum there.

The residual connection spread beyond vision within two years. Every sublayer of the Transformer (2017), attention and feed-forward alike, is wrapped as LayerNorm(x + Sublayer(x)), and the paper cites ResNet for the residual connection. Later work also found several ways to read the same equation. The unit update above has the form of one Euler step of a differential equation with depth as time. Lu et al. and Haber & Ruthotto developed that view in 2017, and Neural ODEs (2018) took it to the continuous limit; the DiffusionBlocks explainer builds on it. Veit et al. (2016) read a ResNet as an ensemble of many paths of different lengths and found that the relatively short ones carry most of the gradient during training. Li et al. (2018) visualized loss surfaces and found that skip connections keep them smooth at depths where the plain network's surface becomes chaotic. None of these readings is in the 2015 paper.