GoogLeNet (Inception v1): Going Deeper with Convolutions

Run 1×1, 3×3 and 5×5 convolutions side by side, shrink their inputs with 1×1 convolutions first, and a 22-layer network wins ImageNet 2014 on 1.5 billion multiply-adds per image.

GoogLeNet is built from nine Inception modules. Each module applies several convolution sizes and a pooling step to the same input and stacks the results; cheap 1×1 convolutions cut the channel count before every expensive one. The network has about 7 million weights, against 60 million for the 2012 winner, and its seven-model ensemble reached 6.67% top-5 error.

This page counts the weights and multiply-adds of every layer from the paper's Table 1, shows how much the 1×1 reductions save in each module, follows the gradient paths of the two auxiliary classifiers, and walks through the 144 test crops and seven models behind the winning number.

Making a ConvNet bigger in 2014

The 2012 ImageNet winner, AlexNet (the SuperVision team), was a convolutional network with 60 million parameters. It classified the 1000-class ILSVRC images with a top-5 error of 16.4%: for 16.4% of test images the correct class was not among the network's five highest-scoring guesses. The challenge ranks entries by this number. ILSVRC 2014 had about 1.2 million training images, 50,000 for validation and 100,000 for testing.

The direct way to improve on AlexNet was to make the network bigger: more layers (depth) or more channels per layer (width). The paper names two costs. A bigger network has more parameters and overfits more easily when labeled data is limited, and it needs more computation.

The second cost follows from what a convolution layer computes. At each position of its input feature map, a layer with kernel size $k$ reads a $k \times k$ patch across all $C_{\text{in}}$ input channels and produces $C_{\text{out}}$ output values, one per filter. Each output value is a weighted sum of $k^2 C_{\text{in}}$ inputs, so for an output map of $H \times W$ positions

A multiply-add is one multiplication followed by one addition, counted once; the paper measures every cost in these units, and this page leaves out biases, as Table 1 does. A 3×3 convolution from 192 to 128 channels has $9 \cdot 192 \cdot 128 = 221{,}184$ weights. On a 28×28 map it repeats those weights at 784 positions, which takes 173.4 million multiply-adds.

Now chain two such layers and double the filters in both. The second layer's $C_{\text{in}}$ and $C_{\text{out}}$ both double, so by (1) its cost goes up four times. The paper calls this the quadratic increase of computation under a uniform increase in width, and adds that the cost is wasted if many of the new weights end up close to zero.

The team set a budget instead. Most of their models were designed to need 1.5 billion multiply-adds per image at inference, so that they could run in practice, including on devices with limited compute and memory. GoogLeNet comes to 1.50 billion when every layer of Table 1 is counted, and the rest of the paper describes how that budget was spent.

The sparse-structure argument

The design started from a theoretical result. Arora, Bhaskara, Ge and Ma (2013) showed that if a dataset's distribution can be represented by a large, very sparse deep network, the best network can be built layer by layer: look at the correlations between the outputs of the last layer, group units whose outputs are highly correlated, and make each group a unit of the next layer. The paper connects this to the Hebbian principle from neuroscience, “neurons that fire together, wire together”. The proof needs strong assumptions, and the paper uses it only as motivation.

A sparse network in this sense has most connections between consecutive layers absent, and hardware runs such networks badly: dense matrix multiplication is extremely well optimized on CPUs and GPUs, and sparse arithmetic is not; the paper estimates that even with 100 times fewer arithmetic operations, the lookups and cache misses of non-uniform sparse matrices would make a sparse network slower. So the question becomes how to approximate an optimal sparse structure with dense components that hardware runs well.

Applied to a convolutional network, the argument runs as follows. Each unit in an early layer responds to a small region of the image. Correlated units there tend to sit in the same region, and a 1×1 convolution in the next layer covers such a cluster: it combines all channels at one position. There should also be fewer clusters that are spread out over a larger area, which a 3×3 convolution covers, and fewer still over larger areas, covered by a 5×5. A layer that contains all three sizes, with their outputs stacked, approximates the layer the clustering procedure would have built. The sizes stop at 5×5 for convenience, the paper says.

Section 3 of the paper ends with its own caution: although the architecture worked, it is “questionable whether its quality can be attributed to the guiding principles that have lead to its construction.”

How the Inception module works

An Inception module takes one feature map and sends it down four branches at once: a 1×1 convolution, a 3×3 convolution, a 5×5 convolution and a 3×3 max pooling. Every branch uses stride 1 and pads its input so that its output keeps the input's height and width. The four outputs are then concatenated along the channel axis: a 28×28 map with 64 channels and a 28×28 map with 128 channels become one 28×28 map with 192 channels. The pooling branch is there because pooling layers were part of every successful network at the time, and the paper expected a parallel pooling path to help as well.

A conventional network has to choose one kernel size per layer. In the module the next layer receives features computed over a 1×1, a 3×3 and a 5×5 neighbourhood side by side, and its weights decide how much each one contributes. An analogy from signal processing is a filter bank that runs several filters over the same signal in parallel and keeps every output for the next stage. The analogy breaks in two places: the filters here are learned, and they differ in spatial extent, not in frequency band.

The module GoogLeNet names 3a shows the sizes. Its input is the 28×28×192 map produced by the network's stem. Its 1×1 branch has 64 filters, the 3×3 branch 128, the 5×5 branch 32, and its output has 256 channels. In the paper's first version of the module, Figure 2(a), which it calls naive, the 3×3 and 5×5 convolutions run directly on all 192 input channels, and the pooling branch passes its 192 channels straight to the output. Two problems follow. The 5×5 branch alone has $25 \cdot 192 \cdot 32 = 153{,}600$ weights and 120.4 million multiply-adds. And the output has $64 + 128 + 32 + 192 = 416$ channels, more than the 192 that came in, because max pooling keeps the channel count of its input. The next module sees 416 channels and passes all of them through its own pooling branch, so the channel count grows at every stage.

With the naive module selected, the pooling segment of the output bar takes 192 of 416 channels at 3a and 528 of 1,232 at 4e. In the GoogLeNet version it takes 32 of 256 and 128 of 832, and the 5×5 branch of 3a drops from 120.4M to 12.4M multiply-adds.

Why 1×1 convolutions make it affordable

A 1×1 convolution looks at one position at a time. At each of the $H \times W$ positions it takes the vector of $C_{\text{in}}$ channel values, multiplies it by the same $C_{\text{in}} \times C_{\text{out}}$ weight matrix and applies a ReLU. It mixes channels and never mixes neighbouring pixels. Lin, Chen and Yan introduced it in Network in Network (2013) as a way to make each layer more expressive: a small fully connected network applied at every position, which they showed is the same as a 1×1 convolution. GoogLeNet uses it mainly for a second purpose. With $C_{\text{out}} < C_{\text{in}}$ it reduces the number of channels, and so the cost of whatever comes next.

The paper's Figure 2(b) puts a 1×1 reduction in front of the 3×3 and the 5×5 convolution, and a 1×1 projection after the max pooling. For a branch whose $k \times k$ convolution maps $C_{\text{in}}$ channels to $C_{\text{out}}$ through a reduction to $r$ channels, the weight count changes from the left side to the right side of

In module 3a the 5×5 branch reduces 192 channels to $r = 16$ before its 32 filters. That costs

weights instead of 153,600, 9.7 times fewer, and 12.4 million multiply-adds instead of 120.4 million. The 3×3 branch reduces to 96 channels and saves less: 129,024 weights instead of 221,184, a factor of 1.7, because 96 is half of 192 while 16 is a twelfth. Across all nine modules the 5×5 reductions are the most aggressive, from 192 to 16 at 3a and from 832 to 48 at 5b.

The obvious alternative is to keep the 5×5 convolution on all input channels and give it fewer filters. That shrinks what the branch produces. The reduction shrinks what it reads: each 5×5 filter still sees all 192 input channels, through a learned summary of 16 numbers per position. The paper's argument for why a summary is enough is that even low-dimensional embeddings can hold a lot of information about an image patch. It also argues against compressing everywhere, since compressed representations are harder to model, and reduces only right before the 3×3 and 5×5 convolutions, where many signals are aggregated at once. Each reduction layer has its own ReLU, so it also adds a nonlinearity and a layer of depth. The paper calls the 1×1 convolutions dual-purpose for this reason.

A reduction resembles projecting data onto its first few principal components before an expensive computation. The resemblance stops at three points: the projection is learned for the classification task, where PCA keeps the directions of largest variance; it is followed by a ReLU, so it is nonlinear; and it is retrained along with everything else.

The pooling projection deals with the second problem. After max pooling, a 1×1 convolution with 32 filters (at 3a) replaces the 192 passed-through channels, so the module's output width is set by its branch widths and no longer by its input. Figure 2 below stacks all nine modules of GoogLeNet and lets you switch the two fixes on and off separately. Look at the bottom row of numbers, the output channels of each module.

With neither fix, the nine modules need 12.1 billion multiply-adds, eleven times GoogLeNet's 1.10 billion for the same modules, and 5b outputs 4,944 channels. The reductions alone bring the cost down to 1.64 billion but leave the channel growth unchanged, so the final map is still 7×7×4,944. The pooling projections alone stop the growth, but at 3.95 billion the modules would use more than twice the network's whole budget, which is why GoogLeNet uses both.

GoogLeNet layer by layer

The paper's Table 1 lists the full network. It starts with ordinary convolutions and uses Inception modules only higher up; the paper says this was for memory efficiency during training in its implementation. A 224×224 RGB image (with the mean subtracted) goes through a 7×7 convolution with stride 2 to a 112×112×64 map, a 3×3 max pooling with stride 2 to 56×56×64, a 1×1 convolution with 64 filters followed by a 3×3 convolution with 192, and another max pooling to 28×28×192. The paper's Figure 3 also shows local response normalization (a fixed rescaling of each activation by the activity of neighbouring channels, taken from AlexNet) after the first pooling and after the 3×3 convolution; Table 1 leaves those layers out. Then come the Inception modules in three groups, separated by stride-2 max pooling:

• 3a and 3b at 28×28, ending with 480 channels;
• 4a to 4e at 14×14, ending with 832 channels;
• 5a and 5b at 7×7, ending with 1,024 channels.

The classifier is a 7×7 average pooling over the final map, which leaves one 1,024-long vector per image, a dropout layer that zeroes 40% of its entries during training, and one linear layer to the 1000 classes, followed by a softmax. Global average pooling comes from Network in Network. AlexNet and VGG used two large fully connected layers at this point instead; the paper reports that replacing them with average pooling improved top-1 accuracy (the fraction of images whose highest-scoring class is correct) by about 0.6%, and that dropout was still essential afterwards. The extra linear layer was kept so the network could be fine-tuned to other label sets.

Counting only layers with weights, GoogLeNet is 22 layers deep: the stem's 3 (the 7×7, the 1×1 and the 3×3), two in each of the nine modules (a reduction and then a convolution, on the longest path), and the final linear layer. Counting the five pooling layers between them gives 27. The paper puts the number of independent building blocks at about 100.

The compute is spread over the stem (394 million multiply-adds), stage 3 (432 million) and stage 4 (551 million); stage 5 needs only 122 million because it runs at 7×7. The single most expensive row is the stem's 1×1-then-3×3 pair (Table 1's conv2 row), 360 million multiply-adds (24% of the total), because its 114,688 weights run at 56×56 = 3,136 positions; 3b follows at 304 million. The weights pile up at the end: inception 5b has 1.44 million and the linear layer 1.02 million, and they run on 49 and 1 positions. In total the recount gives 6,983,576 weights and 1.50 billion multiply-adds. By comparison VGG-16, which came second in the same challenge, has 138 million weights.

Table 1's first row does not match a plain convolution. A 7×7 convolution from 3 to 64 channels has 9,408 weights and 118 million multiply-adds at 112×112; the table prints 2.7K and 34M. Those two numbers match a depthwise-separable 7×7 instead: a 7×7 filter on each input channel with 8 outputs per channel (3·8·49 = 1,176 weights), then a 1×1 convolution from 24 to 64 channels (1,536 weights), 2,712 weights in all. A 2015 follow-up by Ioffe and Szegedy, which introduced batch normalization and a revised network called BN-Inception, says that network's first layer was built that way. The widely used Caffe replication of GoogLeNet uses an ordinary 7×7, which brings the network to 1.58 billion multiply-adds.

Auxiliary classifiers

In 2014 it was not clear that gradients would travel well through 22 layers; batch normalization and residual connections were both published in 2015. The paper also noted that much shallower networks did well on ImageNet, which suggested that features in the middle of GoogLeNet should already be good enough to classify from. So during training it attaches two small classifiers to intermediate modules, 4a and 4d, each of them a miniature version of the network's head:

• a 5×5 average pooling with stride 3, which turns the 14×14 map into 4×4 (512 channels at 4a, 528 at 4d);
• a 1×1 convolution to 128 channels with ReLU, leaving 4·4·128 = 2,048 numbers;
• a fully connected layer with 1,024 units and ReLU;
• dropout that zeroes 70% of those units;
• a linear layer to the 1000 classes, with a softmax loss on the same labels as the main classifier.

The training loss adds the two auxiliary losses to the main one with weight 0.3 each:

Each term is a cross-entropy, minus the log of the softmax probability given to the correct class. At test time both heads are removed and only the main classifier's output is used.

The heads change how far the error signal travels. Take the output of module 3a. To reach it from the main loss, the gradient passes back through 17 weight layers: 3b, the five modules of stage 4 and the two of stage 5 (two layers each on the longest path), and the linear classifier. From the 4a head it passes through 7: the head's linear layer, its fully connected layer and its 1×1 convolution, then 4a and 3b. The paper gives three reasons for the heads: stronger gradient signal in the lower layers, more discriminative features in the middle of the network, and extra regularization.

Each head has about 3.19 million weights, 2,097,152 of them in the fully connected layer from 2,048 inputs to 1,024 units, so the two heads together hold 6.4 million weights, nearly as many as the network they are attached to.

Whether the heads did what they were meant to do was tested a year later, in the same group's Inception-v3 paper. There, networks with and without an auxiliary head trained at the same speed early on; near the end, the network with the head reached a slightly higher accuracy, and removing the lower of the two heads made no difference to the final result. The authors concluded that the heads act mostly as a regularizer, not as a fix for weak gradients.

The module and the loss in code

Google never released the original GoogLeNet code, which ran on its internal DistBelief system. The listing below is a PyTorch transcription of module 3a from Table 1. Run on a batch of eight 28×28×192 maps it returns eight 28×28×256 maps, and its convolution weights add up to the 163,328 counted in the previous sections.

import torch, torch.nn as nn, torch.nn.functional as F

def conv(cin, cout, k):          # every convolution in GoogLeNet is followed by a ReLU
    return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2), nn.ReLU())

class Inception(nn.Module):
    def __init__(self, cin, n1, r3, n3, r5, n5, pp):
        super().__init__()
        self.b1 = conv(cin, n1, 1)                              # 1x1 branch
        self.b3 = nn.Sequential(conv(cin, r3, 1), conv(r3, n3, 3))  # reduce, 3x3
        self.b5 = nn.Sequential(conv(cin, r5, 1), conv(r5, n5, 5))  # reduce, 5x5
        self.bp = conv(cin, pp, 1)                              # pool projection

    def forward(self, x):                            # x: B x cin x H x W
        p = F.max_pool2d(x, 3, stride=1, padding=1)  # still H x W, cin channels
        outs = [self.b1(x), self.b3(x), self.b5(x), self.bp(p)]
        return torch.cat(outs, dim=1)                # B x (n1+n3+n5+pp) x H x W

inc3a = Inception(192, 64, 96, 128, 16, 32, 32)     # Table 1, row 3a
y = inc3a(torch.randn(8, 192, 28, 28))              # y: 8 x 256 x 28 x 28
# 163,328 weights (biases excluded); x 28 x 28 positions = 128.0M multiply-adds

The auxiliary head and the training loss of (3) follow. Every layer of the network receives gradient from the main loss; layers at or below 4a also receive gradient from both heads, and 4b to 4d from the second head.

class AuxHead(nn.Module):                    # one on 4a, one on 4d
    def __init__(self, cin):                 # cin = 512 at 4a, 528 at 4d
        super().__init__()
        self.conv = conv(cin, 128, 1)
        self.fc1 = nn.Linear(4 * 4 * 128, 1024)
        self.fc2 = nn.Linear(1024, 1000)

    def forward(self, x):                     # x: B x cin x 14 x 14
        x = F.avg_pool2d(x, 5, stride=3)      # B x cin x 4 x 4
        x = self.conv(x).flatten(1)           # B x 2048
        x = F.dropout(F.relu(self.fc1(x)), p=0.7, training=self.training)
        return self.fc2(x)                    # 1000 logits

def training_loss(net, images, labels):
    main, aux1, aux2 = net(images)            # logits from 5b, the 4a head, the 4d head
    return (F.cross_entropy(main, labels)
            + 0.3 * F.cross_entropy(aux1, labels)
            + 0.3 * F.cross_entropy(aux2, labels))
# loss.backward() reaches 4a and the layers below it along three paths;
# at test time the two heads are deleted and only main is used

How it was trained

The networks were trained on CPUs with DistBelief, using a modest amount of model and data parallelism. The optimizer was asynchronous stochastic gradient descent with momentum 0.9: many workers compute gradients on their own mini-batches and apply them to shared parameters without waiting for each other. The learning rate was cut by 4% every 8 epochs. The Caffe replication implements this as multiplying the rate by 0.96 every 320,000 iterations of 32 images, which is 8 passes over the 1.28 million training images. The model used at test time was a Polyak average, a running average of the weights over the course of training, which is less noisy than the final iterate. The paper estimates that GoogLeNet could be trained to convergence on a few high-end GPUs within a week, limited mainly by memory.

The paper says that its data augmentation changed over the months before the competition, and that it cannot say which single recipe works best. It gives one recipe that it verified to work well after the competition. Each training example is a patch of the image whose area is drawn uniformly between 8% and 100% of the image area and whose aspect ratio is drawn between 3/4 and 4/3, resized to 224×224. This is the “Inception crop”: PyTorch's RandomResizedCrop uses exactly these ranges as its defaults. On top of it come photometric distortions from Andrew Howard's 2013 entry and, late in the project, a random choice of interpolation method for resizing (bilinear, area, nearest neighbour or cubic), whose effect the authors could not isolate.

Test-time crops and the ensemble

The competition entry was an ensemble: seven GoogLeNets trained independently and averaged at test time. One of them was a wider version, the other six used the Table 1 topology. They started from the same initial weights (the paper says this was an oversight) and differed only in how training images were sampled and in the order the images were seen.

Each test image was also cropped many times. The image is resized four times, so that its shorter side is 256, 288, 320 and 352 pixels. From each resized image the left, center and right squares are taken (top, center and bottom for a portrait image). From each square come six 224×224 inputs: the four corner crops, the center crop and the whole square shrunk to 224×224. Each of these is used as is and mirrored left to right, which makes $4 \times 3 \times 6 \times 2 = 144$ crops per image. Figure 4 steps through them.

The scale changes how much of the scene one input covers. At scale 256 a corner crop spans 88% of the square's side; at scale 352 it spans 64%, so the object appears larger in the input.

For the final prediction, the softmax probabilities are averaged over all crops and all models. The paper tried taking the maximum over crops instead, and averaging over classifiers in other ways, and found simple averaging best. Its Table 3 measures what each part buys on the validation set. Cost is in forward passes of one GoogLeNet, models times crops; the single model is the one with the lowest top-1 validation error.

One model on one crop gives 10.07%. Ten crops bring it to 9.15% and 144 crops to 7.89%. Seven models on one crop each reach 8.09% for 7 forward passes, close to what one model gets from 144. All of it together, 1,008 forward passes and about 1.5 trillion multiply-adds per image, gives 6.67%, 3.40 points below the single crop. The paper notes that such aggressive cropping may not be necessary in real applications.

Results on ILSVRC 2014

GoogLeNet won the 2014 classification challenge with 6.67% top-5 error on both the validation and the test set, without any training data beyond the challenge's own. VGG came second with 7.32% and MSRA third with 7.35%. The 2012 winner had 16.4%, or 15.3% with extra training data from the 22,000-class ImageNet; the 2013 winner, Clarifai, had 11.7%, or 11.2% with extra data. Against the extra-data numbers, GoogLeNet cut the error by 56.5% relative to 2012 and by about 40% relative to 2013. In practical terms, the 2014 ensemble missed the correct class in its top five for about one test image in fifteen, where AlexNet missed one in six.

The team also entered the detection challenge, which asks for a box around every object from 200 classes in each image. A detection counts if its class is right and its box overlaps the true box by at least 50% (intersection over union); the score is mean average precision (mAP), the area under each class's precision-recall curve averaged over the 200 classes. The pipeline followed R-CNN (Girshick et al., 2014): propose candidate regions with a cheap image-segmentation method, then classify each region with a convolutional network, here GoogLeNet. That method, Selective Search, builds its proposals by merging superpixels (small regions of similar colour). Doubling the superpixel size halved the number of proposals, and 200 boxes from MultiBox (a network by some of the same authors that predicts boxes directly) were added back. Together these changes left about 60% of R-CNN's proposal count, raised the share of objects covered from 92% to 93%, and added 1 point of mAP for a single model.

An ensemble of six networks scored 43.9% mAP and won, ahead of CUHK DeepID-Net at 40.7% and Deep Insight at 40.5%. The 2013 winner had 22.6%, so the score almost doubled in a year. GoogLeNet's single model scored 38.02%, below Deep Insight's single model at 40.2%, and it used neither bounding-box regression (which R-CNN uses to adjust boxes) nor context models; the paper says there was no time for box regression. Later detectors such as Faster R-CNN replaced the separate proposal step with a network that shares features with the classifier.

What changed after GoogLeNet

The same group revised the design several times. BN-Inception (Ioffe and Szegedy, 2015) added batch normalization to every layer and replaced each 5×5 convolution with two stacked 3×3 convolutions. Inception-v3 (Szegedy et al., 2015) factored convolutions further, for example a 3×3 into a 3×1 followed by a 1×3, and kept one auxiliary head, with batch normalization, as a regularizer.

The 1×1 reduction spread beyond Inception. The bottleneck block of ResNet, which won ILSVRC 2015, is a 1×1 convolution that reduces channels, a 3×3 convolution, and a 1×1 convolution that restores them. MobileNetV2 and EfficientNet build their blocks from 1×1 convolutions around a cheap depthwise convolution. Global average pooling in front of a single linear layer became the standard classifier head for convolutional networks.

The sparse-structure program from Section 3, building networks automatically from correlation statistics, was not carried out in this paper. Its tested claims are the ones on this page: a 22-layer network with about 7 million weights and 1.5 billion multiply-adds per image, 6.67% top-5 error with seven models and 144 crops, and 43.9% detection mAP with six.