EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Scale a convolutional network's depth, width and input resolution together by fixed ratios, and it reaches higher accuracy for the same compute than scaling any one of them.

EfficientNet sets depth to 1.2^φ, width to 1.1^φ and input resolution to 1.15^φ, so each step of the coefficient φ roughly doubles the FLOPS. Applied to a small network found by architecture search, the rule gives eight models, B0 to B7. B7 matched the best ImageNet accuracy of 2019, 84.3% top-1, with 8.4 times fewer parameters than the previous record holder.

This page counts what each dimension costs, rebuilds B0 to B7 layer by layer from the official code (their coefficients do not follow one φ each), opens up the MBConv block that does the computing, and separates the scaling rule's share of the gains from the share of the baseline and the training recipe.

Three ways to make a ConvNet bigger

An image classifier is usually designed at one size and then enlarged when more compute is available. In 2019 there were three standard ways to enlarge a convolutional network (ConvNet). ResNet grew by depth, from 18 layers to 200. Wide ResNets and MobileNets grew by width, the number of channels in each layer. GPipe, the ImageNet record holder when this paper appeared, used input images of 480 by 480 pixels instead of the usual 224.

The three dimensions are easiest to see in the shapes of the tensors. A ConvNet layer takes a feature map of shape $\langle H, W, C\rangle$ (height, width in pixels, channels) and produces another. The paper groups layers into stages, where every layer in a stage has the same type and the stage repeats it $L_i$ times, and writes a whole network as

where $\mathcal{F}_i$ is the layer type of stage $i$, the circle-dot means composition (stage 1 first), and $\langle H_i, W_i, C_i\rangle$ is the input shape of stage $i$. EfficientNet-B0, the baseline this paper builds, has nine stages. Its input is $\langle 224, 224, 3\rangle$; a 3×3 convolution with stride 2 turns that into $\langle 112, 112, 32\rangle$, and seven stages of MBConv layers (described below) shrink the spatial size to 7×7 while the channels grow to 320. A final 1×1 convolution to 1280 channels, an average over the 7×7 positions and a fully connected layer produce 1000 class scores.

Scaling keeps every layer type $\mathcal{F}_i$ fixed and changes only the numbers around it. Depth multiplies every $L_i$ by a coefficient $d$, width multiplies every $C_i$ by $w$, and resolution multiplies $H_i$ and $W_i$ by $r$. The paper scales all stages by the same three coefficients, so finding the best enlarged network under a compute budget reduces to choosing three numbers:

Hats mark the baseline's values. For B0, $\hat{L}$ runs 1, 2, 2, 3, 3, 4, 1 over the seven MBConv stages, 16 layers in all. Setting $d = 2$ gives 32 layers, $w = 2$ doubles every channel count (32 to 64 in the stem, 320 to 640 in the last stage), and $r = 2$ means 448×448 input images. FLOPS here means the number of multiply-add operations needed to classify one image; the paper counts a multiply and the add that follows it as one operation, which is how B0 comes out at 0.39 billion. Accuracy is ImageNet top-1: the fraction of the 50,000 validation images for which the class with the highest score, out of 1000, is the correct one.

Equation (2) is too expensive to solve directly, since every candidate is a network that has to be trained on ImageNet. The paper first studies one dimension at a time.

Scaling one dimension at a time

Each dimension has a plausible reason to help. A deeper network composes more layers, so it can build more complex features out of simpler ones. A wider network has more channels per layer, more distinct patterns it can detect at each stage. A higher-resolution input carries finer detail: at 224 pixels a bird's eye may span a handful of pixels, at 448 four times as many.

Each also runs out. ResNet-1000 is about as accurate as ResNet-101. Very wide but shallow networks fail to build high-level features. Image resolution helps until the object is already well resolved. The paper's Figure 3 scales B0 in each dimension separately, up to $w = 5$, $d = 8$ and $r = 2.5$ (560 pixels). In each case accuracy climbs quickly at first and flattens once it reaches about 80% top-1. The paper states this as

Observation 1: Scaling up any dimension of network width, depth, or resolution improves accuracy, but the accuracy gain diminishes for bigger models.

Figure 1 below plots the experiments where the paper puts single-dimension and compound scaling side by side at matched compute. Each baseline was enlarged to roughly four times its FLOPS: by depth alone ($d = 4$), width alone ($w = 2$), resolution alone ($r = 2$), or all three at once. Compare the teal bar with the best amber bar on each baseline.

Compound scaling wins on every baseline, by 0.6 points on MobileNetV2, 0.7 on ResNet-50, 1.4 on MobileNetV1 and 2.0 on EfficientNet-B0. On B0 the three single dimensions land within 0.2 points of each other (78.9 to 79.1%), 2.0 points below compound scaling.

What each dimension costs in FLOPS

To share a budget among the three, you need the price of each. Take one regular convolution with a $k \times k$ kernel that maps a $H \times W \times C_\text{in}$ feature map to $H \times W \times C_\text{out}$. Every output value is a weighted sum over a $k \times k \times C_\text{in}$ window, so the layer costs

multiply-accumulates. Now apply the three coefficients. Depth $d$ multiplies the number of such layers, so the cost grows by $d$. Width $w$ multiplies both $C_\text{in}$ and $C_\text{out}$, so the cost grows by $w^2$. Resolution $r$ multiplies both $H$ and $W$, so the cost grows by $r^2$. Summed over a network made of such layers,

Doubling the compute therefore takes $d = 2$, or $w = \sqrt{2} \approx 1.41$, or $r = \sqrt{2}$ (a 317-pixel image instead of 224). Counting B0 layer by layer with those settings gives 2.03×, 1.93× and 2.03× its compute, close to the formula.

The formula assumes regular convolutions. Each of B0's MBConv layers also contains a depthwise convolution, which filters each channel on its own: it costs $H \cdot W \cdot k^2 \cdot C$, linear in the channel count, so it grows with $w$ instead of $w^2$. In B0 the depthwise convolutions are only 9.0% of the multiply-accumulates. The 1×1 convolutions that expand and project channels inside each MBConv layer take 43.1% and 39.5%, the 1×1 head another 5.2%, and all of those scale with $w^2$. So doubling B0's width costs 3.76×, a little less than 4×.

In Figure 2 the top bar is the formula, split on a log scale into the doublings each factor costs; the bottom bar is the counted total for the network the sliders describe. The presets reproduce the scalings of Figure 1.

The figure opens on the compound setting of Table 7 (d = 1.4, w = 1.2, a 292-pixel input), where the estimate is 3.4× and the counted network 4.7× (Table 7 lists 0.4B to 1.8B). The baseline button empties both bars: B0 is 0.386 billion multiply-accumulates. From there, push depth to 1.1 and the counted bar jumps by more than the estimate, because B0 has stages of one and two layers, and a 1.1× multiplier rounds one layer up to two. The code always rounds the layer count up (the paper's footnote 3 mentions rounding), and on a network with stages this short the rounding is large. With width and resolution at their maximum the counted bar falls short of the estimate, because of the depthwise share.

Why the dimensions should grow together

The second experiment in Section 3 scales width while holding depth and resolution at four combinations: $(d, r) = (1, 1)$, $(2, 1)$, $(1, 1.3)$ and $(2, 1.3)$. With $d = 1$ and $r = 1$, widening the network saturates quickly. With $d = 2$ and $r = 1.3$ (36 layers at 299 pixels), the same widening reaches clearly higher accuracy at the same FLOPS.

Two mechanisms explain why. The first is the receptive field, the patch of input pixels one output value can depend on. A 3×3 layer working on a feature map that has been downsampled 8× widens the receptive field by $2 \times 8 = 16$ input pixels; the field grows only by adding layers. At 224 pixels, suppose an object spans 100 pixels; at 448 it spans 200, while every layer's receptive field, measured in pixels, is unchanged. Unless depth grows too, the late layers' receptive fields cover a smaller fraction of the object than before. The second is the number of channels. A larger image resolves more distinct patterns (individual feathers rather than texture), and each pattern a stage can detect needs channels to represent it, so higher resolution with the same channel count runs out of places to put the extra detail.

An analogy: a document scanned at twice the resolution. To read the same word, a reader looking through a fixed-size window needs a larger window (more depth), and the sharper scan only helps if the reader can tell more letter shapes apart (more width). The analogy breaks in one place: a network is not required to recognize the same features at the higher resolution, and the extra pixels can carry information the low-resolution image did not contain.

Observation 2: In order to pursue better accuracy and efficiency, it is critical to balance all dimensions of network width, depth, and resolution during ConvNet scaling.

Earlier architecture-search papers (NASNet, AmoebaNet) had balanced width and depth by hand. This paper fixes the balance once, with the rule in equation (3) below, and applies the same ratios at every model size.

How compound scaling works

Compound scaling ties the three coefficients to one number, the compound coefficient $\phi$:

$\alpha$, $\beta$ and $\gamma$ set the ratio between the dimensions, and $\phi$ sets the overall size. Since FLOPS grow as $d \cdot w^2 \cdot r^2$, a network scaled by (3) costs $(\alpha \beta^2 \gamma^2)^{\phi}$ times the baseline, and the constraint makes that about $2^{\phi}$: each unit of $\phi$ doubles the compute.

The constants are found in two steps:

1. Fix $\phi = 1$, a budget of twice B0's FLOPS, and grid-search $\alpha$, $\beta$, $\gamma$ under the constraint, training each candidate. The best values for B0 are $\alpha = 1.2$, $\beta = 1.1$, $\gamma = 1.15$.
2. Keep those constants and raise $\phi$ to get larger models, with no further search.

The search runs only on the small model, where training a candidate is cheap. The paper notes that searching $\alpha, \beta, \gamma$ directly around a large model could do better, at a cost it calls prohibitive. The constraint leaves two free numbers (pick $\alpha$ and $\beta$, and $\gamma$ is determined), so the grid is two-dimensional.

Check the found values against the constraint:

So one unit of $\phi$ multiplies the estimated FLOPS by 1.92. In doublings, depth takes $\log_2 1.2 = 0.263$, width $2\log_2 1.1 = 0.275$ and resolution $2\log_2 1.15 = 0.403$, which is 28%, 29% and 43% of the budget. For comparison, scaling all three by one shared factor $x$ under the same budget gives $x^5 = 2$, $x = 1.149$; the search moved depth up from that and width down.

A worked example at $\phi = 3$: $d = 1.2^3 = 1.728$, $w = 1.1^3 = 1.331$, $r = 1.15^3 = 1.521$. The input becomes $224 \times 1.521 \approx 341$ pixels. Each stage's layer count is multiplied by 1.728 and rounded up, turning 1, 2, 2, 3, 3, 4, 1 into 2, 4, 4, 6, 6, 7, 2 (31 layers). The channels become 24, 32, 56, 104, 152, 256, 424 after rounding to multiples of 8. The estimate says $1.92^3 = 7.08$ times B0's FLOPS; counting the actual network gives 8.43 times, 3.25 billion multiply-accumulates; the excess comes from rounding layer counts up (31 layers is 1.94 times 16, not 1.728) and channel counts to multiples of 8.

Slide $\phi$ from 0 to 7 and all three grow at once: more bars, taller bars, wider bars, with resolution contributing the most to the compute. At $\phi = 7$ the input is 596 pixels and the counted network is 103 times B0, against an estimate of $1.92^7 = 96$.

The B1 to B7 coefficients in the code

The paper says EfficientNet-B1 to B7 are obtained by scaling B0 "with different $\phi$ using Equation 3", and refers to Table 2 for details. Table 2 lists accuracy, parameters and FLOPS, but no $\phi$ and no coefficients. Those are in the official code, as a table of (width, depth, resolution):

The first three columns come from efficientnet_builder.py, the layer counts from applying its rounding rules, and the last three from the paper's Table 2. Counting the networks layer by layer reproduces Table 2's parameters and FLOPS to within 5% for every model (for example 37.7B and 66.7M for B7), which also confirms that the paper's FLOPS are multiply-adds.

Invert each coefficient to get the $\phi$ it implies: $\phi_d = \log d / \log 1.2$, and likewise for width with 1.1 and for resolution with 1.15. B2 (1.1, 1.2, 260) is $\phi = 1$ to rounding; its counted size, 0.994B, equals the 0.993B counted for the network equation (3) builds at $\phi = 1$ (both sit above the 1.92× estimate because of rounding, explained below). B3 is $\phi = 2$ by the same check. B1 is about $\phi = 0.5$ for depth and resolution with width left at 1.0. From B4 on, the three coefficients disagree: B7's depth implies $\phi = 6.2$, its width 7.3 and its resolution 7.0. The larger released models grow width and resolution faster than depth, compared with equation (3). Pick the B-model buttons in Figure 3 to read each model's implied values.

The rounding also explains why B2 costs 2.58 times B0, not the 1.92 times of one doubling. B0 has two stages with a single layer; multiplying by 1.2 and rounding up gives two layers each, and the 16 layers become 23, a 1.44× increase in depth instead of 1.2×.

EfficientNet-B0 from architecture search

Scaling never changes the layer types $\hat{\mathcal{F}}_i$, so the quality of the baseline caps the quality of every scaled model. The authors built a new baseline with neural architecture search (NAS), an automated search in which a controller proposes network designs, each is trained briefly, and the controller is steered toward designs that score well. They reused the search space of MnasNet, an earlier mobile-network search by the same group, and its multi-objective reward:

Here $m$ is a candidate model, $T$ = 400 million FLOPS is the target, and the exponent $w$ (a different symbol from the width coefficient, despite the shared letter) trades accuracy against size. A model at the target is scored by its accuracy. A model at twice the target is multiplied by $2^{-0.07} = 0.953$, so it scores the same as a model at the target only if its accuracy is about 5% higher (relative), which is the rule MnasNet used to pick −0.07: doubling latency usually brings about 5% relative accuracy. MnasNet optimized measured phone latency; this search optimizes FLOPS, since EfficientNet does not target a particular device.

The result, EfficientNet-B0, is the paper's Table 1:

The resolution column is each stage's input size, and the channel column its output channels. A stage whose next stage has a smaller resolution downsamples in its first layer with stride 2 (stages 1, 3, 4, 5 and 7). "MBConv6, k5×5" means an MBConv layer with expansion ratio 6 and a 5×5 depthwise kernel. B0 is slightly larger than MnasNet because its FLOPS target is larger: 5.3M parameters, 0.39B FLOPS, 77.1% top-1 on ImageNet.

Inside an MBConv layer

MBConv is the inverted residual block of MobileNetV2, with a squeeze-and-excitation step added. One layer with input $H \times W \times C$ does five things in order:

1. A 1×1 convolution expands the channels by the expansion ratio, usually 6: $C$ to $6C$. Followed by batch normalization and the SiLU activation $x \cdot \sigma(x)$ (also called Swish).
2. A $k \times k$ depthwise convolution filters each of the $6C$ channels separately, with stride 2 if the layer downsamples. Batch norm and SiLU again.
3. Squeeze-and-excitation (SE): average each channel over all positions (one number per channel), pass the $6C$ numbers through a small two-layer network that narrows to $C/4$ and widens back, and multiply every channel by the sigmoid of its output, a factor between 0 and 1. The factors depend on the whole image, so the same layer can turn a channel down for one image and leave it on for another.
4. A 1×1 convolution projects $6C$ back down to the stage's output channels, with batch norm and no activation. MobileNetV2 calls this the linear bottleneck: a nonlinearity on the narrow tensor would discard information that the next layer cannot recover.
5. If the stride is 1 and the input and output channel counts match, the input is added back (a residual connection, as in ResNet).

"Inverted" refers to where the wide part sits. A ResNet bottleneck block narrows the channels, does its 3×3 convolution on the narrow tensor and widens back, with the skip connection between the wide ends. MBConv widens first, filters the wide tensor with a cheap depthwise convolution, and narrows back, with the skip between the narrow ends. A regular 3×3 convolution at 240 channels would cost 240 times as much as the depthwise one, which is why MBConv can afford to filter the expanded tensor at all.

A worked example from B0, the first layer of stage 4: input 56×56×24. The expansion makes 56×56×144 at a cost of $56^2 \cdot 24 \cdot 144$ = 10.8 million multiply-accumulates. The 5×5 depthwise convolution with stride 2 produces 28×28×144 for 2.8 million. SE squeezes 144 channels to $\lfloor 24/4 \rfloor = 6$ and back for 1,728. The projection to 28×28×40 costs 4.5 million. Of the 18.2 million total, the two 1×1 convolutions take 84%. Stride 2 and the change from 24 to 40 channels rule out the skip connection here; the second layer of the stage has one.

Stage 2 has expansion ratio 1, so it skips the expansion convolution and the depthwise convolution takes a larger share. Everywhere else the 1×1 convolutions dominate. That is why the $w^2$ estimate from the cost section is close for EfficientNet even though depthwise convolutions scale linearly. The SE step, which operates on one number per channel, is a fraction of a percent of the arithmetic.

Scaling and the block in code

The scaling logic of the official TensorFlow code fits in a few functions. Given the coefficients, it rounds channels to multiples of 8 (friendlier to hardware) and rounds layer counts up:

ALPHA, BETA, GAMMA = 1.2, 1.1, 1.15     # grid-searched once, on B0, at phi = 1

def coefficients(phi):
    return ALPHA ** phi, BETA ** phi, round(224 * GAMMA ** phi)  # d, w, res

def round_filters(c, w, div=8):          # channels: nearest multiple of 8
    new = max(div, int(c * w + div / 2) // div * div)
    if new < 0.9 * c * w:                # never round down by more than 10%
        new += div
    return new

def round_repeats(n, d):                 # layers per stage: always round up
    return math.ceil(n * d)

# B0 stages 2-8 of Table 1: (repeats, kernel, stride, expand, out_channels)
B0 = [(1, 3, 1, 1, 16), (2, 3, 2, 6, 24), (2, 5, 2, 6, 40), (3, 3, 2, 6, 80),
      (3, 5, 1, 6, 112), (4, 5, 2, 6, 192), (1, 3, 1, 6, 320)]

d, w, res = 1.4, 1.2, 300                # B3 as released in efficientnet_builder.py
stages = [(round_repeats(n, d), k, s, e, round_filters(c, w))
          for n, k, s, e, c in B0]
# repeats  2  3  3  5   5   6   2   (26 MBConv layers instead of 16)
# channels 24 32 48 96 136 232 384  (stem 32 -> 40, head 1280 -> 1536)

The stem and head are scaled for width too: B3's stem has 40 channels and its head 1536. Depth applies only to the seven MBConv stages; the stem, the head and the classifier stay single layers. The block itself, written out with the shapes:

def mbconv(x, k, stride, expand, c_out, survival_prob):  # x: H x W x c_in
    c_in = x.shape[-1]
    h = x
    if expand != 1:
        h = silu(bn(conv1x1(h, c_in * expand)))     # H x W x 6*c_in
    h = silu(bn(depthwise(h, k, stride)))           # H/s x W/s x 6*c_in
    s = mean(h, axis=(0, 1))                        # squeeze: one value per channel
    s = conv1x1(silu(conv1x1(s, int(c_in * 0.25))), h.shape[-1])
    h = h * sigmoid(s)                              # excite: rescale every channel
    h = bn(conv1x1(h, c_out))                       # linear projection, no SiLU
    if stride == 1 and c_in == c_out:
        h = x + drop_connect(h, survival_prob)      # stochastic depth on the branch
    return h

The first layer of each stage takes the stage's stride and input channels; the repeats use stride 1 and c_out channels in and out, so only the repeats have skip connections. Every weight in the block, including the SE layers, is trained by ordinary backpropagation of the classification loss. The scaling coefficients are fixed before training starts and receive no gradient.

How the models were trained

All models were trained on ImageNet (1.28 million images, 1000 classes) with settings borrowed from MnasNet: RMSProp with decay 0.9 and momentum 0.9, batch norm momentum 0.99, weight decay 1e-5, and an initial learning rate of 0.256 multiplied by 0.97 every 2.4 epochs. On top of those:

• SiLU activations everywhere instead of ReLU.
• AutoAugment, a data-augmentation policy found by search.
• Stochastic depth: during training each MBConv layer's residual branch is dropped at random, so the layer becomes an identity for that example. The paper gives the survival probability as 0.8; the code decays it linearly from 1.0 at the first layer to 0.8 at the last, and divides the surviving branch by its survival probability so that nothing changes at test time.
• Dropout before the final classifier, raised with model size: 0.2 for B0 and B1, 0.3 for B2 and B3, 0.4 for B4 and B5, 0.5 for B6 and B7.

The authors held out 25,000 training images as a "minival" set, early-stopped on it, and report the early-stopped checkpoint's accuracy on the standard 50,000-image validation set. An appendix checks the test server too: B7 scores 84.26% on validation and 84.33% on the held-out test set.

Results on ImageNet and transfer tasks

Table 2 of the paper places each EfficientNet next to published models of similar accuracy. Figure 5 plots all of them; click a point or use the menu, and the readout pairs that model with the smallest EfficientNet at least as accurate.

Some of the pairs, with what they mean in practice:

• B7 reaches 84.3% top-1 with 66M parameters and 37B FLOPS. GPipe reaches the same 84.3% with 557M parameters, 8.4 times as many, and needed a pipeline-parallelism library to split it across accelerators for training.
• B3 beats ResNeXt-101 (81.6% vs 80.9%) with 7.0 times fewer parameters and 18 times fewer FLOPS.
• B4 and ResNet-50 cost about the same, 4.2B and 4.1B FLOPS, and B4 is 6.9 points more accurate (82.9% vs 76.0%). The introduction quotes this as 76.3% to 83.0%, from different runs.
• On a CPU (average of 20 runs), B1 classifies an image in 0.098 s at 78.8% against 0.554 s at 77.8% for ResNet-152, and B7 takes 3.1 s against 19.0 s for GPipe. These are the 5.7× and 6.1× speedups of Table 4.

These comparisons mix three changes at once: a searched baseline, the scaling rule and a stronger training recipe (the paper says so: "better architectures, better scaling, and better training settings"). The cleanest test of the rule alone is Table 7, the B0 rows of Figure 1: same baseline, same recipe, about 1.8B FLOPS, and compound scaling scores 81.1% against 79.0%, 78.9% and 79.1% for depth, width and resolution alone. The paper also shows class activation maps (a heat map of which pixels drove the prediction) for these models; the compound-scaled one covers more of each object, a qualitative picture rather than a measurement.

For transfer learning the authors fine-tune ImageNet-trained EfficientNets on eight datasets (CIFAR-10, CIFAR-100, Birdsnap, Stanford Cars, Flowers, FGVC Aircraft, Oxford-IIIT Pets, Food-101). Against publicly available models such as NASNet-A and Inception-v4 they are more accurate on six of the eight, tie on Flowers and trail by 0.2 points on FGVC Aircraft, with 4.7 times fewer parameters on average (up to 21 times). Against the best reported results, including GPipe and domain adaptive transfer learning, the paper counts five of eight wins. Three are strict (CIFAR-100 91.7% vs 91.3%, Birdsnap 84.3% vs 83.6%, Flowers 98.8% vs 97.7%), two are exact ties (FGVC Aircraft 92.9%, Food-101 93.0%), and the remaining three are losses of 0.1 to 0.5 points. Where parameter counts are known, the EfficientNets use 9.6 times fewer on average.

What later work changed

The rule budgets FLOPS, while training time also depends on memory traffic and on how well each operation uses the hardware. Two years later the same authors listed three problems with training EfficientNets in the EfficientNetV2 paper. Very large images are slow: B7 trains at 600 pixels, its activations fill accelerator memory, and the batch has to shrink. Depthwise convolutions do little arithmetic per byte of memory they read, so GPUs and TPUs run them far below peak, especially in the early high-resolution stages. And scaling every stage by the same factor is sub-optimal. EfficientNetV2 replaces the early MBConv layers with a fused version (one regular 3×3 convolution instead of the 1×1 expansion plus depthwise), caps the image size at 480, and adds layers mainly to later stages.

Bello et al. (2021) revisited ResNets with modern training and found that the best scaling strategy depends on the regime: add depth where overfitting can occur (long training runs), add width otherwise, and raise resolution more slowly than compound scaling does. Their ResNet-RS models run 1.7 to 2.7 times faster than EfficientNets on TPUs at similar accuracy. Both findings narrow the paper's claim without contradicting its Observation 2: the dimensions do interact, but one fixed ratio searched at $\phi = 1$ is a heuristic, and the released B4 to B7 already depart from it.

The question the paper asked, how to spend more compute on the same architecture, came back for language models as neural scaling laws and Chinchilla, which split a training budget between parameters and data instead of between depth, width and resolution. Image classification research moved largely to transformers after the Vision Transformer (2020); the same group's EfficientDet detectors (Tan, Pang & Le, 2020) use EfficientNet backbones and their own compound scaling rule.