Dropout: Improving neural networks by preventing co-adaptation of feature detectors

Switch off half the network at random on every training example, and it overfits far less.

At test time you run the full network once, with each hidden unit's outgoing weights halved. For a network with one hidden layer, that single pass computes exactly the combined opinion of every sub-network that training could have sampled.

The MNIST network in this paper has 2400 hidden units, so a coin flip per unit gives 2²⁴⁰⁰ different sub-networks to draw from. Training draws 180 million of them, 60,000 images seen 3,000 times over, one draw per example, and almost certainly never draws the same one twice.

In 2012 a preprint from Geoffrey Hinton's group at Toronto, six pages of argument followed by eleven of appendices, reported new records on speech and object recognition using a change you can write in one line. During training, before an activation is passed to the next layer, flip a coin for each hidden unit and zero out the losers, half of them on every example. The paper calls this dropout, and by the end of the decade it was in nearly every network anyone trained.

The paper gives two reasons dropout should help: a hypothesis about what goes wrong inside an overfitted network, and a small theorem about ensembles. It also uses three companion settings (a hard cap on each unit's weights, a learning rate of 10, momentum ramped to 0.99), each of which is explained by what dropout does to the optimization. We will go through all of it, and through the places where the paper's own text does not hold up to checking.

Many networks fit the training set

Start with the problem the title names. Take a network with more capacity than the task needs and a training set that is small relative to that capacity. The obvious response is to use a smaller network, and that does work; what you give up is capacity, which this same paper's ImageNet appendix says outright you may badly want. Keep the capacity and there will be many different settings of the weights that classify the training set almost perfectly, and they will disagree with each other on data none of them has seen. Almost all of them will do worse on held-out data than on the data they were fit to. That gap is overfitting, and the paper's opening paragraph ends on a claim about what almost all of those solutions have in common:

"the feature detectors have been tuned to work well together on the training data but not on the test data."

A hidden unit computes a number. Backpropagation computes each unit's update given what every other unit is currently doing, and the loss depends only on the total. So a network can settle on an arrangement where unit 3 systematically overshoots and unit 4 exists mainly to subtract unit 3's overshoot. Neither unit is individually sensible. Together they balance, on the training set. The paper's name for that arrangement is a co-adaptation: a feature detector that is only helpful in the context of several other specific feature detectors.

The figure below makes the arrangement literal, on a task small enough to solve on paper: fit one smooth curve over a single input $x$. Five hidden units, each a fixed function of $x$, feed one output unit that is free to weight them however it likes. Two pairs among the five are near duplicates of each other, which is the situation that lets cancellation happen. Solving for the output weights by ordinary least squares fits the target to ten decimal places, and it does so by giving one of those pairs weights of $+9.14$ and $-8.65$. Click any unit to remove it and watch what the remaining four do.

With every unit present, both networks track the amber target. Drop u3 and the violet curve leaves the panel: unit 4 is still contributing its $-8.65$ share with nothing left to cancel, and the mean squared error goes from 0.000 to 43. The target never leaves the band between $-1.11$ and $+1.11$, so an error of 43 means the output is off, on average, by about six times the height of the thing it is fitting. Drop both members of that pair and the plain network is almost fine again, at 0.155, because the pair was only ever cancelling itself. Drop one from each pair and the error reaches 88. The teal curve barely moves under any of these, because no unit in it was carrying anyone else's error. Dropout makes the context unreliable on purpose. If unit 4's job were to cancel unit 3, then half the time unit 3 is gone and unit 4's correction becomes an uncompensated error that raises the loss. Averaged over presentations, the weights that survive are the ones that lower the loss across the combinatorially many random committees a unit can land in.

The teal weights in the figure are computed exactly, because for a linear output unit the dropout objective has a closed form. Average the squared error over the masks and it splits the way any random quantity does, into the squared error of its average plus its variance around that average, and the variance term turns out to be a weighted L2 penalty. Let $q$ be the probability that a unit is kept, $r_j$ its 0/1 coin flip (1 with probability $q$), and $h_j(x)$ the fixed hidden features. Write the train-time weights $\tilde{w}_j$ and the weights used at test time, $w_j = q\,\tilde{w}_j$, and it reads:

At $q = 1/2$ the factor $(1-q)/q$ is exactly 1, so the teal solution is an ordinary ridge regression in which each weight is penalized in proportion to its own feature's mean square. This result is not in the 2012 paper; the exact linear-regression version appears in the journal version, and the general first-order statement for generalized linear models is Wager, Wang and Liang (2013). Quoting it here settles one thing: for a linear readout, the co-adaptation story and a penalty story are the same objective written two ways. In the hidden layers no such closed form exists, and whether dropout acts as a penalty there is still disputed.

None of that is a measurement, and the paper does not treat it as one. Appendix A.3 shows 100 first-layer filters from a 784-500-500 network trained each way, observes that the dropout ones look like pen strokes while the plain ones are harder to read, and concludes "This confirms that dropout indeed forces the discriminative model to learn good features which are less co-adapted." A picture of 100 filters confirms nothing, and no interpretability metric, sparsity statistic or repeated seed backs that panel up. The journal version two years later marks the mechanism as a hypothesis before showing its own version of the picture ("We hypothesize that for each hidden unit, dropout prevents co-adaptation by making the presence of other hidden units unreliable") and afterwards claims only that dropout "does break up co-adaptations, which is probably the main reason" for the gain. Treat co-adaptation as the authors' hypothesis about the mechanism. The second argument, in the next-but-one section, is a theorem.

One coin flip per unit, per example

Now the mechanics, in three details: what gets zeroed, where the mask is applied, and how often it is redrawn.

Dropout zeroes units, not weights. One Bernoulli variable per neuron gates all of that neuron's incoming and outgoing connections together, so the unit leaves the network with its whole wiring rather than losing individual edges. Masking individual weights instead is DropConnect, a different and later method (Wan et al., 2013). The mask multiplies the activation after the nonlinearity: zeroing the weighted sum going into a logistic unit would leave it outputting $f(0) = 0.5$, so the unit would still send a signal. And the mask is drawn fresh for every training case, in the paper's words "on each presentation of each training case", so a minibatch of 100 examples uses 100 different networks.

The paper describes all of this in prose and never writes the mask down as an equation. The one everyone quotes comes from the journal version, and reads:

In (2), $q$ is the probability of keeping a unit, $y^{(l)}$ is the vector of post-nonlinearity activations at layer $l$, and $\odot$ is element-wise multiplication. In the paper's fully-connected networks that means $q = 0.5$ on every hidden layer and $q = 0.8$ on the input pixels, and the softmax outputs are never masked, since you always need all ten class scores. The convolutional networks are the exception: their convolutional layers get no dropout, for reasons given with Figure 6 below. A warning about the letter $p$, which you will trip on if you read the appendices: in this paper $p$ means momentum. The dropout rate appears only as "0.5", "50%" and "20%" in prose. The journal version later uses $p$ for the keep probability, and PyTorch uses it for the zeroing probability, so the same symbol means three things across the literature. They agree at 0.5, which is why the clash only bites at the input layer: this paper's "20% dropout on visible units" is the journal's $p = 0.8$ and PyTorch's Dropout(0.2).

Backpropagation needs no new rule for any of this: a mask is an ordinary element-wise product, so its local derivative is the mask itself, and the only requirement is that the backward pass reuse the same mask the forward pass sampled:

A dropped unit contributes nothing downstream, so nothing comes back to it, so its incoming weights, its bias and its outgoing weights all sit out that example. Take the paper's 784-1200-1200-10 network: 784 pixels in, two hidden layers of 1200 units, ten classes out.

# one training case through 784-1200-1200-10, the paper's convention
x  = pixels                        # [784]
x  = x * bernoulli(0.8, 784)       # 20% of the pixels dropped
h1 = f(W1 @ x + b1)                # [1200]
h1 = h1 * bernoulli(0.5, 1200)     # 50% of the units dropped
h2 = f(W2 @ h1 + b2)               # [1200]
h2 = h2 * bernoulli(0.5, 1200)     # a fresh mask, not a copy of the last
z  = W3 @ h2 + b3                  # [10] logits; outputs are never dropped
loss = cross_entropy(z, label)
# The backward pass reuses the SAME masks. If unit j of layer 1 was
# zeroed, nothing downstream saw it and nothing flows back, so its 784
# incoming weights, its bias, and its 1200 outgoing weights in W2 all
# get exactly zero gradient from this example. No special calculus.

Each forward pass runs a sub-network, but the weight update does not. With a minibatch of 100 examples the chance that a given unit is dropped in all 100 is $2^{-100}$, so essentially every weight still receives gradient at every step.

Press Play below and watch the network change from case to case. Each scatter dot is the output of the sub-network that actually ran on that case; the teal line is what the full network computes with every unit kept and its outgoing weights scaled by the keep probability, which the paper calls the “mean network”. Two of the four numbers printed under the scatter strip, the spread of the scatter and the mask entropy in bits, change when you drag the drop probability, and between them they explain the paper's choice of 0.5.

At $q = 0.5$ all $2^{10}$ masks are equally likely, which spreads the draw as evenly as it can over all of them and puts the mask entropy at its largest, a full 10 bits for 10 units. The spread of the outputs, $\sqrt{q(1-q)\sum_j c_j^2}$, where $c_j$ is what unit $j$ adds to the output when it is kept, peaks there too. Push the slider either way and both fall, and both extremes fail: toward a drop probability of 0 the ensemble collapses to a single network and there is nothing left to average; toward a drop probability of 1 the surviving units carry so little of the signal that the task itself becomes impossible. The paper reports the same conclusion from experiments rather than from entropy: "more extreme probabilities tend to be worse, which is why we have used 0.5 throughout this paper." For the input layer it says the opposite, that it is "often better to retain more than 50% of the inputs", which is what the 20% pixel drop reflects. Inputs carry information no other unit can supply, so throwing half of them away costs more than it buys.

One pass instead of all of them

Training has now built an enormous population of networks that share one set of weights. Running all $2^N$ of them at test time is not an option, and sampling a few thousand and averaging is expensive. The paper runs exactly one network:

"we use the ‘mean network’ that contains all of the hidden units but with their outgoing weights halved to compensate for the fact that twice as many of them are active."

Halving first restores the scale: during training the next layer received $\sum_j r_j w_j y_j$ with each $r_j$ a coin flip, whose expectation is $q\sum_j w_j y_j$. Turn every unit back on and the sum is twice as large as anything the layer above saw during training, so halve the weights and the scale is restored. Biases are not scaled, and must not be: a bias is present under every mask.

# test time: every unit present, one pass, weights scaled instead
h1 = f((0.8 * W1) @ x + b1)        # the pixels survived 80% of the time
h2 = f((0.5 * W2) @ h1 + b2)       # h1 survived half the time
z  = (0.5 * W3) @ h2 + b3          # h2 likewise. Biases are NOT scaled:
                                   # a bias is present under every mask.
# The paper writes only the two halvings. The 0.8 on W1 that its own
# 20% pixel dropout requires never appears anywhere in the text.

For a network with a single hidden layer of $N$ units and a softmax output, halving does not approximate the ensemble, it computes the ensemble exactly, and one line of algebra shows why. The logit for class $k$ is a sum with one term per hidden unit, and the mask sets which terms are present:

The logit is affine in the mask, and averaging an affine function over the masks replaces each $m_j$ by its mean of $1/2$. Every term survives at half weight and the bias is untouched, which is exactly what halving the outgoing weights computes. Note where the averaging is happening: one step upstream of the softmax, on the logits. The activations $h_j$ are computed before any mask is applied, so the hidden nonlinearity plays no part in the identity.

That averaged logit then goes through the softmax, which turns an arithmetic mean of logits into something other than an average. Exponentiating a mean of logs gives a product:

Each subnetwork's own softmax normalizer is the same for every class, so it appears in both the numerator and the denominator of (5) and cancels exactly. What is left, $G_k$, is the geometric mean of the $2^N$ predicted distributions: the average taken in log space rather than in probability space. Combining that way makes a label plausible only if nearly every subnetwork gives it a decent probability, since a single near-zero entry drags the whole product down, whereas an ordinary average lets one large entry carry a label on its own.

The word the paper leaves out of "exactly equivalent to taking the geometric mean" is renormalized. A geometric mean of probability vectors is not itself a probability vector. Class by class, the geometric mean of those $2^N$ numbers is no larger than their plain average; the plain averages are themselves a probability vector, summing to exactly 1; so the geometric ones sum to some $Z \le 1$, with equality only when every subnetwork predicts identically. So the mean network computes the renormalized $G_k/Z$, and the paper's next claim follows from that division by $Z$.

That claim, attributed to Hinton's 2002 products-of-experts paper, is that the mean network assigns a higher log probability to the correct answer than the average of the log probabilities the individual dropout networks assign. Subtract one from the other and the missing normalizer is all that remains:

Both halves of that are easy to check in the figure. The gain is strictly positive unless every subnetwork agrees, which is the hedge the paper attaches ("Assuming the dropout networks do not all make identical predictions"). And the gain is the same number for every class, not just for the correct one, because $Z$ does not depend on $k$. The mean network's advantage is a uniform additive lift in log space: it lowers the expected log-loss below the average member's and leaves the ranking of labels unchanged. The figure's one knob scales the output weights, which is the same thing as dialling how far apart the 16 subnetworks are allowed to drift. Turn it and watch the three per-class gains stay locked together.

For regression, with linear output units, no exponentiation happens, so the mean network computes the literal arithmetic mean $\bar{y}$ of the subnetworks' outputs, and the comparison of squared errors is the ambiguity decomposition:

The average member's error equals the mean network's error plus the spread across members, and that spread has a closed form because the mask bits are independent coins: each unit contributes $q(1-q)$ times the square of its own contribution $w_j h_j$, which is a quarter of it at $q = 1/2$. Since a variance cannot be negative, the mean network is never worse. The paper writes "always better", which overstates it by exactly the case where the variance is zero, meaning every subnetwork outputs the same number. The paper attaches that same hedge to the classification claim one sentence earlier, then drops it here.

Where the exactness runs out

Equation (4) needed the logit to be affine in the mask, and that condition is narrow. It holds when dropout happens only in the layer feeding the readout, and when the input is not dropped. Break either and the mask has to pass through a nonlinearity before it reaches the logits, averaging no longer commutes with the forward pass, and halving the weights becomes an approximation.

The paper is careful here, scoping the claim to "networks with a single hidden layer of $N$ units and a ‘softmax’ output layer". What the paper does not say is that none of its own networks meets that description. Every architecture it reports has at least two hidden layers, and the best MNIST numbers additionally drop 20% of the input pixels, which makes the hidden activations themselves depend on the mask. Switch Figure 3 to two hidden layers, which enumerates all 256 masks, and both properties break at once: the mean network and the geometric mean now differ in the second decimal place, and the three per-class log-probability gains come apart. At the top of the weight-scale slider one of them even turns negative, so the guarantee that was a theorem in the one-layer case is not automatic in the two-layer one.

Two years later the journal version drops the phrase "exactly equivalent" and renames the method accordingly: an "approximate averaging method" that "approximat[es] an equally weighted geometric mean". That version also measures the gap, by sampling subnetworks and averaging their predictions properly, and finds Monte-Carlo averaging catches up with weight scaling at around 50 samples and then stays within one standard deviation of it, while weight scaling costs one forward pass instead of fifty.

A hard cap on each unit's weights

Three companion settings sit in the appendices: the weight constraint, the learning rate, and the momentum.

The standard way to keep weights small is to add $\tfrac{\lambda}{2}\|w\|^2$ to the loss, so the gradient acquires a term pulling every weight toward zero. The paper does not do that. Appendix A.1 says flatly "No weight decay is used." Instead:

"we set an upper bound on the L2 norm of the incoming weight vector for each individual hidden unit. If a weight-update violates this constraint, we renormalize the weights of the hidden unit by division. Using a constraint rather than a penalty prevents weights from growing very large no matter how large the proposed weight-update is."

In symbols, each hidden unit's incoming weight vector is projected back onto a ball after every update:

Note the square. Appendix A.1 constrains "a maximum squared length of $\ell$" and reports that $\ell = 15$ worked best in cross-validation, so the bound on the norm itself is $\sqrt{15}\approx 3.87$, not 15. Reading 15 as a norm inflates the ball by a factor of 3.87 and puts it far outside the range anyone uses. The journal version writes the same device in un-squared units, $\|w\|_2 \le c$, and runs it at $c = 2$ on MNIST and $c = 4$ on speech, which brackets $\sqrt{15}$ neatly. For scale: at initialization the weights are drawn from $\mathcal{N}(0, 0.01^2)$ with 784 inputs, so a fresh unit's incoming vector has norm about 0.28, about fourteen times smaller than the ceiling. The constraint does nothing until training pushes a unit into it.

A penalty and a constraint come apart after one very large step. A penalty is a restoring force proportional to how far out you already are, so a single violent update can still fling a weight arbitrarily far, and the penalty pulls it back only over many later steps. A projection bounds you at the end of every step, whatever the step was. The same distinction shows up in decoupled weight decay, where the argument is about whether the shrinkage should be entangled with the gradient.

The constraint makes the second setting, a learning rate of 10, workable. The MNIST runs start at a learning rate of $\epsilon_0 = 10.0$, applied to the minibatch-average gradient, decayed by a factor of 0.998 after every epoch. Over 3000 epochs that is $0.998^{3000} = 2.46\times10^{-3}$, so the rate ends at 0.0246, a 406-fold reduction with a half-life of 346 epochs. A rate of 10 would normally send a network to infinity in a few steps. It does not here, because whatever the update proposes, the projection puts the unit back inside its ball before the next forward pass. Drag the learning rate below to 10 and then switch to the penalty.

The figure shows two contrasts. The obvious one is stability: at the paper's rate of 10 the penalty is at a squared length of 11,800 after a single step and off the panel entirely by the third, while the constraint reads 15.00 after every step it ever takes. The less obvious one concerns what each method controls. The penalty, at rates where it stays bounded, leaves the squared length at 44 or 136 or whatever else the curvature of the loss, the value of $\lambda$ and the gradient noise between them produce. The constraint holds whatever number you gave it, at every rate. The paper wants the big rate for search: high rates early move the weight vector a long way around its sphere each step, and the decaying rate lets it settle, "allowing a far more thorough search of the weight-space than methods that start with small weights and use a small learning rate."

Every gradient, a different network

The third of those settings is momentum ramped from 0.5 up to 0.99, and the paper ties it directly to dropout. Ordinary stochastic gradient descent already has noisy gradients because each minibatch is a random sample of the data. Under dropout each minibatch also reports the gradient of a different network, and the paper says so when it explains the setting: "Using a high final momentum distributes gradient information over a large number of updates making learning stable in this scenario where each gradient computation is for a different stochastic network."

The update rule is written in Appendix A.1 like this:

with $\langle\cdot\rangle$ the average gradient over a minibatch of 100. The unusual part is the $(1-p^t)$ factor on the gradient, which the prose states as "the learning rate is multiplied by a factor of (1-momentum)". Unroll (9) and it says the step is an exponential moving average of past gradients with weights summing to one, which is the same object as Adam's first moment, without the bias correction and without the second-moment divisor.

Textbook heavy-ball momentum, $\Delta w = p\,\Delta w - \epsilon g$, has a steady state of $-\epsilon g/(1-p)$, so at $p = 0.99$ it multiplies your step by a hundred, and a learning rate of 10 would mean an effective step of 1000. Equation (9) does not do that. Set $\Delta w^t = \Delta w^{t-1} = \Delta w^*$ for a constant gradient and the $(1-p)$ on the right cancels the $1/(1-p)$ the geometric sum would otherwise produce:

So at a learning rate of 10 the effective step is 10. The momentum is there to average, and the averaging is slow to arrive: it has a time constant of $1/(1-p)$ updates, which is 100 at $p = 0.99$, so the step only reaches its asymptotic size after a few hundred minibatches. Drag the slider below from 0 to 0.99 and watch the teal line go from a copy of the raw mask noise to a clean average, while the dashed steady-state line never moves.

The ramp itself contains a genuine typo: the printed formula and the prose one paragraph above it say opposite things. The prose: "The momentum starts off at a value of 0.5 and is increased linearly to 0.99 over the first 500 epochs, after which it stays at 0.99." The formula, with $p_i = 0.5$, $p_f = 0.99$ and $T = 500$:

Put $t=0$ into the printed version and you get 0.99, not 0.5. At $t = 499$ you get 0.50098, and then the second branch snaps back to 0.99 at $t = 500$. So as printed the momentum falls from 0.99 to 0.5 and jumps discontinuously back, which is the reverse of what the text says and would put the highest momentum at the point when the weights are moving fastest. Swapping the two subscripts fixes it: 0.5, then 0.745 at the halfway point, then 0.98902, continuous with 0.99 at $t = T$. The two forms agree exactly at $t = T/2$, which is the one place checking would not have caught it.

Five benchmarks at one setting

The paper makes its case by breadth: five datasets across three modalities, one hyperparameter held at 0.5 throughout, and improvement in the same direction everywhere. Pick a model below to see which layers were dropped and what the numbers were.

Take MNIST first, since the errors there are counts out of 10,000 test images and easy to hold in your head. The best published result for a plain feedforward network with no pretraining, no weight sharing and no extra training data was 160 errors, or 1.60%. Fifty percent dropout on the hidden units, with the per-unit weight constraint, brings that to about 130. Adding the 20% pixel dropout brings it to about 110. The paper does not cite the 160; it is Simard, Steinkraus and Platt (2003), an 800-unit two-layer network trained on 50,000 of the 60,000 training images, which matters because it makes the headline MNIST comparison one against somebody else's network rather than the same network run twice.

The same paragraph of the paper also runs a like-for-like comparison, which the 160-to-110 comparison hides. Salakhutdinov and Hinton's deep Boltzmann machine was pretrained five separate times, and a deep belief net once. Pretraining, in the 2012 sense, means fitting the layers one at a time without using the labels and keeping those weights as the starting point for ordinary supervised training; in the years before dropout it was how anyone got a deep network to train. Each of the six pretrained networks was then finetuned twice: once with ordinary backpropagation and once with 50% dropout, everything else held fixed. The averages below cover only the five deep Boltzmann machine pairs.

Appendix A.2 adds a small arithmetic slip: it writes "a mean of about 79 errors with dropout whereas usual finetuning gives about 94 errors." The five backpropagation runs are 103, 97, 94, 93 and 88, which average 95, not 94. Ninety-four is their median. The dropout side genuinely is a mean, since 83, 79, 78, 78 and 77 sum to 395. The like-for-like comparison is 95 to 79, an improvement of 16 errors rather than 15.

On TIMIT, a 4000-unit four-layer network pretrained as a deep belief net and finetuned two ways goes from 22.7% to 19.7%. The paper calls this a "recognition rate", which inverts the sense of the number, and calls it "frame"-level, which it is not: it comes out of a Viterbi decode of a whole utterance. The journal version lists the identical pair under a table headed "Phone error rate on the TIMIT core test set", so lower is better and the metric is per phone. The record claim is restricted: "a record for methods that do not use any information about speaker identity". Mohamed and colleagues had already reported 19.6% a year earlier using speaker-adapted, discriminatively transformed features.

The two vision results need reading with the appendices open. On ImageNet, the main text says the 48.6% baseline network has "two globally connected layers" and that dropout in "the sixth hidden layer" brings it to 42.4% top-1 error. Appendix H says something else: dropout goes in both of the 4096-unit globally-connected layers, and "Our model for ImageNet without dropout is similar, but without the two globally-connected layers which create serious overfitting when used without dropout." CIFAR-10 has the same shape of problem. The main text attributes the locally-connected fourth layer to the 16.6% baseline; Appendix G builds that baseline as three convolutional layers, three pooling layers and a softmax, and adds the fourth layer only for the dropout model, "because dropout imposes a strong regularization on the network, we are able to use more parameters."

So on both vision benchmarks the gap measures dropout plus the extra capacity dropout makes affordable, against no extra capacity. Appendix H states that reading in its own words ("using non-convolutional higher layers with a lot of parameters leads to a big improvement with dropout but makes things worse without dropout").

The fifth benchmark, Reuters, is the only text task: a 2000-2000-1000-50 network sorting 201,369 newswire stories into 50 mutually exclusive topics from nothing but 2000 word counts, going from 31.05% to 29.62% error on the same architecture trained twice. Reuters, TIMIT and the MNIST finetuning runs are the three places in the paper where the dropout and no-dropout runs share an architecture, and all three move the same way.

Auditing the four analogies

The discussion section offers four ways to think about dropout: Bayesian model averaging, bagging, naive Bayes, and the role of sex in evolution. The journal version two years later keeps two and drops two. The evolution analogy is promoted all the way to the front, into a Motivation section that opens the paper; the Bayesian comparison gets a section and an experiment of its own; bagging and naive Bayes disappear without a word, and the word "bagging" does not appear in the journal version.

None of the four supplied the mathematics of the previous sections. That came from reference [2], products of experts, cited back where the mean network was introduced. Combining distributions by a normalized geometric mean is what Hinton (2002) calls a logarithmic opinion pool, and the inequality that gives dropout its guarantee is a property of such pools in general. All the paper adds is the observation that a single halved-weight forward pass computes one.

The bagging analogy does not hold up. The paper says dropout "can be seen as an extreme form of bagging in which each model is trained on a single case and each parameter of the model is very strongly regularized by sharing it with the corresponding parameter in all the other models." Breiman's bagging draws bootstrap replicates of the entire training set, fits each model with its own parameters, and combines them arithmetically, and its variance-reduction argument depends on decorrelating unstable learners. Dropout's members share every weight, so they are as correlated as models can be; they are never individually fit; and they are combined geometrically. Coverage is illusory too: even the paper's smaller 784-800-800-10 MNIST network has $2^{1600}$ masks against $1.8\times10^{8}$ training presentations, so essentially every "member" of this ensemble is visited at most once and never revisited. Weight sharing compensates: one update to unit $j$'s weights improves every mask that contains $j$, which is half of all of them, so a member the optimizer never visits is still trained by every step that touches any of its units. Bagging has nothing like that, and it does not need it, because it only ever fits a few hundred models. Warde-Farley and colleagues untied the weights in 2014, training 360 independent networks on bootstrap resamples of MNIST, each with one dropout mask fixed in place, and combining them geometrically the way dropout implicitly does. That ensemble reached 1.66% test error against 1.06% for a single dropout-trained network. They do not read the gap as a refutation of the analogy; their section opens by calling dropout "bagging on an ensemble that is of size exponential in the number of hidden units, where each member of the ensemble shares parameters", and they read the result as evidence that the sharing is the active ingredient. They flag the obvious confound themselves: 360 members is a very small ensemble next to $2^{1600}$.

Naive Bayes is offered as "a familiar and extreme case of dropout" in which "each input feature is trained separately to predict the class label and then the predictive distributions of all the features are multiplied together at test time." As a picture of combining by multiplication it is fine. As a description of naive Bayes it is off by a prior: multiplying per-feature posteriors $p(c\mid x_i)$ counts the class prior once per feature, and matches naive Bayes only when that prior is uniform.

The last analogy has stuck the hardest. Reference [17] is a theory of why sex exists: recombination breaks up co-adapted gene complexes, so selection comes to favour alleles with high average fitness across many genetic backgrounds over alleles with peak fitness in one. Swap "allele" for "hidden unit" and "genetic background" for "which other units happen to be present" and the structure is the co-adaptation argument. It is a good analogy for the mechanism, and the paper then pushes it one step too far by calling brittleness under environmental change "a phenomenon which is known as ‘overfitting’ in the field of machine learning." Overfitting is a train-and-test gap under one fixed distribution, which is what every number in this paper measures; robustness to a shifting environment is a different thing, and the cited authors never use the word. While we are on that reference, do not copy its author list, or most of the others: the preprint's bibliography has absorbed Jonathan Dushoff's initials into Adi Livnat's name and dropped him as an author, and does something similar to at least three other entries.

What the field kept

This paper rescales at test time, by halving the outgoing weights. Every modern framework rescales at training time instead, dividing the surviving activations by the keep probability so that inference needs no adjustment and the rate can be changed without touching inference code. That variant is called inverted dropout, and it defines the same function. Write $W$ for this paper's weights and $W'$ for the weights an inverted-dropout network learns; with $W' = qW$, the two schemes produce identical forward passes mask for mask and an identical test network.

The optimization runs differ, though, and the difference shows up when you port an old recipe. Since $\partial L/\partial W' = q^{-1}\,\partial L/\partial W$, a nominal step of $\epsilon$ on the inverted weights corresponds to a step of $\epsilon/q^2 = 4\epsilon$ on this paper's weights at $q = 0.5$. Two factors of $1/q$, for two separate reasons: the gradient with respect to $W'$ is $1/q$ times the gradient with respect to $W$, and a displacement in $W'$ implies a displacement $1/q$ times larger in $W$, since $W = W'/q$. Any bound on weight magnitude rescales the same way: the squared-length bound $\|w\|^2\le15$ here becomes $\|w'\|^2\le 3.75$ there, so the same function is reached along a different trajectory with different constants.

The rate itself came down. Fifty percent was tuned for fully-connected layers on datasets of tens of thousands of examples, and even inside this paper the convolutional layers get none, for the reason Appendix F gives: weight sharing already limits how much a convolutional layer can overfit. VGG kept 0.5, but only on its first two fully-connected layers. The Transformer uses 0.1 on the output of every sub-layer and on the embedding sums, and BERT uses 0.1 throughout. Modern large-scale language pretraining, where the model sees each token roughly once and overfitting is not the binding constraint, often sets it to zero.

Dropout also stopped being the only tool. Batch normalization claimed in 2015 that it "also acts as a regularizer, in some cases eliminating the need for Dropout", a claim to read exactly as written: in some cases, and their own ensemble models still used dropout at 5% or 10%.

And the ImageNet network in Appendix H has a direct descendant. Five months later, three of the same authors published AlexNet, which won ILSVRC-2012. It carries this paper's max-with-zero nonlinearity, its local response normalization, its 224×224 crops from 256×256 images with horizontal reflections, its ten-crop test-time averaging, and 50% dropout in both 4096-unit fully-connected layers, of which AlexNet's own text says that without it "our network exhibits substantial overfitting". AlexNet cites this preprint. The two are not the same network: this one has 64 banks of 11×11 filters where AlexNet has 96, reports 42.4% top-1 on ILSVRC-2010 where AlexNet reports 37.5%, and includes a second input path from a 56×56 subsampled copy of the image that AlexNet drops.

The paper everyone reads, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", is a different paper: Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov, JMLR 2014, with a different first author, most of the experiments redone, and whole sections this preprint does not contain (dropout restricted Boltzmann machines, multiplicative Gaussian noise, the marginalized linear-regression form, the dropout-rate sweep, the sparsity measurements, and the ablation separating dropout from the max-norm constraint). Its reference list contains no entry for arXiv:1207.0580.