The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

A random dense network contains a subnetwork one tenth its size that trains as well on its own, provided it keeps its original initial weights.

Frankle and Carbin train a network, delete the 20% of weights with the smallest magnitudes in each layer, rewind the rest to their values at iteration 0, and retrain. Repeated, this finds subnetworks of 3.6% to 20% of the original size that learn faster and score higher than the full network, and the same subnetworks fail when their weights are randomly re-drawn.

Why a pruned network trains from scratch only when it starts from the initialization it was pruned out of, and what that says about the 90% of weights training seems not to need.

Pruned networks train badly from scratch

Pruning removes weights from a trained network. The standard recipe, from Han et al. (2015), has three steps: train the dense network, delete every connection whose weight has magnitude below a threshold, then keep training the survivors from their trained values. On Lenet-300-100, a fully connected network for MNIST with hidden layers of 300 and 100 units, that recipe takes the parameter count from 267K to 22K with no loss of accuracy. On AlexNet it removes 9 of every 10 weights, on VGG-16 12 of every 13.

So why train the big network at all? A 22K-weight network that classifies MNIST needs roughly a twelfth of the multiply-adds per training step. The pruning literature's answer was that training it directly does not work. Han et al. report that after pruning it is better to keep the surviving weights than to re-initialize them: gradient descent finds a good solution from the original dense start but not after re-initializing pruned layers. Li et al. (2016) write that training a pruned model from scratch performs worse than retraining the pruned model.

Frankle and Carbin change one step. They train the dense network and prune its smallest-magnitude weights, and then, instead of keeping the survivors' trained values, they reset each surviving weight to the value it had before training started, at iteration 0. The pruned network trains from that state with the pruned weights held at zero. On the networks in the paper this pruned-and-reset network trains at least as fast as the dense one and reaches at least its test accuracy, down to 10% to 20% of the original weight count, and on Lenet down to 3.6%. Re-draw the surviving weights from the same random distribution instead, keeping the same sparse structure, and the subnetwork learns slower and scores lower, from the first pruning round onward.

A pruned network trains badly from a fresh random start and trains well from the specific random start it was pruned out of. The rest of this page states that claim precisely, walks the procedure that finds these subnetworks, works the numbers on Lenet, reads the results and the control experiments, and covers where the procedure breaks on deeper networks and what the paper claims versus what it conjectures.

The hypothesis, stated precisely

Take a dense feed-forward network $f(x;\theta)$ with parameters $\theta$. Its initial values are a random draw, $\theta_0 \sim \mathcal{D}_\theta$; for every experiment in the main text $\mathcal{D}_\theta$ is a zero-mean Gaussian with the Glorot variance $2/(n_{\text{in}} + n_{\text{out}})$ per layer, the scale at which activations and gradients keep the same variance from layer to layer. Train it with stochastic gradient descent on a training set. Its validation loss falls, bottoms out, and rises as the network overfits; call the iteration of that minimum $j$ (an iteration is one minibatch gradient step) and the test accuracy there $a$.

A subnetwork is the same function with a fixed binary mask over the parameters:

Here $\odot$ is elementwise multiplication, so a 0 in $m$ zeroes that weight for good, and $\lVert m \rVert_0$ counts the ones. $P_m$ is the fraction of weights remaining. The paper calls $P_m$ the sparsity of the mask, which is the opposite of the usual sense; read every $P_m$ on this page as "fraction kept", so $P_m = 25\%$ means 75% of the weights are gone. Train the masked network from the masked initialization $m \odot \theta_0$, with $m$ held fixed, and record its own minimum-validation-loss iteration $j'$ and test accuracy $a'$ there. The hypothesis is an existence claim about the mask:

All three conditions hold at once, for a single mask, with the subnetwork starting from the values $\theta_0$ the dense network started from. The paper names such a subnetwork a winning ticket: in a large random draw of initial weights, some small combination of connections and values happened to be one that trains well, and the dense network contains it the way a stack of lottery tickets contains the winning one. The analogy covers only the draw: the values were fixed at iteration 0, before any data was seen, and nothing is drawn again afterwards.

The three weight matrices of Lenet have $784 \times 300$, $300 \times 100$ and $100 \times 10$ entries, so $|\theta| = 266{,}200$. A mask with $P_m = 21.1\%$ has 56,095 ones and 210,105 zeros. The hypothesis predicts that for a well-chosen such mask, the 56,095 surviving weights, started from their Glorot draws, reach minimum validation loss no later than the full 266,200 did and score no worse on the 10,000 MNIST test images. Section 2 of the paper finds masks that reach it 38% earlier and score higher.

Train, prune, reset

The hypothesis says a good mask exists; it does not say how to find one. The paper's procedure uses the trained dense network as the search: weights that end training small are the ones to drop. Four steps, one round:

1. Randomly initialize $f(x;\theta_0)$ with $\theta_0 \sim \mathcal{D}_\theta$, and save $\theta_0$.
2. Train for $j$ iterations, reaching $\theta_j$.
3. Prune $p\%$ of the weights of $\theta_j$ by magnitude, producing the mask $m$.
4. Reset the surviving weights to their values in $\theta_0$. The ticket is $f(x;\, m \odot \theta_0)$.

Step 3 is layer-wise magnitude pruning as in Han et al.: within each layer, sort the surviving weights by absolute value and drop the smallest $p\%$. The official code does this. For each layer it sorts abs(final_weight[mask == 1]), takes the value at index round(percent * size) as the cutoff, and zeroes every mask entry whose weight has magnitude at or below the cutoff. Because the sort runs over the currently unmasked weights only, the percentage is always of the survivors, which matters once the procedure iterates. Connections into the 10 output units are pruned at half the rate of the other layers (10% per round instead of 20%), because the output layer holds only 1,000 of the 266,200 weights, and at 20% per round it would be down to 35 connections for 10 output units after 15 rounds, against 206 at 10%.

Step 4 departs from Han et al. The magnitudes of $\theta_j$ decide which connections survive, and then the values of $\theta_j$ are thrown away. In the code the training function returns the weights it read back immediately after the variable initializer ran, before the first gradient step, and every later round passes that same dictionary in as the preset initializer, so "reset" means the exact floating-point values of iteration 0, not a new draw from the same distribution. A masked layer multiplies its weight matrix by a non-trainable mask in the forward pass:

# masked dense layer (foundations/model_base.py, lines 93-116)
W = get_variable([fan_in, units], init=constant(preset))   # preset = theta_0
m = get_variable(W.shape, init=constant(mask), trainable=False)
y = matmul(x, W * m) + b
# dL/dW = (x^T dL/dy) * m : a pruned entry gets gradient 0 and never moves

The gradient with respect to W is the dense gradient multiplied elementwise by the mask, so a pruned entry receives zero gradient, stays at whatever value it holds, and contributes nothing to the output. With Adam, a zero gradient gives zero first and second moments and a zero update, so the same holds. Biases are left unmasked and start at zero in every round, which is also their initialization. Nothing else in training changes: same optimizer, same learning rate, same batch size, same number of iterations as the dense run.

The figure runs one round of this on a synthetic 12-input, 8-output layer with a Glorot initialization of standard deviation $\sqrt{2/20} = 0.316$. The trained panel marks the weights the cutoff will remove with a cross. The third panel shows what the pruned layer starts from under three protocols: the paper's reset to $\theta_0$, the control that redraws $\theta_0'$ from the same Gaussian under the same mask, and the Han et al. fine-tuning that keeps $\theta_j$. Click any cell to read its initial value, trained value and rank.

Two things to notice in the third panel with reset selected. The surviving initial values are not the largest initial values; some of the largest entries of $\theta_0$ are pruned because training shrank them, and some small ones survive because training grew them. And the third panel under reset and under redraw looks statistically identical, two samples from the same Gaussian on the same support. The control experiments below compare those two panels at every size.

Iterating: 20% per round

Pruning 80% in one shot works, but pruning 20% and repeating works better. Iterative pruning runs the four steps as a loop: train, prune 20% of the survivors in each layer, reset to $\theta_0$, train again from the smaller mask, prune again. The mask only ever loses ones. Every round starts from the same saved $\theta_0$, so the ticket after round $r$ is the pair (mask after $r$ prunings, $\theta_0$), and the trained weights of each round are used for one thing, choosing the next mask.

# iterative magnitude pruning with resetting (Appendix B, Strategy 1)
theta0 = glorot_normal(shapes)            # sampled once, kept for every round
mask   = {k: ones_like(v) for k, v in theta0.items()}
thetaj = train(theta0, mask, iters=50_000)          # round 0: the dense net
for r in range(1, n_rounds + 1):
    for k in mask:                        # per layer, survivors only
        alive  = sort(abs(thetaj[k])[mask[k] == 1])
        cutoff = alive[round(rate[k] * alive.size)]  # rate: 0.2 (0.1 output)
        mask[k] = where(abs(thetaj[k]) <= cutoff, 0, mask[k])
    thetaj = train(theta0, mask, iters=50_000)      # reset, then retrain
# the ticket after round r is (mask, theta0); thetaj only picks the next mask

Because each round prunes a fraction of the survivors, the fraction remaining compounds. A hidden layer keeps $0.8^n$ of its weights after $n$ rounds and the output layer keeps $0.9^n$, so on Lenet

which gives 51.3% at $n = 3$, 21.1% at $n = 7$, 13.5% at $n = 9$ and 3.6% at $n = 15$, the four sizes the paper's Figure 3 labels. Reaching 3.6% takes 16 full trainings of 50,000 iterations, one dense and fifteen pruned, and the paper runs five trials of each, which is why Section 6 lists the cost of the search as the reason ImageNet is left out. Drag the round count and read the count per layer:

The comparison with one-shot pruning is in Section 2. One-shot tickets reach minimum validation loss earlier than the dense network only while $67.5\% \ge P_m \ge 17.6\%$ and exceed its test accuracy only while $95.0\% \ge P_m \ge 5.17\%$; the iterative tickets keep both advantages further, to 3.6%. Appendix G.4 varies the per-round rate: 10% and 20% per round keep accuracy and speed to the smallest sizes, 40% and above lose them earlier, and the paper settles on 20% as the rate that keeps most of what 10% gives at half the number of trainings. Appendix B tests the other obvious loop, in which each round continues from the trained weights and only the final network is reset to $\theta_0$; resetting every round holds accuracy and learning speed to smaller networks on Lenet and on all three convolutional networks. A plausible reason, which the paper does not test: pruning a network that was trained from $\theta_0$ selects connections by how their initial values trained, and the reset then reuses those values.

Measuring learning speed

Condition $j' \le j$ in (2) needs a definition of how fast a network learns. The paper uses the iteration at which an early-stopping rule would halt: the iteration of minimum validation loss. On MNIST the 60,000 training images are split into 55,000 for training and 5,000 for validation, with the 10,000 test images untouched; validation and test performance are evaluated every 100 iterations. Training never actually stops early. Every run goes the full 50,000 iterations (20,000 to 30,000 for the convolutional networks), and the early-stopping iteration is read off the saved validation curve afterwards. A network that hits its validation-loss minimum at iteration 6,000 instead of 10,000 counts as learning 1.67× faster, or 40% earlier.

Appendix C checks that this proxy tracks the obvious alternative: the order in which the pruned networks reach their early-stopping iteration in Figure 3 of the paper is the same as the order in which they cross a fixed test accuracy threshold. Accuracy is always reported at the early-stopping iteration, so a faster network is not being credited with the accuracy of a longer run. For VGG-19 and Resnet-18 the criterion is dropped, because their learning-rate schedules step down at fixed iterations and the validation minimum lands wherever the schedule puts it; Section 4 reports accuracy at fixed checkpoints instead.

What the tickets do

On Lenet with Adam at learning rate 0.0012 and batch size 60, tickets learn faster as they shrink from 100% to 21% of the weights, where early stopping arrives 38% earlier than for the dense network. Below 21% learning slows again, and at 3.6% the early-stopping iteration is back at the dense network's. Test accuracy at early stopping rises with pruning to a peak of more than 0.3 percentage points above the dense network at 13.5%, then falls back to the dense level, again at 3.6%. Every curve is the average of five trials, with error bars at the minimum and maximum trial.

The convolutional networks show the same shape with larger numbers. Conv-2, Conv-4 and Conv-6 are VGG-style stacks of two, four or six 3×3 convolutional layers (64, 128 and 256 channels, max-pooling after every pair) followed by two 256-unit fully connected layers and a 10-way output, trained on CIFAR10 with Adam. Their tickets reach minimum validation loss 3.5× sooner than dense at $P_m = 8.8\%$ (Conv-2), 3.5× at 9.2% (Conv-4) and 2.5× at 15.1% (Conv-6), and their best test accuracies are 3.4, 3.5 and 3.3 points above dense, at 4.6%, 11.1% and 26.4%. All three stay above the dense accuracy while more than 2% of the weights remain. Hover the marks:

At early stopping, training accuracy rises alongside test accuracy, which by itself would say the tickets optimize faster and nothing about generalization. The paper therefore also reports the end of training. At iteration 50,000 almost every Lenet network, dense or pruned to 2%, has 100% training accuracy, and the iteratively pruned tickets are still up to 0.35 points more accurate on the test set. The same holds for the convolutional networks at their final iterations: a smaller generalization gap at equal training accuracy.

Test accuracy against size traces a hill: up from the dense network, a peak somewhere between 5% and 30%, down again as the network runs out of capacity. Section 5 borrows Rasmussen and Ghahramani's name for this shape, Occam's hill, with the dense network on the too-complex side and the 2% network on the too-simple side. The paper leaves open why a subnetwork with a tenth of the weights generalizes better than the network it came from and offers two candidates: the mask, found using the training data, may encode an inductive bias for the task, and the initialization may land in a region the optimizer handles well.

Dropout, which disables a random half of the units at each training step and so trains a random subnetwork each time, raises the dense Conv-2/4/6 accuracies by 2.1, 3.0 and 2.4 points, and iterative pruning on top of dropout adds up to 2.3, 4.6 and 4.7 more. The two are complementary in these experiments; the speed gains shrink under dropout, to 1.58× for Conv-4 against 4.27× without dropout in Appendix H.5, which measures speed on the validation set and reports a larger Conv-4 gain than the main text's 3.5×.

Separating structure from initialization

A pruned-and-reset network differs from the dense one in which weights it has and in what those weights start from. The paper separates the two with one control: keep the mask, redraw the surviving values from $\mathcal{D}_\theta$, train, and compare. Each ticket is redrawn three times, so every point on the control curves averages 15 runs.

On Lenet the redrawn networks learn slower with every round of pruning instead of faster, and their test accuracy falls below the dense network's once $P_m$ reaches 21.1%, where the reset ticket is at its fastest; the reset ticket stays at dense accuracy down to 2.9%. At $P_m = 21\%$ the ticket reaches minimum validation loss 2.51× sooner than its redrawn twin and is half a point more accurate. Both have 100% training accuracy at iteration 50,000 for every size above 5%, so the gap is generalization. The mask and the distribution of starting values are the same in both runs, so the difference is which particular value sits on which connection.

The redrawn controls of Conv-2, Conv-4 and Conv-6 also slow down with pruning and lose accuracy sooner than the tickets. But at moderate pruning the redrawn Conv-2 and Conv-4 lose no early-stopping accuracy and even improve on the dense network, so on those networks a sparse network with fresh values beats the dense one for a while. Appendix E adds a third arm, a random mask of the same size with random values: on Lenet the pruned mask with redrawn values beats a random mask, on all the convolutional networks the two are indistinguishable. Structure carries some of the effect on Lenet, where only the central pixels of an MNIST digit carry information and a mask found by training encodes that, and close to none of it on the convolutional networks, where a filter is not tied to one location.

Appendix F narrows down what about the values matters, on Lenet. Adding Gaussian noise to the ticket's initialization with standard deviation half that of the layer's Glorot draw barely changes accuracy; noise of three times that standard deviation still leaves the ticket ahead of the redrawn control. Redrawing from the ticket's own empirical distribution of surviving values, which is bimodal with peaks on either side of zero in the second and output layers, does little better than redrawing from the Gaussian, so the ticket keeps the per-connection assignment of values, not only their distribution. Pruning the smallest initial magnitudes at iteration 0, before any training, does worse than the redrawn control. And the ticket's weights are not close to their trained values: over the dense training run, the weights that end up in the ticket move further than the weights that get pruned, and more often away from zero. The value on a connection has to be the one the dense run trained from, and the paper stops short of saying what property of that value does the work.

Deeper networks: global pruning and warmup

Section 4 moves to two networks with the ingredients of a practical CIFAR10 model: VGG-19 with 20.0 million parameters and the 20-layer Resnet-18 with 271K, both trained with batch normalization, weight decay 1e-4, flips and crops, SGD with momentum 0.9 and a learning rate that drops by 10× twice. VGG-19 trains for 160 epochs, 112,480 iterations at batch size 64, with drops after epochs 80 and 120; Resnet-18 trains for 30,000 iterations at batch size 128 with drops at 20,000 and 25,000, a schedule cut from the original 64,000 so that 15 to 30 consecutive trainings per experiment stay affordable.

Two changes to the procedure. Pruning is global: instead of removing the smallest 20% within each layer, the code pools every convolutional weight, finds one cutoff, and removes the smallest 20% of the pool. Never pruned: VGG-19's 5,120-weight output layer, Resnet-18's 640-weight output layer, and the 2,560 weights in Resnet-18's downsampling shortcuts. Layer-wise pruning has a bottleneck: the first convolutional layer of VGG-19 has 1,728 weights and the last 2,359,296, and at $P_m = 1.5\%$ the layer-wise rule leaves the first layer 26 of them. Global pruning lets the 20.0M-weight budget fall where the magnitudes put it. Appendix I.1 measures the difference: at learning rate 0.1 with warmup, global pruning yields winning tickets while $P_m \ge 1.5\%$ and layer-wise only while $P_m \ge 6.9\%$. Drag the round count to 19, where $P_m$ is 1.4%, and compare the first layer's count under the two rules:

The second change responds to a failure at the standard learning rate. At 0.1, iterative pruning finds no winning tickets in either network: the pruned-and-reset subnetworks do no better than their redrawn controls at any size. At learning rate 0.01 the pattern from the small networks returns. VGG-19 subnetworks stay within one point of the dense network while $P_m \ge 3.5\%$, though they never match it, and Resnet-18 subnetworks reach 89.5% for $41.7\% \ge P_m \ge 21.9\%$, above the dense network at 0.01 but below the 90.5% the dense network reaches at 0.1. Appendix I.4 tries the rates in between and finds tickets at none above 0.01.

Linear learning-rate warmup recovers the tickets: the rate climbs from 0 to its base value over the first $k$ iterations, then follows the usual schedule. With $k = 10{,}000$ at rate 0.1, the dense VGG-19 gains about one point and iterative pruning finds winning tickets down to $P_m = 1.5\%$. With $k = 20{,}000$ at rate 0.03, Resnet-18 tickets reach 90.5% at $P_m = 27.1\%$, matching the dense network at 0.1, and remain winning tickets down to 11.8%. No amount of warmup produces Resnet-18 tickets at rate 0.1 itself. Appendix I.5 sweeps $k$: accuracy improves quickly up to about 5,000 iterations and slowly after. Pick a network, a base rate and a warmup length, and the readout quotes the paper's result for that combination:

The paper does not explain why warmup is needed and lists it as future work. The follow-up by Frankle, Dziugaite, Roy and Carbin (2020) gives the mechanism the original could not: at learning rate 0.1 the dense network's early training is unstable to the random order of minibatches: two runs from the same $\theta_0$ with different minibatch orders reach minima with a loss barrier on the straight line between them. Iterative pruning only finds matching subnetworks once the network has become stable. On MNIST that is already true at iteration 0; on VGG-19 and Resnet-18 it becomes true a little way into training. Resetting to that early iteration instead of 0 uses the fact directly; the original paper's warmup and lower learning rate plausibly work by making the first iterations gentle enough that iteration 0 is close enough to the point where training has become stable.

Liu et al. (2019), the paper the VGG-19 setup is borrowed from, report that at the standard learning rate a pruned VGG-19 redrawn from scratch matches the dense network down to 20% of the weights, and that with the optimal learning rate the winning-ticket initialization brings no improvement over random. Frankle and Carbin agree on the overlap: at rate 0.1 their redrawn networks also match dense down to 20%, and their tickets do no better than the redrawn networks there. The two papers differ on what happens below 20%, where Liu et al. present no data and the lottery ticket experiment, with warmup, keeps finding tickets to 1.5% while the redrawn networks lose accuracy. Section 5 puts it as a hypothesis: a heavily overparameterized network can be pruned, redrawn and retrained down to some sparsity, and past that sparsity only the original initialization keeps accuracy.

What the hypothesis does and does not claim

The hypothesis as stated is an existence claim about dense, randomly initialized, feed-forward networks, and the paper's evidence for it is the set of tickets the procedure found on six architectures and two datasets. The headline "10% to 20% of the size" is the size at which every tested network still yields tickets; Lenet goes to 3.6%, VGG-19 with warmup to 1.5%, Resnet-18 with warmup only to 11.8%. The paper marks a second claim as a conjecture and does not test it: that SGD seeks out and trains a well-initialized subset of the network, and that dense networks are easier to train than pruned ones because a dense draw contains more candidate subsets.

The evidence supports a narrower statement. A magnitude-pruned network trained from its own iteration-0 values is a different object from the same network trained from a fresh draw, and on these benchmarks it is a better one: faster to the validation minimum, more accurate at the end, at a fraction of the size. The finding does not yet make training cheaper, because the ticket is found by training the dense network 15 to 30 times, and the unstructured sparse networks it produces run no faster on a GPU than the dense ones, because dense matrix kernels do not skip individual zero weights; Section 6 lists both as limitations.

The later work that built on this page's procedure is summarized in the questions below: resetting to an early iteration instead of iteration 0 extends the result to ImageNet-scale networks, the sign pattern of the initial values is what the small networks need, and a stronger form of the hypothesis was proven for sufficiently wide random networks. The procedure to reproduce the original result is the loop in the code listing above, with the per-round sizes of (3): 20% per round, reset to the saved $\theta_0$, 50,000 iterations of Adam at 0.0012 on Lenet, and the early-stopping iteration read off the validation curve afterwards.