The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

A random dense network contains a subnetwork one tenth its size that trains as well on its own, provided it keeps its original initial weights.

Frankle and Carbin train a network, delete the 20% of weights with the smallest magnitudes in each layer, set the rest back to their values at iteration 0, and repeat. Down to 3.6% of the weights on MNIST, the subnetworks this finds reach their best validation loss no later than the full network and score at least as high on the test set; at 21% they get there 38% sooner.

Give the same subnetworks freshly drawn starting values and they learn slower and score lower. The result depends on which random number each surviving connection started from.

Why pruned networks were trained dense first

Pruning removes weights from a trained network. The standard recipe in 2018 came from Han et al. (2015) and has three steps: train the dense network, delete every connection whose weight has magnitude below a threshold (a quality parameter times the standard deviation of that layer's weights), then keep training the survivors from their trained values. On Lenet-300-100, a fully connected MNIST classifier with hidden layers of 300 and 100 units, the recipe takes the weight count from 267K to 22K, 12× fewer, with no loss of accuracy. On AlexNet it removes 9× (61M to 6.7M, about 89% of the weights, 8 of every 9), and on VGG-16 13× (138M to 10.3M, 12 of every 13).

If a 22K-weight network classifies MNIST as well as the 267K-weight one, it needs about a twelfth of the multiply-adds per training step, and training it directly would be cheaper. The pruning literature reported that this does not work. Han et al. write that it is better to retain the weights from the initial training phase than to re-initialize the pruned layers, and Li et al. (2016) that training a pruned model from scratch performs worse than retraining a pruned model.

Frankle and Carbin change one step of the recipe. They train the dense network and prune its smallest-magnitude weights as before. Then, instead of keeping the survivors' trained values, they set every surviving weight back to the value it had at iteration 0, before training started, and train that sparse network with the pruned weights held at zero. On the paper's networks this pruned-and-reset network trains at least as fast as the dense one and reaches at least its test accuracy down to 10% to 20% of the original weight count (on Lenet down to 3.6%), although the two deepest networks, VGG-19 and Resnet-18, need a lower learning rate or learning-rate warmup for it to work. Give the same sparse network new values drawn from the same random distribution and it learns slower and scores lower.

The sections below state the claim formally, walk through the procedure that finds these subnetworks with the Lenet numbers, read the results and the control experiments (including a small run of the same experiment you can step through), and then cover where the procedure breaks on deeper networks and what the paper claims versus conjectures.

The hypothesis, stated precisely

Take a dense feed-forward network $f(x;\theta)$ with parameters $\theta$. Its initial values are a random draw $\theta_0 \sim \mathcal{D}_\theta$. In every experiment in the paper $\mathcal{D}_\theta$ is the Gaussian Glorot initialization: zero mean and variance $2/(n_{\text{in}} + n_{\text{out}})$ for each layer, the scale at which activations and gradients keep roughly the same variance from layer to layer. Train the network with stochastic gradient descent. Its validation loss falls, bottoms out and rises again as the network starts to overfit. Call the iteration of that minimum $j$ (an iteration is one minibatch gradient step) and the test accuracy at that iteration $a$.

A subnetwork is the same function with a fixed binary mask over the parameters:

$\odot$ is elementwise multiplication, so a 0 in $m$ zeroes that weight permanently, and $\lVert m \rVert_0$ counts the ones. $P_m$ is therefore the fraction of weights that remain. The paper calls $P_m$ the "sparsity" of the mask, the reverse of the usual meaning, and gives the example $P_m = 25\%$ when 75% of the weights are pruned. Every $P_m$ on this page means fraction kept.

Now train the subnetwork from the masked initialization $m \odot \theta_0$, with $m$ held fixed, and record its own minimum-validation-loss iteration $j'$ and its test accuracy $a'$ there. The hypothesis says a mask exists that satisfies three conditions together:

All three hold for one mask at once, and the subnetwork starts from the same $\theta_0$ the dense network started from. The paper calls such a subnetwork a winning ticket. The name comes from an analogy: a large random initialization is like buying many lottery tickets, and some small combination of connections and starting values in it happens to be one that trains well. The analogy stops at the draw. The values were fixed at iteration 0, before any data was seen, and nothing is redrawn later.

A worked case on Lenet. Its three weight matrices are $784 \times 300$, $300 \times 100$ and $100 \times 10$, so $|\theta| = 266{,}200$. A mask with $P_m = 21.1\%$ keeps 56,095 weights and zeroes 210,105. The hypothesis predicts that for some such mask the 56,095 surviving weights, started from their own Glorot draws in $\theta_0$, reach minimum validation loss no later than the full network does and score no worse on the 10,000 MNIST test images. Section 2 of the paper finds a mask of that size that reaches minimum validation loss 38% earlier than the dense network, which is 1.61× faster.

How to find a winning ticket: train, prune, reset

Equation (2) says a good mask exists and gives no way to find one. The paper uses the trained dense network as the search: the connections whose weights end training with small magnitudes are the ones to drop. One round has four steps:

1. Randomly initialize $f(x;\theta_0)$ with $\theta_0 \sim \mathcal{D}_\theta$, and save $\theta_0$.
2. Train for $j$ iterations, reaching $\theta_j$.
3. Prune $p\%$ of the weights of $\theta_j$ by magnitude, producing the mask $m$.
4. Reset the surviving weights to their values in $\theta_0$. The ticket is $f(x;\, m \odot \theta_0)$.

Steps 3 and 4 on one row of ten weights, with $p = 20\%$ and these initial and trained values:

The two smallest trained magnitudes are 0.02 (position 5) and 0.03 (position 2), so the mask is $m = [1,0,1,1,0,1,1,1,1,1]$, and the ticket starts from

Position 5 started at 0.20, one of the larger initial values, and is pruned because training shrank it to 0.02. Position 3 started at 0.05, the smallest, and survives because training grew it to 0.27. Han et al.'s fine-tuning would continue from $\theta_j$ instead, and the paper's control experiment would replace the eight surviving values with eight new draws from the same Gaussian.

The official code prunes per layer. It sorts abs(final_weight[mask == 1]), takes the value at index round(percent * size) as the cutoff, and zeroes every mask entry whose weight has magnitude at or below the cutoff. Because the comparison is <=, it removes one weight more than round(percent * size): in the ten-weight row above it would also drop position 6 (0.11), and on Lenet's first layer in round 1 it removes 47,041 weights instead of 47,040. Because the sort runs over the currently unmasked weights only, the percentage is always a percentage of the survivors, which matters once the procedure repeats. Connections into the 10 output units are pruned at half the rate, 10% per round instead of 20%, because the output layer holds only 1,000 of the 266,200 weights. At 20% it would be down to 35 connections for 10 output units after 15 rounds; at 10% it keeps 206.

"Reset" means the exact floating-point values of iteration 0, not a new draw from the same distribution. In the code the training function reads the weights back immediately after the variable initializer runs, before the first gradient step, and every later round passes that same dictionary in as the preset initializer. A masked layer multiplies its weight matrix by a non-trainable mask in the forward pass:

# masked dense layer (foundations/model_base.py, lines 93-116)
W = get_variable([fan_in, units], init=constant(preset))   # preset = theta_0
m = get_variable(W.shape, init=constant(mask), trainable=False)
y = matmul(x, W * m) + b
# dL/dW = (x^T dL/dy) * m : a pruned entry gets gradient 0 and never moves

The gradient with respect to W is the dense gradient multiplied elementwise by the mask, so a pruned entry receives zero gradient and contributes nothing to the output. With Adam a zero gradient leaves both moment estimates at zero and the update at zero, so the same holds. Biases are not masked and start at zero in every round, which is also their initialization. Nothing else about training changes: same optimizer, learning rate, batch size and number of iterations as the dense run.

Figure 1 runs one round on a synthetic 8×12 layer with Glorot standard deviation $\sqrt{2/20} = 0.316$. The middle panel crosses out the weights the cutoff removes. The third panel shows what the pruned layer starts from under the three protocols above: reset to $\theta_0$, a fresh draw $\theta_0'$ under the same mask, or the trained $\theta_j$. Click cells in the first two panels to find large initial values that were pruned and small ones that survived.

With reset selected and then redraw, the third panel changes values but not its statistics: both are samples from the same Gaussian on the same set of connections. Any difference in how the two train has to come from which value sits on which connection, and the control experiments later on this page measure that difference at every size.

Iterative pruning, 20% per round

Pruning 80% of the weights in one shot finds tickets, but pruning 20% and repeating finds smaller ones. Iterative pruning runs the four steps as a loop: train, prune 20% of the survivors in each layer, reset to $\theta_0$, train again with the smaller mask, prune again. The mask only ever loses ones. Every round starts from the same saved $\theta_0$, so the ticket after round $r$ is the pair (mask after $r$ prunings, $\theta_0$), and each round's trained weights are used only to choose the next mask.

# iterative magnitude pruning with resetting (Appendix B, Strategy 1)
theta0 = glorot_normal(shapes)            # sampled once, kept for every round
mask   = {k: ones_like(v) for k, v in theta0.items()}
thetaj = train(theta0, mask, iters=50_000)          # round 0: the dense net
for r in range(1, n_rounds + 1):
    for k in mask:                        # per layer, survivors only
        alive  = sort(abs(thetaj[k])[mask[k] == 1])
        cutoff = alive[round(rate[k] * alive.size)]  # rate: 0.2 (0.1 output)
        mask[k] = where(abs(thetaj[k]) <= cutoff, 0, mask[k])
    thetaj = train(theta0, mask, iters=50_000)      # reset, then retrain
# the ticket after round r is (mask, theta0); thetaj only picks the next mask

Each round removes a fraction of the survivors, so the fraction remaining compounds. A hidden layer keeps $0.8^n$ of its weights after $n$ rounds and the output layer keeps $0.9^n$. On Lenet, whose two hidden layers hold 265,200 of the weights and whose output layer holds 1,000:

This gives 51.3% at $n = 3$, 21.1% at $n = 7$, 13.5% at $n = 9$ and 3.6% at $n = 15$, the four sizes labeled in the paper's Figure 3. Reaching 3.6% takes 16 full trainings of 50,000 iterations (one dense, fifteen pruned), and the paper runs five trials of each, 80 trainings in all. That cost is the reason Section 6 gives for leaving ImageNet out. Drag the round count in Figure 2 and read the count per layer.

Section 2 compares this with one-shot pruning. One-shot tickets reach minimum validation loss earlier than the dense network only while $67.5\% \ge P_m \ge 17.6\%$ and beat its test accuracy only while $95.0\% \ge P_m \ge 5.17\%$. Iterative tickets keep both advantages down to 3.6%. Appendix G.4 varies the per-round rate: 10% and 20% keep accuracy and speed to the smallest sizes, 40% and above lose them earlier, and the paper uses 20% because it keeps most of what 10% gives with fewer trainings.

Appendix B tests a second way to loop. In Strategy 2 each round continues training from the previous round's trained weights, and only the final pruned network is reset to $\theta_0$. Resetting every round (Strategy 1, the loop above) holds accuracy and learning speed to smaller networks on Lenet and on all three convolutional networks. A plausible reason, which the paper does not test: under Strategy 1 each new mask is chosen by training the previous ticket itself (the previous mask applied to $\theta_0$), so the magnitudes that pick the next mask come from the same starting values the next ticket will use.

Measuring learning speed

Condition $j' \le j$ in (2) needs a definition of how fast a network learns. The paper uses the iteration at which an early-stopping rule would halt, the iteration of minimum validation loss. On MNIST the 60,000 training images are split into 55,000 for training and 5,000 for validation, and the 10,000 test images are kept separate. Validation and test performance are evaluated every 100 iterations.

Training never actually stops early. Every run goes the full 50,000 iterations (20,000 to 30,000 for the convolutional networks), and the early-stopping iteration is read off the saved validation curve afterwards. A network whose validation loss bottoms out at iteration 6,000 instead of 10,000 counts as learning 1.67× faster, or 40% earlier. Accuracy is always reported at the early-stopping iteration, so a fast network is not credited with the accuracy of a longer run.

Appendix C checks this proxy against the obvious alternative: the order in which the pruned networks reach their early-stopping iteration is the same order in which they first cross a fixed test-accuracy threshold. For VGG-19 and Resnet-18 the criterion is dropped, because their learning rates step down at fixed iterations and the validation minimum lands wherever the schedule puts it. Section 4 reports their accuracy at fixed points in training instead.

What the tickets do

On Lenet, trained with Adam at learning rate 0.0012 and batch size 60, tickets learn faster as they shrink from 100% to 21% of the weights, where early stopping comes 38% earlier than for the dense network. Below 21% learning slows again, and at 3.6% the early-stopping iteration is back at the dense network's. Test accuracy at early stopping rises with pruning to a peak more than 0.3 percentage points above the dense network at 13.5%, then falls back to the dense level, again at 3.6%. Every curve is the average of five trials, with error bars at the minimum and maximum trial.

The convolutional networks show the same pattern with larger gains. Conv-2, Conv-4 and Conv-6 are scaled-down VGG-style stacks of two, four or six 3×3 convolutional layers (64, 128 and 256 channels, max-pooling after every pair) followed by fully connected layers of 256, 256 and 10 units, trained on CIFAR10 with Adam for 20K, 25K and 30K iterations at batch size 60. Their tickets reach minimum validation loss at best 3.5× sooner than dense at $P_m = 8.8\%$ (Conv-2), 3.5× at 9.2% (Conv-4) and 2.5× at 15.1% (Conv-6). Their best test accuracies are 3.4, 3.5 and 3.3 points above dense, at 4.6%, 11.1% and 26.4%. All three stay above the dense accuracy while more than 2% of the weights remain. Figure 3 puts every network on one axis; hover a mark for its number.

At early stopping, training accuracy rises along with test accuracy, which by itself would say only that tickets optimize faster. The paper also reports the end of training. At iteration 50,000 nearly every Lenet network, dense or heavily pruned, has 100% training accuracy, and the iteratively pruned tickets are still up to 0.35 points more accurate on the test set. Equal training accuracy and higher test accuracy means a smaller gap between the two: the tickets generalize better, and the convolutional networks show the same at their final iterations.

Test accuracy plotted against size goes up from the dense network, peaks somewhere between about 5% and 30%, and comes down as the network runs out of capacity. Section 5 borrows Rasmussen and Ghahramani's name for this shape, Occam's hill, with the dense network on the too-complex side and the smallest networks on the too-simple side. The paper leaves open why a subnetwork with a tenth of the weights generalizes better than the network it came from, and offers two candidates: the mask, which was chosen using the training data, may encode an inductive bias suited to the task, and the initialization may land in a region of the loss landscape the optimizer handles well.

Dropout also trains a random subnetwork at every step, by disabling each unit with probability 0.5. On Conv-2/4/6 it raises the dense accuracy by 2.1, 3.0 and 2.4 points, and iterative pruning on top of dropout adds up to 2.3, 4.6 and 4.7 more, so in these experiments the two gains stack. The speed gain shrinks under dropout: Appendix H.5, which measures speed on the validation set at its own learning rate, reports 1.58× for Conv-4 with dropout against 4.27× without. The 4.27× is a different measurement from the main text's 3.5×.

Separating the mask from the initial values

A pruned-and-reset network differs from the dense one in which weights it has and in what those weights start from. The paper separates the two with one control: keep the mask, redraw the surviving values from $\mathcal{D}_\theta$, train, and compare. Each ticket is redrawn three times, so every point on a control curve averages 15 runs.

On Lenet the redrawn networks learn slower with each round of pruning instead of faster. Their test accuracy falls below the dense network's once $P_m$ reaches 21.1%, the size where the reset ticket is at its fastest, while the reset ticket holds dense accuracy down to 2.9%. At $P_m = 21\%$ the ticket reaches minimum validation loss 2.51× sooner than its redrawn twin and is half a point more accurate. All networks have 100% training accuracy for every $P_m \ge 5\%$, so the accuracy gap is in generalization.

Figure 4 is the same experiment on a network small enough to plot. We trained a 2–50–50–3 ReLU network (2,750 weights) to classify points from three interleaved spiral arms, with 300 training points of which 20% carry a wrong label. A network this size can memorize wrong labels (the effect studied in Zhang et al., 2017), so the dense network overfits and early stopping has a minimum to find. The protocol is the paper's: Glorot Gaussian initialization, Adam, 4,000 iterations at batch size 50, validation every 100 iterations, 20% pruned per round (10% in the output layer), reset to $\theta_0$, and a control that redraws the values once per round under the same mask. Five seeds. The two maps show what each network learned at the chosen round; the plot below them is test accuracy at the minimum-validation-loss iteration for every round.

The dense network scores 94.5% (mean of five seeds). Through the first four rounds the two arms stay within a point of each other and both improve on dense, peaking near 96% between 65% and 34% of the weights. A sparser network has less capacity to fit the wrong labels whatever its starting values, and the redrawn Conv-2 and Conv-4 in the paper also improve at moderate pruning. Below 34% the arms separate: the redrawn control falls below dense at 27.2% of the weights, the reset ticket only at 17.7%. At 17.7% the ticket scores 93.6% against 86.6%, at 11.5% it scores 81.7% against 72.7%, and at 9.2% it scores 69.2% against 55.8%.

The run does not reproduce the speed result. Only the first-round ticket, at 80.4% of the weights, reaches its validation minimum sooner than the dense network (iteration 2,460 against 2,740); every smaller ticket reaches it later. Two redrawn controls also beat dense, at 80.4% (iteration 2,600) and 64.7% (iteration 2,620). Below about 20% most runs are still improving at iteration 4,000. By condition (2) this toy has no small winning tickets, only the gap between reset and redraw.

Below about 15% both arms also lose most of the first layer, which has only 100 weights (2 inputs × 50 units). Pruning 20% per layer per round, plus the one extra weight the code's cutoff rule removes each round, leaves it 9 weights after round 9 and a single weight after round 13. At round 13 each network reads only one input coordinate, and both maps split the plane into vertical bands. Global pruning, used on VGG-19 further down the page, avoids this kind of bottleneck.

On the paper's convolutional networks the picture is mixed in the same way. The redrawn controls of Conv-2, Conv-4 and Conv-6 also slow down with pruning and lose accuracy sooner than the tickets, but at moderate pruning the redrawn Conv-2 and Conv-4 hold or even improve their early-stopping accuracy, so a sparse network with fresh values beats the dense one for a while. Appendix E adds a third arm, a random mask of the same size with random values. On Lenet the pruned mask with redrawn values beats the random mask; on all the convolutional networks the two are indistinguishable. The paper's hypothesis for the Lenet case is that only some parts of an MNIST image carry information, so a mask found by training can encode where they are, while a convolutional filter is applied at every location.

Appendix F narrows down, on Lenet, what about the values matters:

• Adding Gaussian noise to the ticket's initial values with standard deviation $0.5\sigma$, where $\sigma$ is the layer's initialization standard deviation, barely changes the ticket's accuracy. At $3\sigma$ the ticket still beats the redrawn control.
• The surviving initial values have a bimodal distribution, with peaks on either side of zero, in the second hidden layer and the output layer. Redrawing from that distribution does little better than redrawing from the original Gaussian, so the ticket depends on the per-connection assignment of values, not only on their distribution.
• Pruning the smallest initial magnitudes at iteration 0, without any training, does worse than the redrawn control.
• During the dense training run, the weights that end up in the ticket move further than the weights that get pruned, and more often away from zero. The ticket's initial values are not close to their trained values.

VGG-19 and Resnet-18: global pruning and warmup

Section 4 moves to two CIFAR10 networks with the ingredients of a practical model: VGG-19 with 20.0 million parameters and the 20-layer Resnet-18 with 271,000, both trained with batch normalization, weight decay 0.0001, random flips and crops, SGD with momentum 0.9, and a learning rate that drops by 10× twice. VGG-19 trains for 160 epochs, 112,480 iterations at batch size 64, with drops after epochs 80 and 120 (iterations 56,240 and 84,360). Resnet-18 trains for 30,000 iterations at batch size 128 with drops at 20,000 and 25,000, a schedule shortened from He et al.'s 64,000 because each experiment trains the network 15 to 30 times in a row.

Pruning on these networks is global. Instead of removing the smallest 20% within each layer, the procedure pools every convolutional weight, finds one magnitude cutoff, and removes the smallest 20% of the pool. Three groups are never pruned: VGG-19's 5,120 output-layer weights, Resnet-18's 640 output-layer weights, and the 2,560 weights in Resnet-18's downsampling shortcuts. Layer sizes span three orders of magnitude: the first convolutional layer of VGG-19 has 1,728 weights and the last 2,359,296, so at $P_m = 1.5\%$ a layer-wise rule leaves the first layer 26 weights to carry all of the image into the network. Under one global cutoff a layer keeps as many weights as clear the shared threshold. Appendix I.1 measures the difference on VGG-19 at learning rate 0.1 with warmup: global pruning finds winning tickets while $P_m \ge 1.5\%$, layer-wise only while $P_m \ge 6.9\%$. In Figure 5, drag the round count to 19, where $P_m$ is 1.4%, and compare the first layer under the two rules.

The second change responds to a failure at the standard learning rate. At 0.1, iterative pruning finds no winning tickets in either network: the pruned-and-reset subnetworks do no better than their redrawn controls. At learning rate 0.01 the pattern from the small networks returns, partly. VGG-19 subnetworks stay within one point of the dense network while $P_m \ge 3.5\%$ but never match it, so they are not winning tickets. Resnet-18 tickets reach 89.5% for $41.7\% \ge P_m \ge 21.9\%$, short of the 90.5% the dense network reaches at 0.1. Appendix I.4 tries the rates in between and finds tickets at none above 0.01.

Linear learning-rate warmup recovers the tickets. The rate climbs from 0 to its base value over the first $k$ iterations and then follows the usual schedule. With $k = 10{,}000$ at rate 0.1, the dense VGG-19 gains about one point and iterative pruning finds winning tickets down to $P_m = 1.5\%$. With $k = 20{,}000$ at rate 0.03, Resnet-18 tickets reach 90.5% at $P_m = 27.1\%$, matching the dense network at 0.1, and remain winning tickets down to 11.8%. No warmup length the paper tried produces Resnet-18 tickets at rate 0.1 itself; warmup raised the usable rate from 0.01 to 0.03 and no further. Appendix I.5 sweeps $k$: accuracy improves quickly up to about 5,000 iterations and with diminishing returns after. In Figure 6, pick a network, a base rate and a warmup length; the readout quotes the paper's result for that combination.

The paper does not explain why warmup is needed and lists it as future work. The follow-up by Frankle, Dziugaite, Roy and Carbin (2020) proposes a mechanism. Take the network at some iteration $t$, train two copies from there with different random minibatch orders, and check whether the error stays low along the straight line between the two final weight vectors. When it does, training is stable to SGD noise from iteration $t$. They find that iterative pruning reaches full accuracy only when training is stable in this sense, which on MNIST holds at iteration 0 and on larger networks only a little way into training. Resetting to an early iteration instead of 0 uses that directly. The original paper's warmup and lower learning rate plausibly work by making the first iterations gentle enough that iteration 0 is already close to the stable point; this reading is ours, not the paper's.

Liu et al. (2019), the source of the VGG-19 training setup, report that a pruned VGG-19 redrawn from scratch matches the dense network down to 20% of the weights, and that with the optimal learning rate the winning-ticket initialization brings no improvement over random initialization. Frankle and Carbin agree where the two papers overlap: at rate 0.1 their redrawn networks also match dense down to 20%, and their tickets do no better there. They differ below 20%, where Liu et al. present no data and the lottery ticket experiment with warmup keeps finding tickets down to 1.5% while the redrawn networks lose accuracy. Section 5 states its reconciliation as a hypothesis: a heavily overparameterized network can be pruned, redrawn and retrained down to some sparsity, and past that sparsity only the original initialization keeps the accuracy.

What the lottery ticket hypothesis claims and what it conjectures

The hypothesis is an existence claim about dense, randomly initialized, feed-forward networks, and the evidence for it is the set of tickets the procedure found on six architectures and two datasets. The abstract's "less than 10-20% of the size" is the range every tested network reaches; Lenet goes to 3.6%, VGG-19 with warmup to 1.5%, and Resnet-18 with warmup only to 11.8%. A second claim is labeled a conjecture and not tested: that SGD seeks out and trains a well-initialized subset of the weights, and that dense networks are easier to train than sparse ones because a dense initialization contains more candidate subnetworks.

The results do not yet make training cheaper. Finding a ticket costs 15 to 30 dense-sized trainings, and the unstructured sparse networks it produces run no faster on a GPU than dense ones, because dense matrix kernels do not skip individual zero weights. Section 6 lists both limitations, along with the restriction to small vision datasets (MNIST and CIFAR10) and the open question of why warmup is needed.

To reproduce the Lenet result: the loop in the second code listing, with the per-round sizes of (3), 20% per round (10% for the output layer), reset to the saved $\theta_0$, 50,000 iterations of Adam at 0.0012 and batch size 60, five trials, and the early-stopping iteration read off the validation curve afterwards. The questions below cover what later work changed: resetting to an early iteration instead of iteration 0 extends the result to ImageNet, the signs of the initial values carry much of the effect on small networks, and a stronger form was proven for sufficiently wide random networks.