Mixed Precision Training

Train in 16-bit floats and match 32-bit accuracy: keep a 32-bit copy of the weights, scale the loss so small gradients survive, and add up products in 32 bits.

A 16-bit float takes half the memory of a 32-bit one and runs several times faster on GPUs built for it, but it keeps only 11 significant bits and cannot represent anything between 0 and about 6 × 10⁻⁸. The paper finds the three places where that loses information during training and adds a fix for each.

Baidu and NVIDIA, October 2017, written as the Volta V100 shipped with hardware for 16-bit matrix multiplies. The recipe is what PyTorch's automatic mixed precision still implements.

Every number in a neural network, whether a weight, an activation or a gradient, was by default stored as a 32-bit IEEE float (FP32). The paper trains the same networks with most of those numbers in 16-bit IEEE floats (FP16), on image classification, object detection, speech recognition, translation, language modeling and a GAN, and matches or beats the FP32 accuracy each time without changing a hyperparameter. It needs three changes, each aimed at one way FP16 loses information:

1. keep an FP32 master copy of the weights and apply updates to it, because many updates are too small to change an FP16 weight;
2. multiply the loss by a constant before backpropagation (loss scaling), because many gradients are too small for FP16 to represent at all;
3. add up the products inside each dot product in FP32, because an FP16 running sum stops absorbing small terms once it grows.

Below, each change is worked through with concrete FP16 numbers, after a short tour of the format itself.

Why train in 16 bits

The paper names three things that limit how fast any program runs: arithmetic throughput, memory bandwidth and latency. Halving the bytes per number addresses the first two. A tensor in FP16 is half the size, so it takes half as long to read from or write to GPU memory and half the space to keep there, and GPUs with dedicated half-precision units do 2 to 8 times as much FP16 math per second as FP32 math. On the V100 that dedicated unit is the Tensor Core, which multiplies FP16 matrices and adds the products into either FP16 or FP32.

Earlier reduced-precision work (binary weights, 2- to 6-bit quantized activations) lost accuracy on large ImageNet models, needed wider layers, or kept the gradients in FP32 so the backward pass cost as much as before. The paper sets a stricter target: every tensor in the forward and backward pass in FP16, no hyperparameter changes, and no accuracy loss against an FP32 baseline.

How FP16 stores a number

A floating-point number is scientific notation in base 2: a sign, an exponent that says which power of two the number sits near, and a mantissa that gives its leading digits. FP16 uses 1 sign bit, 5 exponent bits and 10 mantissa bits. For an ordinary (normal) number the value is

where $E$ is the 5-bit exponent field read as an integer, 15 is the exponent bias, and $M$ is the 10-bit mantissa field. The leading 1 is implied and not stored, so a normal FP16 number carries 11 significant bits, a little over three decimal digits. The exponent therefore runs from $-14$ to $15$, and the largest value is $(2 - 2^{-10}) \cdot 2^{15} = 65504$. FP32 has 8 exponent bits and 23 mantissa bits: 24 significant bits and a range up to about $3.4 \times 10^{38}$.

Take 0.1. It lies between $2^{-4}$ and $2^{-3}$, so $E = 11$; then $0.1 / 0.0625 = 1.6$, and $0.6 \times 1024 = 614.4$ rounds to $M = 614$. The stored bits are 0 01011 1001100110 and the stored value is 0.0999755859375, off by 0.024%. Rounding a normal number to FP16 always moves it by at most half a step, a relative error of at most $2^{-11} \approx 0.05\%$.

That step is the spacing between neighbouring FP16 numbers, and it depends on the exponent. Between 1 and 2 the spacing is $2^{-10} \approx 0.00098$; between 1024 and 2048 it is 1; between 2048 and 4096 it is 2. FP16 can count 2048 exactly, and 2049 is not representable.

Below $2^{-14} \approx 6.1 \times 10^{-5}$ the exponent field is 0 and the format switches to subnormal numbers: no implied leading 1, a fixed spacing of $2^{-24} \approx 5.96 \times 10^{-8}$, and fewer significant bits the smaller the number gets. The smallest positive FP16 value is $2^{-24}$ itself, with one significant bit. Under the IEEE default rounding (round to nearest, ties to even) anything with magnitude at or below $2^{-25}$ rounds to 0, and anything at or above 65520 rounds to infinity. The paper quotes the zero threshold as $2^{-24}$; values between the two round up to $2^{-24}$ rather than to zero.

Figure 1 rounds any number you choose to FP16 and shows the 16 bits. Start at 0.1, then drag down past $2^{-14}$ and watch the significant-bit count fall, one bit per halving, until the value becomes 0.

The normal range spans about 30 powers of two. Training uses only part of it: the paper's two gradient histograms put most values between $2^{-40}$ and $2^{-8}$, much of that mass in the subnormal band and below it, while the range above 1 goes nearly unused.

The FP32 master copy of the weights

An SGD step changes each weight by $-\eta g$, the learning rate $\eta$ times the gradient $g$. Done entirely in FP16, the step can fail in two ways.

First, the product $\eta g$ can itself be too small for FP16. With $\eta = 10^{-4}$ and a gradient of $2 \times 10^{-4}$, the update is $2 \times 10^{-8}$, below $2^{-25}$, and rounds to 0. With a gradient of $10^{-3}$ the update $10^{-7}$ survives as $2 \times 2^{-24} \approx 1.19 \times 10^{-7}$, 19% too large, because subnormals that small carry only one or two significant bits.

Second, an update that FP16 can represent can still vanish when it is added to the weight. Adding two floats lines up their binary points by shifting the smaller number right, then rounds the sum back to 11 significant bits. If the update is less than half the spacing at the weight, the rounded sum is the weight itself. At $w = 1$ the spacing is $2^{-10}$, so any update below $2^{-11} \approx 4.9 \times 10^{-4}$ disappears, which is a weight-to-update ratio of 2048. The paper states the rule as that ratio, and it is exact for a weight that is a power of two. In general the spacing is between $w/2048$ and $w/1024$, so the critical ratio lies between 2048 and 4096: at $w = 1.5$ an update 3000 times smaller than the weight still moves it by one step.

Here $\mathrm{fl}_{16}$ means rounding to FP16, and (2) holds for normal $w$. At exactly half the spacing the tie rounds to whichever neighbour has an even mantissa, so that case can go either way.

The fix keeps an FP32 copy of every weight for the whole run. Each iteration rounds the master copy to FP16 for the forward and backward pass, and the optimizer computes $\eta g$ and adds it to the master copy in FP32, where the spacing at 1 is $2^{-23}$. Updates too small to change the FP16 copy in one step accumulate in the master copy until they add up to a full FP16 step, and then the FP16 copy changes.

In Figure 2 one weight receives the same update on each of 4000 steps. At the default settings the weight is 1 and the update is $2^{-11.5} \approx 3.45 \times 10^{-4}$, a ratio of 2896. The FP32 master copy climbs to 2.381. The FP16 weight never moves. Now drag the weight down to about 0.62: the FP16 spacing there is $2^{-11}$, half of it is below the update, and every step rounds up to a full spacing of $4.9 \times 10^{-4}$. The FP16 weight rises 41% faster than it should, reaches 1.0 in under 800 steps, and stops there because the spacing doubles at 1.

Both failures need a small learning rate, a small gradient, or a large weight, and they hit unevenly: the weights whose gradients are consistently small stop training while the rest carry on. The paper shows the effect on a 215-million-parameter Mandarin speech model. With FP16 storage and an FP32 master copy, training and validation curves match FP32. Updating FP16 weights directly gives an 80% relative accuracy loss. The paper says not every network needs a master copy; it reports a run without one only for this model and uses the master copy in every other experiment.

The master copy costs 4 bytes per weight on top of the 2-byte FP16 copy, 50% more weight memory than FP32 training. The memory section below shows why the total still goes down.

How loss scaling works

The master copy protects updates. It does nothing for gradients that are too small to store in FP16 during backpropagation, before the optimizer step runs. The paper's Figure 3 histograms every activation gradient (the gradient of the loss with respect to a layer's output) over the training of the SSD object detector: 67% of the values are exactly zero, most of the rest lie between $2^{-40}$ and $2^{-20}$, and almost nothing is above $2^{-10}$. In FP16 everything below $2^{-24}$ is lost, while the range from $2^{-10}$ up to 65504 sits empty. Trained this way in FP16, SSD diverges.

The gradients need to move up the exponent range by a constant factor, and backpropagation is linear in the loss. If the loss is multiplied by $S$ before the backward pass, the chain rule multiplies every gradient in the network by the same $S$:

This costs one multiply on a scalar. Every layer's backward computation is linear in the gradient arriving from above, so the factor passes through unchanged; no extra operations run during backpropagation. A power of two is the natural choice, because multiplying by $2^k$ adds $k$ to the exponent and changes no mantissa bits, so scaling and the later division are exact for every value that stays in range. After the backward pass the weight gradients are converted to FP32 and divided by $S$, which returns them to their true size, and only then does anything else touch them. The paper is specific about the order: unscale before gradient clipping, weight decay or any other gradient computation, so that a clipping threshold of 1.0 still means 1.0.

A worked value: an activation gradient of $2^{-26} \approx 1.5 \times 10^{-8}$ rounds to 0 in FP16. With $S = 8$ it is computed as $2^{-23} \approx 1.19 \times 10^{-7}$, which FP16 stores exactly. For SSD a scale of 8 is enough to match FP32 (77.1% mAP against 76.9%). The paper concludes that for this network, gradient values below $2^{-27}$ do not matter for training and values in $[2^{-27}, 2^{-24})$ do.

The upper limit on $S$ is overflow. If $S$ times the largest gradient magnitude reaches 65520, that value becomes infinity, and the infinities and NaNs (not-a-number values, from operations such as $\infty - \infty$) spread into the weight gradients and would ruin the weights if applied. With gradient statistics in hand, choose $S$ so that $S \cdot \max|g| < 65504$; for a largest gradient of 0.5, that allows $2^{16}$ but not $2^{17}$. Below that limit the paper finds no downside to a large $S$. Overflow is cheap to detect, since every weight gradient is inspected anyway when it is unscaled, and the paper's suggestion is to skip that iteration's update and continue with the next batch.

Figure 3 applies a loss scale to an illustrative gradient histogram with the shape of the paper's SSD plot. At $S = 1$, 63% of the values round to zero and 1.4% are normal FP16 numbers. Drag $k$ up and the whole histogram slides right; the share lost to zero falls to 1.5% at $S = 2^{15}$, and overflow appears at $k = 25$.

In this histogram every scale from $2^{12}$ to $2^{24}$ keeps the share lost to zero under 5% without overflowing, a factor of 4096 between the smallest and largest workable scale. The paper trained networks with constant scales from 8 to 32,768, and many needed no scaling: the six ImageNet classifiers and the GAN trained without it, while SSD, the translation model and the bigLSTM language model (scale 128) needed it.

Summing products in FP32

The paper sorts the arithmetic in a network into three kinds: dot products (inside every matrix multiply and convolution), reductions (sums over the elements of a vector), and point-wise operations (activations, element-wise products), and treats each one differently.

A dot product $\sum_i a_i b_i$ over a hidden dimension of 1024 or 4096 adds thousands of terms into one running sum. If that sum is held in FP16, its spacing grows as it grows, and each new term is rounded to that spacing. Once the sum reaches 2048 the spacing is 2, so a term below 1 is dropped entirely: adding 1.0 to an FP16 2048 gives 2048. The paper found that some networks miss FP32 accuracy unless FP16 products are accumulated in FP32 and the finished result is converted to FP16 before being written to memory. Volta's Tensor Cores do this in hardware, multiplying FP16 inputs and accumulating into FP32. The product of two FP16 numbers is exact in FP32 (11 + 11 significant bits fit in 24), so the only rounding left is in the FP32 sum and in the final conversion.

In Figure 4 the inputs are FP16 numbers in $[0, 1)$, and their products are summed in three ways. At 4096 terms the exact sum is 1000.64, the FP32 accumulator gives 1000.635, and the FP16 accumulator gives 937.5, 6.3% low. At 32,768 terms the FP16 sum is stuck at 2048 while the true sum is 8148. Switch to signed inputs and the error shrinks, because the running sum stays near zero where FP16 spacing is fine; switch to all ones and the FP16 sum stops at exactly 2048.

Large reductions follow the same logic. Batch normalization sums each channel over the whole batch to compute a mean and variance (see the batch normalization explainer), and softmax sums exponentials over a whole vocabulary or sequence. The paper's implementations read and write FP16 tensors for these layers and do the arithmetic in FP32. That costs no time, because these layers spend their time moving data, not computing: their speed is set by memory bandwidth, which FP16 storage still halves. Point-wise operations are bandwidth-bound for the same reason, and either precision works for them.

One training iteration

Put the three techniques together and one iteration for one layer looks like the paper's Figure 1, with the loss scale added. The FP16 copy of the weights exists only for the duration of the step; the FP32 master copy is the persistent state.

# One mixed-precision iteration (the paper's Figure 1, plus loss scaling).
# master: FP32 weights, kept for the whole run. S: the loss scale.
w16  = master.to(fp16)            # FP16 copy, used by forward and backward
y    = forward(x16, w16)          # FP16 tensors; matmuls sum products in FP32
loss = loss_fn(y, target)         # softmax and other reductions computed in FP32
g16  = backward(loss * S)         # every gradient is S times its true value
g32  = g16.to(fp32) / S           # unscale in FP32, before clipping or decay
if not all_finite(g32):           # inf or NaN: S * |g| passed 65504 somewhere
    pass                          # skip this update, keep master as it was
else:
    g32 = clip(g32, max_norm)     # thresholds keep their FP32 meaning
    master -= lr * g32            # FP32 multiply and add: small updates survive

Follow one weight through it, with $S = 1024$ and a learning rate of 0.1. The master value is 0.02 in FP32. The forward pass uses its FP16 copy, 0.0200043, whose FP16 neighbours are $2^{-16} \approx 1.5 \times 10^{-5}$ away. Suppose the weight's true gradient is $2 \times 10^{-8}$. Stored directly in FP16 it would be 0. Scaled by 1024 it is $2.048 \times 10^{-5}$, stored as the subnormal $344 \times 2^{-24} \approx 2.0504 \times 10^{-5}$; converted to FP32 and divided by 1024 it becomes $2.0023 \times 10^{-8}$, 0.12% from the true value. The update $\eta g \approx 2.0 \times 10^{-9}$ is thousands of times smaller than half the FP16 spacing at the weight, so adding it to an FP16 weight would leave the weight unchanged. The FP32 spacing at 0.02 is $2^{-29} \approx 1.86 \times 10^{-9}$, and the master copy moves by one spacing. After 1802 such steps the master copy reaches 0.0199966, the midpoint between 0.0200043 and its lower FP16 neighbour 0.0199890, and the FP16 copy rounds down (ties to even) to 0.0199890.

FP32 has its own version of (2): here the update is rounded to $1.86 \times 10^{-9}$, 7% short. Its 23 mantissa bits put that threshold 13 powers of two below FP16's.

Where the memory goes

The abstract says the method nearly halves memory requirements. Count bytes per parameter for SGD with momentum. FP32 training stores the weight (4), its gradient (4) and the momentum (4): 12 bytes. Mixed precision stores the FP16 weight (2), the FP32 master weight (4), the FP16 gradient (2) and the momentum, still FP32 (4): also 12 bytes. The master copy adds 2 bytes and FP16 gradients remove 2, so the model states cost the same.

The saving comes from activations. Backpropagation needs each layer's outputs from the forward pass, so they are kept in memory until the backward pass uses them, and for convolutional and recurrent networks at realistic batch sizes they take far more memory than the weights. Stored in FP16, they take half as much. The paper's "roughly halved" holds when activations dominate the total.

In Figure 5, $a$ is the number of activation values stored per weight. At $a = 8$ with momentum, FP32 needs 44 bytes per parameter and mixed precision 28, a ratio of 0.64. At $a = 40$ the ratio is 0.53. At $a = 0$ it is 1.00 for every optimizer.

With Adam, which keeps two FP32 state vectors per weight (see the Adam explainer), both columns hold 16 bytes of model state. For large language models, where the weights and optimizer state are often the larger part, mixed precision mostly buys speed and smaller activations, and the 16 bytes per parameter are what ZeRO later partitions across GPUs. Activations can be reduced further by recomputing them instead of storing them, the subject of the gradient checkpointing explainer.

What the experiments show

The baselines ran in FP32 on Maxwell or Pascal GPUs. The mixed-precision runs (labelled MP) used FP16 storage, an FP32 master copy, Tensor Core matrix multiplies with FP32 accumulation on a V100, and loss scaling where noted. Hyperparameters were identical. Six ImageNet classifiers, trained without loss scaling, match their baselines in top-1 accuracy:

The largest difference is 0.28 points (Inception v3), in favour of mixed precision. (VGG-D is the network of the VGG explainer; some baselines differ from published numbers because the paper used single-crop testing and simpler augmentation.)

Loss scaling first becomes necessary in the detectors. Both use a VGG-16 backbone and are scored by mean average precision (mAP) on Pascal VOC 2007:

Faster R-CNN loses half a point without loss scaling and recovers it with scaling; SSD does not train at all without it.

The other experiments, in the order the paper gives them:

• Speech. DeepSpeech 2 models with recurrent GRU layers (the unit from the RNN encoder–decoder explainer), 115 million parameters for English and 215 million for Mandarin. Character error rates fall from 2.20 to 1.99 (English, WSJ '92) and from 15.82 to 15.01 (Mandarin), 9.5% and 5.1% lower. These runs were emulated on Maxwell GPUs with FP16 storage and FP32 arithmetic, which the paper checked against true V100 mixed precision on other networks.
• Translation. English-to-French LSTM models with attention, 3 and 5 layers of 1024 cells, from the TensorFlow sequence-to-sequence tutorial (in the line of the Seq2Seq models). Training perplexity (the exponential of the average per-word loss; lower is better) differs noticeably across three FP32 runs with identical settings; mixed precision with loss scaling matches them, and without loss scaling it is slightly worse.
• Language modeling. bigLSTM on the 1-billion-word benchmark: two layers of 8192 LSTM cells, a 793K-word vocabulary, trained with Adagrad. Without loss scaling the FP16 perplexity curve diverges from FP32 after 300K iterations; a scale of 128 matches it.
• GAN. A DCGAN generating 128 × 128 faces, trained with Adam, needs no loss scaling. With no agreed quality metric for GANs, the paper shows uncurated samples from both runs and calls them comparable.

On speed, the paper reports only per-operation numbers: DeepBench operations on Volta run 2 to 6 times faster than in FP32 when they are limited by memory or arithmetic bandwidth, and less when they are limited by latency. End-to-end training speedups are left as future work, because the libraries and frameworks of 2017 were early versions.

Dynamic loss scaling and bfloat16

The paper's last paragraph proposes automating the scale: raise or lower it by inspecting the weight gradients for overflow, and skip the update when one occurs. Frameworks adopted that scheme. PyTorch's GradScaler starts high and lets overflows pull the scale down:

# Dynamic loss scaling, with the defaults of PyTorch's GradScaler.
S, clean = 2.0**16, 0
for x16, target in data:
    g32 = unscaled_grads(x16, target, S)   # steps 1-5 of the loop above
    if not all_finite(g32):
        S, clean = S * 0.5, 0              # overflow: skip the step, halve S
        continue
    update(master, g32)
    clean += 1
    if clean == 2000:                      # 2000 clean steps in a row
        S, clean = S * 2.0, 0              # try a larger scale

A scale that is too large costs one skipped step and a halving, so starting at 65536 finds a working scale within a few iterations, and the periodic doubling keeps it near the largest value that does not overflow as gradient magnitudes drift during training. In PyTorch the other two techniques come from torch.autocast: the parameters stay in FP32 and serve as the master copy, matrix multiplies and convolutions run in FP16 (on Tensor Cores, which accumulate in FP32), and operations such as softmax, layer norm and sums run in FP32.

A second change removed loss scaling for many users. bfloat16 is a 16-bit format with 1 sign bit, 8 exponent bits and 7 mantissa bits: FP32 with the bottom 16 mantissa bits cut off. It has FP32's range, so gradients that underflow FP16 fit easily, but only 8 significant bits, so the spacing at 1 is $2^{-7}$ and the master copy and FP32 accumulation matter even more than in FP16. On hardware that supports it, bfloat16 with an FP32 master copy and no loss scaling is a common default for training large transformers; the QLoRA explainer, for instance, computes in bfloat16.