VerifiedarXiv:2208.0733920 min
Inference · Quantization

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Cut a transformer's GPU memory in half by storing almost all its weights in 8 bits and using 16-bit multiplication for the few feature dimensions that hold huge values.

Standard 8-bit quantization uses a single scale for a whole tensor, so one large value makes every other value less precise. LLM.int8() gives each row of hidden states and each column of weights its own scale, and sends the few feature dimensions that hold very large values through a separate 16-bit multiplication.

Explaining the paperLLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleDettmers, Lewis, Belkada, Zettlemoyer · NeurIPS 2022 · arXiv:2208.07339 ↗

Six feature dimensions, about 0.1% of the total, hold values up to 20 times larger than the others, and they were the reason 8-bit inference failed on models above a few billion parameters.

Storing the weights of a 175-billion-parameter language model, the size of GPT-3, at 16 bits each takes 350 GB of GPU memory. Eight A100 cards with 40 GB each hold 320 GB, so the weights alone do not fit. Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer posted LLM.int8() in August 2022 with a way to hold almost all of those weights in 8-bit integers, which halves the memory, and to apply it to a finished checkpoint at load time with no retraining.1

Eight-bit inference existed already, but the paper says it had been studied only on models under 350 million parameters. On larger models it damages predictions: at 13 billion parameters the standard method scores a perplexity of 19.08 against 12.45 at full precision. Perplexity is the usual quality score for a language model, the number of equally likely next tokens the model is effectively choosing between, so lower is better. The paper traces the damage to a small set of feature dimensions holding values far larger than the rest, and handles those separately.

The method has two parts. This page works up to them in order:

  1. how a number becomes an 8-bit integer, and why one large value costs every other value precision;
  2. how a matrix multiplication runs on integers, and how a separate scale per row and per column reduces rounding error (vector-wise quantization);
  3. the outlier features that appear across every layer of large transformers;
  4. how those few columns are multiplied in 16-bit while everything else stays in 8-bit (mixed-precision decomposition);
  5. the measured results, the speed, and how to use it.

What the weights cost

Two kinds of tensors matter here. Weights are the trained parameters, stored in GPU memory for the whole run. Hidden states are the intermediate vectors a layer computes as a batch of tokens passes through; they exist briefly and are rebuilt on every forward pass. The memory saving in this paper comes entirely from the weights. The accuracy problem it has to solve comes mostly from the hidden states.

For transformers at and above 6.7B parameters, the feed-forward layers and the linear projections inside attention hold 95% of the parameters and account for 65-85% of the computation; embeddings hold most of the rest. The feed-forward and projection layers are the ones LLM.int8() converts. The attention computation itself (the dot products and the softmax of the Transformer) has no parameters and stays in 16-bit.

A 16-bit float (FP16) spends 1 bit on the sign, 5 on the exponent and 10 on the fraction. It covers magnitudes up to 65,504, with spacing that shrinks as numbers get closer to zero, so a value near 1 and a value near 1000 both keep about three decimal digits of relative precision. An int8 holds whole numbers, 255 of them in practice (−127 to 127), spaced evenly, with nothing between them. The paper uses int8 because it is the only 8-bit type GPUs support in hardware. The product of two int8 values fits in 16 bits, since 127² is 16,129, and a sum of products accumulates in an int32 with room for about 133,000 maximum-size terms. The largest hidden dimension in the paper's tables is 49,152, so the integer multiplication cannot overflow.

How 8-bit quantization works

Converting real numbers to int8 means picking a multiplier, scaling, and rounding. The simplest choice, absmax quantization, scales a tensor so that its largest absolute value lands on 127:

Xi8=⌊127∥Xf16∥∞ Xf16⌉\mathbf{X}_{i8}=\Bigl\lfloor \frac{127}{\lVert \mathbf{X}_{f16}\rVert_{\infty}}\,\mathbf{X}_{f16}\Bigr\rceil

The tensor Xf16\mathbf{X}_{f16} is the FP16 input, ∥⋅∥∞\lVert\cdot\rVert_\infty is the largest absolute value in the tensor, and ⌊⋅⌉\lfloor\cdot\rceil rounds to the nearest integer. The distance between adjacent codes in the original units is ∥X∥∞/127\lVert\mathbf{X}\rVert_\infty/127, and every value lands within half of that distance of a code.

Take a row of a hidden state whose values run from −3.5 to 3.5, which the paper reports as the usual range for ordinary features. The step is 3.5/127, or 0.0276, so 3.5 becomes 127, 0.21 becomes 8, and the stored integers keep the row's shape closely. Now put one value of −44 in the same row. The largest absolute value is 44, the step becomes 44/127, or 0.346, 12.6 times wider, and everything below 0.173 in magnitude rounds to zero. The 0.21 that used to be code 8 is now code 1.

A ruler with 255 marks, centered on zero with 127 on each side, has to reach the longest thing it measures in either direction. Measure one object 44 cm long and the marks end up 3.5 mm apart, so anything shorter than about 1.7 mm reads as zero, for every other object measured with the same ruler.

Figure 1
43.9
step 0.346distinct codes 12/23rounded to 0 1RMS error 0.118
Drag the outlier magnitude from none up to 63, the largest magnitude reported in the paper's outlier table. The top axis is the full range the codes must cover; the lower axes magnify the ordinary range. Amber lines mark values that round to zero. Switch to zeropoint quantization (defined just below) with a large outlier and the step roughly halves.

Absmax centers its range on zero. If the data is all positive, codes −127 to −1 go unused, and with a one-sided outlier the same waste happens at the scale of the outlier. Zeropoint quantization maps the observed range onto all 255 codes with a scale and an offset:

ndx=2⋅127max⁡ijXij−min⁡ijXijnd_x=\frac{2\cdot 127}{\max_{ij}\mathbf{X}_{ij}-\min_{ij}\mathbf{X}_{ij}}
(1)
zpx=⌊ndxmin⁡ijXij⌉+127zp_x=\bigl\lfloor nd_x\min_{ij}\mathbf{X}_{ij}\bigr\rceil+127
(2)
Xi8=⌊ndx X⌉−zpx\mathbf{X}_{i8}=\bigl\lfloor nd_x\,\mathbf{X}\bigr\rceil-zp_x
(3)

The stored value Xi8\mathbf{X}_{i8} sits in [−127, 127], and adding zpxzp_x back gives ⌊ndxX⌉\lfloor nd_x\mathbf{X}\rceil, which is what gets multiplied. For values in [0, 10], the scale ndxnd_x is 25.4, which makes the step half the absmax step of 10/127, and zpxzp_x is 127, so a value of 5 is stored as 0. Equations (2) and (3) are corrected versions of the paper's (the Provenance panel lists what changed). To multiply two zeropoint numbers, add each zeropoint back before the product:

Ci32=(Ai8+zpa)(Bi8+zpb)C_{i32}=(A_{i8}+zp_{a})(B_{i8}+zp_{b})
(4)

On a CPU with a dedicated instruction this is one operation. GPUs and TPUs have no such instruction, so the product has to be unrolled:

Ci32=Ai8Bi8+Ai8 zpb+Bi8 zpa+zpa zpbC_{i32}=A_{i8}B_{i8}+A_{i8}\,zp_{b}+B_{i8}\,zp_{a}+zp_{a}\,zp_{b}Ci32=Ai8Bi8+Ai8 zpb+Bi8 zpa+zpa zpb\begin{aligned}C_{i32}&=A_{i8}B_{i8}+A_{i8}\,zp_{b}\\&\quad+B_{i8}\,zp_{a}+zp_{a}\,zp_{b}\end{aligned}
(5)

The first term runs in int8 and the other three in int16 or int32. The paper notes this makes zeropoint slow on GPUs, which is why absmax is the common choice in practice despite its weaker precision on asymmetric data.

Multiplying matrices in integers

Take hidden states X\mathbf{X} of shape s×hs \times h (ss tokens in the batch, each with hh features) and weights W\mathbf{W} of shape h×oh \times o (oo output features). With one scale constant per tensor, the 8-bit multiplication with FP16 inputs and FP16 outputs is:

Xf16Wf16≈1cxcw Ci32=1cxcw Q(Xf16) Q(Wf16)\mathbf{X}_{f16}\mathbf{W}_{f16}\approx\frac{1}{c_x c_w}\,\mathbf{C}_{i32}=\frac{1}{c_x c_w}\,Q(\mathbf{X}_{f16})\,Q(\mathbf{W}_{f16})Xf16Wf16≈1cxcw Ci32=1cxcw Q(Xf16) Q(Wf16)\begin{aligned}\mathbf{X}_{f16}\mathbf{W}_{f16}&\approx\frac{1}{c_x c_w}\,\mathbf{C}_{i32}\\&=\frac{1}{c_x c_w}\,Q(\mathbf{X}_{f16})\,Q(\mathbf{W}_{f16})\end{aligned}
(6)

QQ is absmax or zeropoint quantization, and cx,cwc_x, c_w are their scale constants (the paper writes them as sxs_x or ndxnd_x). The integers multiply, accumulate in int32, and the int32 sum gets multiplied by 1/(cxcw)1/(c_x c_w) to return to FP16 units. The weights are quantized once when the model loads and stay in int8. The hidden states are quantized on every forward pass, since each batch produces different ones.

A worked inner product, with x\mathbf{x} one row of the hidden state and w\mathbf{w} one column of the weights:

x=(0.21, −1.30, 0.47, 2.80)\mathbf{x}=(0.21,\,-1.30,\,0.47,\,2.80)
w=(1.10, −0.40, 0.90, 0.05)\mathbf{w}=(1.10,\,-0.40,\,0.90,\,0.05)

The exact product is 1.314 (the four terms are 0.231, 0.520, 0.423 and 0.140). The scales are 127/2.80 for the row and 127/1.10 for the column, so the integers are (10, −59, 21, 127) and (127, −46, 104, 6). Their inner product is 6930. Dividing by the product of the two scales, 16129/3.08, gives 1.3234, which is 0.7% above the exact value.

Vector-wise quantization

Matrix multiplication is a set of independent inner products, one per pair of a row of X\mathbf{X} and a column of W\mathbf{W}. Each inner product can use its own scale constants, because the constants factor out of the sum over the feature index hh:

Cij=∑hxihwhj=1ci cj∑h(cixih)(cjwhj)C_{ij}=\sum_h x_{ih}w_{hj}=\frac{1}{c_i\,c_j}\sum_h (c_i x_{ih})(c_j w_{hj})Cij=∑hxihwhj=1ci cj∑h(cixih)(cjwhj)\begin{aligned}C_{ij}&=\sum_h x_{ih}w_{hj}\\&=\frac{1}{c_i\,c_j}\sum_h (c_i x_{ih})(c_j w_{hj})\end{aligned}

So give each of the ss rows of X\mathbf{X} a constant, collected in a vector cx\mathbf{c}_x, and each of the oo columns of W\mathbf{W} a constant, collected in cw\mathbf{c}_w. Denormalizing is element-wise division by their outer product:

Cf16≈1cx⊗cw Q(Xf16) Q(Wf16)\mathbf{C}_{f16}\approx\frac{1}{\mathbf{c}_{x}\otimes\mathbf{c}_{w}}\,Q(\mathbf{X}_{f16})\,Q(\mathbf{W}_{f16})
(7)

A scale per feature hh would seem to be the natural way to deal with a feature that runs large, but it stays inside the sum, so each term would need its own division, and that is the operation integer matrix hardware does not provide. Only row and column constants come out of the sum. The paper calls scaling by rows of X\mathbf{X} alone row-wise quantization (from FBGEMM, Facebook's low-precision matrix library), with one scale for all of W\mathbf{W}, and scaling both rows of X\mathbf{X} and columns of W\mathbf{W} vector-wise quantization; Section 4.3 attributes vector-wise's advantage to the extra precision on the weights. As code:

# X: hidden states, shape [s, h], fp16, new on every forward pass
# W: weights, shape [h, o], fp16, quantized once when the model loads
c_x = 127 / abs(X).max(dim=1).values        # one scale per row of X,  shape [s]
c_w = 127 / abs(W).max(dim=0).values        # one scale per column of W, shape [o]
X_i8 = round(X * c_x[:, None]).int8()       # every entry now in [-127, 127]
W_i8 = round(W * c_w[None, :]).int8()
C_i32 = X_i8.int32() @ W_i8.int32()         # int8 products, int32 accumulator
C_f16 = C_i32 / (c_x[:, None] * c_w[None, :])  # denormalize by the outer product

Figure 2 runs this on a 10×16 hidden state and a 16×6 weight matrix. Finer scaling helps when the planted data is ordinary. Then toggle on a large column, and compare the size of the three bars.

Figure 2
An actual int8 multiplication: X and W are quantized at the chosen granularity, multiplied in integers, denormalized and compared with the exact product. Bars show RMS output error on a log scale. With no outlier, vector-wise error is about half of tensor-wise. With the outlier column on, all three grow by more than ten times, because that column sets the scale of every row.

In the paper's Table 1 (perplexity by model size, plotted in Figure 5), vector-wise quantization alone has lower perplexity than row-wise at 1.3B and 2.7B, ties at 6.7B and 13B, and does worse at 125M, and it cannot help with a single large column. Row-wise and vector-wise both scale by rows of X\mathbf{X}, so when the same feature is large in most rows, each row's maximum is that feature and the row scale is set by it.

Outlier feature dimensions

In the paper's terms, a feature is one of the hh columns of a hidden state. The authors tracked features across all layers of ten pretrained models from three software frameworks, and called a feature an outlier if it met three conditions at once: a magnitude of at least 6.0, present in at least 25% of the layers, and present in at least 6% of the sequence positions.

The ordinary feature dimensions usually stay within [−3.5, 3.5]. A 6.7B model processing a sequence of 2,048 tokens produced about 150,000 outlier values across the whole transformer, and all of them sat in 6 feature dimensions, about 0.1% of the total (the 6.7B row of the paper's Table 5 lists 4,096 dimensions, which makes 6 columns 0.15%). Across the models tested the count of outlier dimensions never exceeded 7, and on the large models almost all of them were one-sided, either always positive or always negative.

To measure the effect of these dimensions the authors set them to zero before each attention projection and compared the largest value in the resulting attention distribution: the mean of that top-1 probability fell from about 40% to about 20%. Propagating the zeroed states through the full model raised perplexity by 600-1000%. Zeroing 7 random non-outlier dimensions instead lowered the top-1 probability by only 0.02-0.3% and raised perplexity by about 0.1%. The outliers are required for the model to work, so they cannot be clipped away to protect the int8 range.

Figure 3
6.7B
FSEQ 6.7B, perplexity 13.36 outlier dims (6 one-sided)quartiles -44, -40, -35top-1 softmax 35% with outliers, 13% without
Each point is one model from Appendix C, Table 4 (hover or use the slider to read its row). Switch the x-axis between parameter count and C4 perplexity, and the measured quantity above. By parameters, layers affected jump from 62% to 100% between 6.0B and 6.7B; by perplexity the same points change more gradually.

Measured by parameter count, the affected share of layers jumps between 6.0B (GPT-J, 62% of layers) and 6.7B (a model trained with Meta's fairseq, 100%), and sequence positions go from 28% to 75%. Those two points come from different model families and training code, so part of the jump may reflect the comparison across stacks. The paper argues that perplexity is the better predictor: plotted against perplexity, the number of outlier dimensions never decreases (1, 2, 2, 2, 2, 4, 5, 6, 6, 7) and the emergence looks gradual, while against parameter count it is non-monotonic, dropping to 2 at GPT-2 1.5B after 4 at 1.3B. The median magnitude of the most common outlier rises from about 16 at 2.7B to about 40 at 6.7B and 58 at 13B.

For quantization, a value of −44 in a row of ordinary features forces a step of 0.346 where 0.0276 would do. The paper describes the result as most quantization bins being empty and small values being quantized to zero. Since nearly every row contains the same large columns, row-wise and vector-wise scales cannot avoid it. Zeropoint quantization handles the one-sided shape better, and in Table 1 it holds out longer, but at 13B the step is large enough that it fails too (13.94 against 12.45).

How LLM.int8() works: mixed-precision decomposition

The large values sit in a handful of columns, so the method pulls those columns out of the int8 multiplication, multiplies them separately in 16-bit, and adds the two results. For hidden states X\mathbf{X} and weights W\mathbf{W}, let OO be the set of feature dimensions that contain at least one value with magnitude above the threshold α\alpha. Then:

Cf16≈∑h∈OXf16hWf16h+Sf16⋅∑h∉OXi8hWi8h\mathbf{C}_{f16}\approx\sum_{h\in O}\mathbf{X}_{f16}^{h}\mathbf{W}_{f16}^{h}+\mathbf{S}_{f16}\cdot\sum_{h\notin O}\mathbf{X}_{i8}^{h}\mathbf{W}_{i8}^{h}Cf16≈∑h∈OXf16hWf16h+Sf16⋅∑h∉OXi8hWi8h\begin{aligned}\mathbf{C}_{f16}&\approx\sum_{h\in O}\mathbf{X}_{f16}^{h}\mathbf{W}_{f16}^{h}\\&\quad+\mathbf{S}_{f16}\cdot\sum_{h\notin O}\mathbf{X}_{i8}^{h}\mathbf{W}_{i8}^{h}\end{aligned}
(8)

Here Xh\mathbf{X}^{h} is column hh of X\mathbf{X} and Wh\mathbf{W}^{h} is row hh of W\mathbf{W}, so each term is one feature's contribution to the output. The sum over OO is the 16-bit branch; the sum over the rest is the int8 branch, with Sf16\mathbf{S}_{f16} the denormalization from (7). The paper sets α\alpha to 6.0 and reports that this is enough to bring perplexity degradation close to zero. The value sits above the ordinary range of [−3.5, 3.5] and below the outliers. Two things are added to the vector-wise int8 multiplication of Eq. (7): the outlier columns stop contributing to the integer sum, and the row maxima are computed without them.

The released code does both, comparing the threshold against the raw FP16 values. For each hidden-state tensor at each layer it collects the columns with any entry of magnitude at least α\alpha, so OO can differ between batches and between layers, and the row maxima ignore those entries (in the kernels, they are set to zero before the maximum is taken). So once the outliers are removed the row's step goes back to the ordinary 0.0276. Continuing the worked example, add a feature with value −44.0 and weight 0.12 to the vectors above. That feature adds −44 × 0.12, or −5.28, so the exact product becomes −3.966.

With plain absmax, the row maximum is 44 and the five rounded values are (1, −4, 1, 8, −127). The ordinary four contribute 463 to the integer sum, which denormalizes to 1.389 (true 1.314, off by 5.7%). With decomposition the ordinary four are computed as in the original worked example, 1.3234 (off by 0.7%), and the outlier term −5.28 is added exactly, for a total of −3.957.

# One 8-bit linear layer: output = X @ W, with outlier columns in 16-bit.
# 1. Find outlier columns in this batch's hidden states (alpha = 6.0).
O = where((abs(X) >= alpha).any(dim=0))     # at most 7 in the paper's models

# 2. Scale the rest. Row maxima ignore any entry >= alpha.
X_rest = X.clone()
X_rest[abs(X_rest) >= alpha] = 0
c_x = 127 / abs(X_rest).max(dim=1).values   # [s]
X_i8 = round(X_rest * c_x[:, None]).int8()  # [s, h]
X_i8[:, O] = 0                              # outlier columns leave the int8 branch

# 3. Integer multiply. W_i8 and c_w were computed once, at load time.
C_i32 = X_i8.int32() @ W_i8.int32()         # [s, o]
output = C_i32 / (c_x[:, None] * c_w[None, :])

# 4. 16-bit branch, added to the 8-bit result.
W_O = W_i8[O, :] / c_w[None, :]             # released code: read back from int8
output += X[:, O] @ W_O                     # shapes [s, |O|] @ [|O|, o]

Moving α in Figure 4 changes which columns leave the int8 branch. Below 3.4, ordinary columns start to count as outliers, and at α = 1 nearly everything goes to the 16-bit branch, which is accurate and saves no memory. Between about 4 and the smallest outlier the split does not change, so the error curve is flat there. Above the largest outlier nothing is removed and the error returns to the plain vector-wise level.

Figure 4
6.00
outlier columns 3/48RMS error 0.0197weight memory 1.06 B/param (fp16 is 2.00)
The 14-by-48 grid is X; amber columns are in the 16-bit branch at the current threshold. The curve is RMS output error against alpha, and the amber area the share of columns in 16-bit. The grid is 48 wide so the columns are visible; at 4,096 features, 6 columns would be 0.15%. The toggle reads the outlier weights back from int8, as the released code does.

The paper puts the cost of the 16-bit branch at about 0.1% of memory, since at most 7 dimensions were found up to 13B parameters. The released code path used for 8-bit loading (has_fp16_weights=False) does not store those weights at 16 bits. It reconstructs the outlier columns of W\mathbf{W} from the int8 weights, multiplies them by the exact hidden-state values in FP16, and discards them after the call. The 16-bit branch therefore adds no stored memory, and its weight side is rounded: a value of −44 times a weight error of up to ∥w∥∞/254\lVert\mathbf{w}\rVert_\infty/254 contributes an output error that the 16-bit copy does not have. Toggling Figure 4 shows the effect. The paper does not say which code path produced its accuracy tables.

Perplexity, accuracy and memory

Table 1 of the paper measures perplexity on validation data from C4, a web-text corpus, for five pretrained models from 125M to 13B parameters, against a 32-bit float baseline. Figure 5 plots each method as its ratio to that baseline, so a line on 1.0 means no degradation.

Figure 5
13B
32-bit float 12.45absmax 19.08 (+53.3%)absmax vector-wise 16.48 (+32.4%)zeropoint vector-wise 13.47 (+8.2%)LLM.int8() absmax 12.45 (+0.0%)
Table 1 of the paper: perplexity of each 8-bit method divided by the 32-bit float baseline at the same model size (log scale). Toggle methods above; hover or use the slider to read a model size. Only the decomposition rows stay on the baseline at 13B.

Plain absmax has 242% higher perplexity than the baseline at 125M and 53% higher at 13B. Absmax vector-wise is still 40% above baseline at 125M and 32% above at 13B. The two LLM.int8() rows stay within 0.7% of the baseline at every size, from 25.83 against 25.65 at 125M to 12.45 against 12.45 at 13B. Adding decomposition takes absmax vector-wise from 35.84 to 25.83 at 125M and from 16.48 to 12.45 at 13B, and Table 4 lists outliers even in the smallest fairseq model (quartiles −40 to −11, in 25% of its layers). Zeropoint LLM.int8() is within 0.14 of absmax LLM.int8() at every size, which the paper takes as evidence that once outliers are removed the remaining features are symmetric.

Table 1 stops at 13B. The 175B evidence is the paper's Figure 1, which plots mean zero-shot accuracy (answering benchmark questions with no examples in the prompt) on WinoGrande, HellaSwag, PIQA and LAMBADA for Meta's OPT models up to 175B parameters: LLM.int8() matches the 16-bit baseline at all sizes, while the 8-bit absmax vector-wise baseline falls to random performance once outliers reach every layer at 6.7B. The paper reports no perplexity numbers at 175B and gives Figure 1 only as a plot.

Multiplying parameter count by bytes per parameter accounts for most of Table 2 of the paper. BLOOM-176B, BigScience's open 176B model, shrinks by 1.96 times overall, close to 2 because almost all parameters are in converted linear layers.

HardwareTotal GPU memoryLargest in 16-bitLargest in 8-bit
8× A100 80 GB640 GBOPT-175B / BLOOMOPT-175B / BLOOM
8× A100 40 GB320 GBOPT-66BOPT-175B / BLOOM
8× RTX 3090 24 GB192 GBOPT-66BOPT-175B / BLOOM
4× RTX 3090 24 GB96 GBOPT-30BOPT-66B
Colab Pro15 GBGPT-J-6BOPT-13B
Colab12 GBGPT-2 1.3BT0/T5-11B

The highlighted row is the one the abstract refers to: OPT-175B at 8 bits needs 175 GB of weights, which fits in 192 GB across eight RTX 3090 cards, and the 350 GB required at 16 bits does not. The figures count weights only. Attention caches and the rest of the runtime also take memory, so the real margin is smaller.

Speed

Appendix D measures the raw cost of one feed-forward multiplication at each GPT-3 size, including quantizing the input and dequantizing the output, which is required whenever the rest of the model runs in FP16.

GPT-3 size2.7B6.7B13B175B
Model dimension25604096514012288
Int8 matmul alone1.63×1.67×2.13×2.29×
Vector-wise0.94×1.18×1.59×2.00×
LLM.int8() (vector-wise + decomposition)0.64×0.86×1.22×1.81×

Values below 1× are slower than FP16. The integer multiplication itself is faster than FP16 once a GPU is well saturated, but the conversion and the decomposition cost more than that saves on models up to 6.7B parameters. At GPT-3 Small (dimension 768), the smallest size in the paper's Table 5 and one not shown above, LLM.int8() reaches 0.14×. End to end on BLOOM-176B with Hugging Face, batch size 1 takes 253 ms per token on 8 A100s against 239 ms in 16-bit (bfloat16), and 8-bit brings the model down to 3 A100s at about the same latency (247 ms). With batch size 32, 8 A100s in bfloat16 take 9.94 ms per token and 3 A100s in 8-bit take 9.11.

Implementation notes

The bitsandbytes library implements this as custom GPU kernels (version 0.41.0 requires a CUDA GPU), and Hugging Face Transformers exposes it through a config. Loading looks like this:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_threshold=6.0)
model = AutoModelForCausalLM.from_pretrained(
    "bigscience/bloom",
    device_map="auto",
    quantization_config=config,
)

With this setting the 8-bit layers are bnb.nn.Linear8bitLt with has_fp16_weights=False, so the stored weights are int8 and the 16-bit branch reads them back as in the LLM.int8() listing above (the W_O line). The last layer, usually lm_head, stays in its original precision (the Hugging Face integration keeps it for stability). Everything else in the model that has parameters, such as embeddings and layer norms, also stays at its original precision: LLM.int8() replaces the linear layers only.

The setting llm_int8_threshold controls the split. Lowering it sends more columns to the 16-bit branch, more accurate at the cost of more FP16 arithmetic. Raising it leaves some outlier columns in the int8 branch, the failure in Figure 2.

Limits and later work

The paper states four limits. It studies only int8, and the authors suspect 8-bit floats would do better if GPUs supported them. It tests models only up to 175B parameters, and further emergent behaviors may appear beyond that. It leaves the attention function in 16-bit, since that function has no parameters and quantizing it needed additional methods. And it covers inference: Appendix E reports that training with 8-bit attention layers degrades results, and that mixed-precision decomposition recovers only part of the loss (8-bit attention projections raise training perplexity from 9.99 to 10.52 at 1.1B parameters, and putting the attention function in 8-bit too raises it from 16.74 to 19.00 at best at 209M).

Two later methods take a different route on the same problem. SmoothQuant moves the large range from the hidden states into the weights using an exact rescale, then quantizes both in 8 bits with no 16-bit branch. QLoRA, also from Dettmers, stores the model in 4 bits and trains a small adapter at higher precision, which addresses the fine-tuning limitation. LLM.int8() was also developed alongside nuQmm and ZeroQuant, which use finer per-group scaling and custom CUDA kernels; ZeroQuant reaches zero degradation at 20B parameters and GLM-130B later applied the insights to a 130B model, storing weights in 8 bits and multiplying in 16.

Provenance Verified against primary literatureHow we verify
Dettmers, Lewis, Belkada & Zettlemoyer (2022)The paper (arXiv:2208.07339v2, NeurIPS 2022). Absmax and zeropoint quantization, vector-wise quantization (Eq. 7), mixed-precision decomposition with alpha = 6.0 (Eq. 8), Table 1 perplexities, the Table 4 outlier statistics, Table 5 and 6 speed.
bitsandbytes 0.41.0, _functions.py lines 327-389, 420-421MatMul8bitLt.forward. Outlier columns are the unique column indices of entries with |x| >= threshold; hidden states are quantized in every call; the 16-bit branch is torch.matmul(subA, state.subB). In the has_fp16_weights=False path, subB is read back from the int8 weights as outliers * SCB / 127.
bitsandbytes 0.41.0, kernels.cu kgetColRowStats and kDoubleRowColQuant; functional.py line 2245Row and column maxima skip entries at or above the threshold (they are zeroed first, lines 2208-2217). Scale constant C = 127, so codes run over [-127, 127]. Dequantization multiplies by rowStats * colStats / (127 * 127).
bitsandbytes 0.41.0, functional.py lines 2257-2267Zeropoint code path: qx = 255 / (max - min), zpx = round(min * qx), stored value round(qx * x - zpx). The zeropoint is computed from min times the dynamic-range scale, and the stored codes are unsigned 0..255 where the paper states [-127, 127].
Hugging Face Transformers quantization_config.py (v4.46.0), HF blog "A Gentle Introduction to 8-bit Matrix Multiplication"llm_int8_threshold defaults to 6.0; load_in_8bit loads with has_fp16_weights=False and leaves lm_head in its original precision.
IEEE 754 binary16 (Wikipedia, "Half-precision floating-point format")Largest finite value (2 - 2^-10) * 2^15 = 65504.
correctionCorrections. (1) Eq. (2) prints the zeropoint as round(X * min X). A zeropoint is a scalar, so X cannot be the matrix; the code uses round(min * nd), and Eq. (3) omits the matching subtraction, which would overflow int8 for one-sided inputs (X in [0, 10] gives nd X up to 254). This page teaches zp = round(nd * min) + 127 and X_i8 = round(nd X) - zp, which lands in [-127, 127] and matches the code up to its unsigned range. (2) Sec. 4.2 writes the maximum 16-bit value as 65535; that is the uint16 maximum, and the FP16 maximum is 65504. (3) The introduction claims modest speedups for models "of size 6.7B parameters or larger", but Table 5 shows LLM.int8() at 0.86x at 6.7B, a slowdown, and App. D says only the 13B and 175B models speed up. (4) Sec. 3.4 says models after 2.7B perform worse than smaller models; in Table 1 absmax at 6.7B (14.59) still beats 2.7B (15.11), and only 13B regresses. (5) Sec. 4.2 calls the outlier count strictly monotonic in perplexity; Table 4 is non-decreasing with ties (1, 2, 2, 2, 2, 4, 5, 6, 6, 7). (6) Figure 2 and Sec. 3.2 multiply the outlier columns with 16-bit weights, but the released default path (has_fp16_weights=False) reads those weights back from int8, so only the hidden-state side stays exact and the extra 0.1% of memory is not used. The paper does not say which path produced Table 1.

Questions you might still have

?

Do I have to retrain or fine-tune the model?
No. LLM.int8() is applied to a finished 16-bit or 32-bit checkpoint when it loads: the model is loaded, converted to int8 and used immediately, with no further training or tuning. Earlier 8-bit methods for BERT-sized models usually needed quantization-aware fine-tuning.

?

Why does a threshold of 6 work across models?
The paper reports it as an empirical finding: perplexity degradation stops once features of magnitude 6 or more are treated as outliers. Table 4 notes that ordinary feature dimensions usually stay in [-3.5, 3.5], while outliers are 3-20 times larger. Nothing in the paper derives 6 from theory, and the threshold is exposed as a setting (llm_int8_threshold) in the code.

?

Is the saving in the hidden states or the weights?
The weights. Hidden states are recomputed on every forward pass and are quantized to int8 only for the duration of each matrix multiplication; the outlier columns stay in 16-bit inside that one operation. The memory halves because the stored weights of the linear layers become int8.

?

Why not store outliers column-wise in int8 with their own scale?
An inner product sums over the feature index h, so a scale that depends on h stays inside the sum and every term would need its own division. Integer matrix multiplication has one scale per row of X and one per column of W for exactly that reason. A separate 16-bit multiply for those few columns avoids the problem. The SmoothQuant explainer covers a different route, moving the range from the hidden states into the weights with an exact rescale.

?

Can I train or fine-tune in 8-bit?
Only with difficulty. Appendix E reports that 8-bit feed-forward layers train without degradation, but 8-bit attention projections degrade at 1.1B parameters and on translation, and mixed-precision decomposition recovers only part of the loss. The QLoRA explainer covers the approach people later used: keep the base model frozen in low precision and train a small adapter at higher precision.

?

What are these outlier dimensions, physically?
The paper measures them and does not explain them. Its related-work section points to Kovaleva et al. (2021), who attribute outlier dimensions in BERT-family models to LayerNorm, and to Puccetti et al. (2022), who tie them empirically to token frequency in the training data. The paper adds the scale result: they appear in all layers once models pass about 6.7B parameters, and removing them halves the attention peak.

?

Does this make inference faster?
Sometimes, for large models. Table 5 reports LLM.int8() matrix multiplications at 1.22x at 13B and 1.81x at 175B, and slower than 16-bit for everything at 6.7B or below, where it reaches 0.14x at the smallest size. End to end on BLOOM-176B the difference in latency per token is small, and 8-bit lets the model run on fewer GPUs.

Footnotes & further reading

  1. The paper: Dettmers, Lewis, Belkada & Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022; arXiv August 2022, revised November 2022). Code: bitsandbytes.
  2. Hugging Face integration: A Gentle Introduction to 8-bit Matrix Multiplication.
  3. Outlier dimensions: Kovaleva, Kulshreshtha, Rogers & Rumshisky, BERT Busters: Outlier Dimensions that Disrupt Transformers (2021); Puccetti, Rogers, Drozd & Dell'Orletta, Outliers Dimensions that Disrupt Transformers are Driven by Frequency (2022).
  4. Parallel multi-billion quantization work: Park et al., nuQmm; Yao et al., ZeroQuant; and Zeng et al., GLM-130B (all 2022). Later: Xiao et al., SmoothQuant (2022), covered in the SmoothQuant explainer.
  5. Row-wise quantization: Khudia et al., FBGEMM (2021). The FP16 format: the binary16 specification and the site's mixed precision training explainer.