LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Cut a transformer's GPU memory in half by storing almost all its weights in 8 bits and using 16-bit multiplication for the few feature dimensions that hold huge values.
Standard 8-bit quantization uses a single scale for a whole tensor, so one large value makes every other value less precise. LLM.int8() gives each row of hidden states and each column of weights its own scale, and sends the few feature dimensions that hold very large values through a separate 16-bit multiplication.
Explaining the paperLLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleSix feature dimensions, about 0.1% of the total, hold values up to 20 times larger than the others, and they were the reason 8-bit inference failed on models above a few billion parameters.
Storing the weights of a 175-billion-parameter language model, the size of GPT-3, at 16 bits each takes 350 GB of GPU memory. Eight A100 cards with 40 GB each hold 320 GB, so the weights alone do not fit. Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer posted LLM.int8() in August 2022 with a way to hold almost all of those weights in 8-bit integers, which halves the memory, and to apply it to a finished checkpoint at load time with no retraining.1
Eight-bit inference existed already, but the paper says it had been studied only on models under 350 million parameters. On larger models it damages predictions: at 13 billion parameters the standard method scores a perplexity of 19.08 against 12.45 at full precision. Perplexity is the usual quality score for a language model, the number of equally likely next tokens the model is effectively choosing between, so lower is better. The paper traces the damage to a small set of feature dimensions holding values far larger than the rest, and handles those separately.
The method has two parts. This page works up to them in order:
- how a number becomes an 8-bit integer, and why one large value costs every other value precision;
- how a matrix multiplication runs on integers, and how a separate scale per row and per column reduces rounding error (vector-wise quantization);
- the outlier features that appear across every layer of large transformers;
- how those few columns are multiplied in 16-bit while everything else stays in 8-bit (mixed-precision decomposition);
- the measured results, the speed, and how to use it.
What the weights cost
Two kinds of tensors matter here. Weights are the trained parameters, stored in GPU memory for the whole run. Hidden states are the intermediate vectors a layer computes as a batch of tokens passes through; they exist briefly and are rebuilt on every forward pass. The memory saving in this paper comes entirely from the weights. The accuracy problem it has to solve comes mostly from the hidden states.
For transformers at and above 6.7B parameters, the feed-forward layers and the linear projections inside attention hold 95% of the parameters and account for 65-85% of the computation; embeddings hold most of the rest. The feed-forward and projection layers are the ones LLM.int8() converts. The attention computation itself (the dot products and the softmax of the Transformer) has no parameters and stays in 16-bit.
A 16-bit float (FP16) spends 1 bit on the sign, 5 on the exponent and 10 on the fraction. It covers magnitudes up to 65,504, with spacing that shrinks as numbers get closer to zero, so a value near 1 and a value near 1000 both keep about three decimal digits of relative precision. An int8 holds whole numbers, 255 of them in practice (−127 to 127), spaced evenly, with nothing between them. The paper uses int8 because it is the only 8-bit type GPUs support in hardware. The product of two int8 values fits in 16 bits, since 127² is 16,129, and a sum of products accumulates in an int32 with room for about 133,000 maximum-size terms. The largest hidden dimension in the paper's tables is 49,152, so the integer multiplication cannot overflow.
How 8-bit quantization works
Converting real numbers to int8 means picking a multiplier, scaling, and rounding. The simplest choice, absmax quantization, scales a tensor so that its largest absolute value lands on 127:
The tensor is the FP16 input, is the largest absolute value in the tensor, and rounds to the nearest integer. The distance between adjacent codes in the original units is , and every value lands within half of that distance of a code.
Take a row of a hidden state whose values run from −3.5 to 3.5, which the paper reports as the usual range for ordinary features. The step is 3.5/127, or 0.0276, so 3.5 becomes 127, 0.21 becomes 8, and the stored integers keep the row's shape closely. Now put one value of −44 in the same row. The largest absolute value is 44, the step becomes 44/127, or 0.346, 12.6 times wider, and everything below 0.173 in magnitude rounds to zero. The 0.21 that used to be code 8 is now code 1.
A ruler with 255 marks, centered on zero with 127 on each side, has to reach the longest thing it measures in either direction. Measure one object 44 cm long and the marks end up 3.5 mm apart, so anything shorter than about 1.7 mm reads as zero, for every other object measured with the same ruler.
Absmax centers its range on zero. If the data is all positive, codes −127 to −1 go unused, and with a one-sided outlier the same waste happens at the scale of the outlier. Zeropoint quantization maps the observed range onto all 255 codes with a scale and an offset:
The stored value sits in [−127, 127], and adding back gives , which is what gets multiplied. For values in [0, 10], the scale is 25.4, which makes the step half the absmax step of 10/127, and is 127, so a value of 5 is stored as 0. Equations (2) and (3) are corrected versions of the paper's (the Provenance panel lists what changed). To multiply two zeropoint numbers, add each zeropoint back before the product:
On a CPU with a dedicated instruction this is one operation. GPUs and TPUs have no such instruction, so the product has to be unrolled:
The first term runs in int8 and the other three in int16 or int32. The paper notes this makes zeropoint slow on GPUs, which is why absmax is the common choice in practice despite its weaker precision on asymmetric data.
Multiplying matrices in integers
Take hidden states of shape ( tokens in the batch, each with features) and weights of shape ( output features). With one scale constant per tensor, the 8-bit multiplication with FP16 inputs and FP16 outputs is:
is absmax or zeropoint quantization, and are their scale constants (the paper writes them as or ). The integers multiply, accumulate in int32, and the int32 sum gets multiplied by to return to FP16 units. The weights are quantized once when the model loads and stay in int8. The hidden states are quantized on every forward pass, since each batch produces different ones.
A worked inner product, with one row of the hidden state and one column of the weights:
The exact product is 1.314 (the four terms are 0.231, 0.520, 0.423 and 0.140). The scales are 127/2.80 for the row and 127/1.10 for the column, so the integers are (10, −59, 21, 127) and (127, −46, 104, 6). Their inner product is 6930. Dividing by the product of the two scales, 16129/3.08, gives 1.3234, which is 0.7% above the exact value.
Vector-wise quantization
Matrix multiplication is a set of independent inner products, one per pair of a row of and a column of . Each inner product can use its own scale constants, because the constants factor out of the sum over the feature index :
So give each of the rows of a constant, collected in a vector , and each of the columns of a constant, collected in . Denormalizing is element-wise division by their outer product:
A scale per feature would seem to be the natural way to deal with a feature that runs large, but it stays inside the sum, so each term would need its own division, and that is the operation integer matrix hardware does not provide. Only row and column constants come out of the sum. The paper calls scaling by rows of alone row-wise quantization (from FBGEMM, Facebook's low-precision matrix library), with one scale for all of , and scaling both rows of and columns of vector-wise quantization; Section 4.3 attributes vector-wise's advantage to the extra precision on the weights. As code:
# X: hidden states, shape [s, h], fp16, new on every forward pass
# W: weights, shape [h, o], fp16, quantized once when the model loads
c_x = 127 / abs(X).max(dim=1).values # one scale per row of X, shape [s]
c_w = 127 / abs(W).max(dim=0).values # one scale per column of W, shape [o]
X_i8 = round(X * c_x[:, None]).int8() # every entry now in [-127, 127]
W_i8 = round(W * c_w[None, :]).int8()
C_i32 = X_i8.int32() @ W_i8.int32() # int8 products, int32 accumulator
C_f16 = C_i32 / (c_x[:, None] * c_w[None, :]) # denormalize by the outer productFigure 2 runs this on a 10×16 hidden state and a 16×6 weight matrix. Finer scaling helps when the planted data is ordinary. Then toggle on a large column, and compare the size of the three bars.
In the paper's Table 1 (perplexity by model size, plotted in Figure 5), vector-wise quantization alone has lower perplexity than row-wise at 1.3B and 2.7B, ties at 6.7B and 13B, and does worse at 125M, and it cannot help with a single large column. Row-wise and vector-wise both scale by rows of , so when the same feature is large in most rows, each row's maximum is that feature and the row scale is set by it.
Outlier feature dimensions
In the paper's terms, a feature is one of the columns of a hidden state. The authors tracked features across all layers of ten pretrained models from three software frameworks, and called a feature an outlier if it met three conditions at once: a magnitude of at least 6.0, present in at least 25% of the layers, and present in at least 6% of the sequence positions.
The ordinary feature dimensions usually stay within [−3.5, 3.5]. A 6.7B model processing a sequence of 2,048 tokens produced about 150,000 outlier values across the whole transformer, and all of them sat in 6 feature dimensions, about 0.1% of the total (the 6.7B row of the paper's Table 5 lists 4,096 dimensions, which makes 6 columns 0.15%). Across the models tested the count of outlier dimensions never exceeded 7, and on the large models almost all of them were one-sided, either always positive or always negative.
To measure the effect of these dimensions the authors set them to zero before each attention projection and compared the largest value in the resulting attention distribution: the mean of that top-1 probability fell from about 40% to about 20%. Propagating the zeroed states through the full model raised perplexity by 600-1000%. Zeroing 7 random non-outlier dimensions instead lowered the top-1 probability by only 0.02-0.3% and raised perplexity by about 0.1%. The outliers are required for the model to work, so they cannot be clipped away to protect the int8 range.
Measured by parameter count, the affected share of layers jumps between 6.0B (GPT-J, 62% of layers) and 6.7B (a model trained with Meta's fairseq, 100%), and sequence positions go from 28% to 75%. Those two points come from different model families and training code, so part of the jump may reflect the comparison across stacks. The paper argues that perplexity is the better predictor: plotted against perplexity, the number of outlier dimensions never decreases (1, 2, 2, 2, 2, 4, 5, 6, 6, 7) and the emergence looks gradual, while against parameter count it is non-monotonic, dropping to 2 at GPT-2 1.5B after 4 at 1.3B. The median magnitude of the most common outlier rises from about 16 at 2.7B to about 40 at 6.7B and 58 at 13B.
For quantization, a value of −44 in a row of ordinary features forces a step of 0.346 where 0.0276 would do. The paper describes the result as most quantization bins being empty and small values being quantized to zero. Since nearly every row contains the same large columns, row-wise and vector-wise scales cannot avoid it. Zeropoint quantization handles the one-sided shape better, and in Table 1 it holds out longer, but at 13B the step is large enough that it fails too (13.94 against 12.45).
How LLM.int8() works: mixed-precision decomposition
The large values sit in a handful of columns, so the method pulls those columns out of the int8 multiplication, multiplies them separately in 16-bit, and adds the two results. For hidden states and weights , let be the set of feature dimensions that contain at least one value with magnitude above the threshold . Then:
Here is column of and is row of , so each term is one feature's contribution to the output. The sum over is the 16-bit branch; the sum over the rest is the int8 branch, with the denormalization from (7). The paper sets to 6.0 and reports that this is enough to bring perplexity degradation close to zero. The value sits above the ordinary range of [−3.5, 3.5] and below the outliers. Two things are added to the vector-wise int8 multiplication of Eq. (7): the outlier columns stop contributing to the integer sum, and the row maxima are computed without them.
The released code does both, comparing the threshold against the raw FP16 values. For each hidden-state tensor at each layer it collects the columns with any entry of magnitude at least , so can differ between batches and between layers, and the row maxima ignore those entries (in the kernels, they are set to zero before the maximum is taken). So once the outliers are removed the row's step goes back to the ordinary 0.0276. Continuing the worked example, add a feature with value −44.0 and weight 0.12 to the vectors above. That feature adds −44 × 0.12, or −5.28, so the exact product becomes −3.966.
With plain absmax, the row maximum is 44 and the five rounded values are (1, −4, 1, 8, −127). The ordinary four contribute 463 to the integer sum, which denormalizes to 1.389 (true 1.314, off by 5.7%). With decomposition the ordinary four are computed as in the original worked example, 1.3234 (off by 0.7%), and the outlier term −5.28 is added exactly, for a total of −3.957.
# One 8-bit linear layer: output = X @ W, with outlier columns in 16-bit.
# 1. Find outlier columns in this batch's hidden states (alpha = 6.0).
O = where((abs(X) >= alpha).any(dim=0)) # at most 7 in the paper's models
# 2. Scale the rest. Row maxima ignore any entry >= alpha.
X_rest = X.clone()
X_rest[abs(X_rest) >= alpha] = 0
c_x = 127 / abs(X_rest).max(dim=1).values # [s]
X_i8 = round(X_rest * c_x[:, None]).int8() # [s, h]
X_i8[:, O] = 0 # outlier columns leave the int8 branch
# 3. Integer multiply. W_i8 and c_w were computed once, at load time.
C_i32 = X_i8.int32() @ W_i8.int32() # [s, o]
output = C_i32 / (c_x[:, None] * c_w[None, :])
# 4. 16-bit branch, added to the 8-bit result.
W_O = W_i8[O, :] / c_w[None, :] # released code: read back from int8
output += X[:, O] @ W_O # shapes [s, |O|] @ [|O|, o]Moving α in Figure 4 changes which columns leave the int8 branch. Below 3.4, ordinary columns start to count as outliers, and at α = 1 nearly everything goes to the 16-bit branch, which is accurate and saves no memory. Between about 4 and the smallest outlier the split does not change, so the error curve is flat there. Above the largest outlier nothing is removed and the error returns to the plain vector-wise level.
The paper puts the cost of the 16-bit branch at about 0.1% of memory, since at most 7 dimensions were found up to 13B parameters. The released code path used for 8-bit loading (has_fp16_weights=False) does not store those weights at 16 bits. It reconstructs the outlier columns of from the int8 weights, multiplies them by the exact hidden-state values in FP16, and discards them after the call. The 16-bit branch therefore adds no stored memory, and its weight side is rounded: a value of −44 times a weight error of up to contributes an output error that the 16-bit copy does not have. Toggling Figure 4 shows the effect. The paper does not say which code path produced its accuracy tables.
Perplexity, accuracy and memory
Table 1 of the paper measures perplexity on validation data from C4, a web-text corpus, for five pretrained models from 125M to 13B parameters, against a 32-bit float baseline. Figure 5 plots each method as its ratio to that baseline, so a line on 1.0 means no degradation.
Plain absmax has 242% higher perplexity than the baseline at 125M and 53% higher at 13B. Absmax vector-wise is still 40% above baseline at 125M and 32% above at 13B. The two LLM.int8() rows stay within 0.7% of the baseline at every size, from 25.83 against 25.65 at 125M to 12.45 against 12.45 at 13B. Adding decomposition takes absmax vector-wise from 35.84 to 25.83 at 125M and from 16.48 to 12.45 at 13B, and Table 4 lists outliers even in the smallest fairseq model (quartiles −40 to −11, in 25% of its layers). Zeropoint LLM.int8() is within 0.14 of absmax LLM.int8() at every size, which the paper takes as evidence that once outliers are removed the remaining features are symmetric.
Table 1 stops at 13B. The 175B evidence is the paper's Figure 1, which plots mean zero-shot accuracy (answering benchmark questions with no examples in the prompt) on WinoGrande, HellaSwag, PIQA and LAMBADA for Meta's OPT models up to 175B parameters: LLM.int8() matches the 16-bit baseline at all sizes, while the 8-bit absmax vector-wise baseline falls to random performance once outliers reach every layer at 6.7B. The paper reports no perplexity numbers at 175B and gives Figure 1 only as a plot.
Multiplying parameter count by bytes per parameter accounts for most of Table 2 of the paper. BLOOM-176B, BigScience's open 176B model, shrinks by 1.96 times overall, close to 2 because almost all parameters are in converted linear layers.
| Hardware | Total GPU memory | Largest in 16-bit | Largest in 8-bit |
|---|---|---|---|
| 8× A100 80 GB | 640 GB | OPT-175B / BLOOM | OPT-175B / BLOOM |
| 8× A100 40 GB | 320 GB | OPT-66B | OPT-175B / BLOOM |
| 8× RTX 3090 24 GB | 192 GB | OPT-66B | OPT-175B / BLOOM |
| 4× RTX 3090 24 GB | 96 GB | OPT-30B | OPT-66B |
| Colab Pro | 15 GB | GPT-J-6B | OPT-13B |
| Colab | 12 GB | GPT-2 1.3B | T0/T5-11B |
The highlighted row is the one the abstract refers to: OPT-175B at 8 bits needs 175 GB of weights, which fits in 192 GB across eight RTX 3090 cards, and the 350 GB required at 16 bits does not. The figures count weights only. Attention caches and the rest of the runtime also take memory, so the real margin is smaller.
Speed
Appendix D measures the raw cost of one feed-forward multiplication at each GPT-3 size, including quantizing the input and dequantizing the output, which is required whenever the rest of the model runs in FP16.
| GPT-3 size | 2.7B | 6.7B | 13B | 175B |
|---|---|---|---|---|
| Model dimension | 2560 | 4096 | 5140 | 12288 |
| Int8 matmul alone | 1.63× | 1.67× | 2.13× | 2.29× |
| Vector-wise | 0.94× | 1.18× | 1.59× | 2.00× |
| LLM.int8() (vector-wise + decomposition) | 0.64× | 0.86× | 1.22× | 1.81× |
Values below 1× are slower than FP16. The integer multiplication itself is faster than FP16 once a GPU is well saturated, but the conversion and the decomposition cost more than that saves on models up to 6.7B parameters. At GPT-3 Small (dimension 768), the smallest size in the paper's Table 5 and one not shown above, LLM.int8() reaches 0.14×. End to end on BLOOM-176B with Hugging Face, batch size 1 takes 253 ms per token on 8 A100s against 239 ms in 16-bit (bfloat16), and 8-bit brings the model down to 3 A100s at about the same latency (247 ms). With batch size 32, 8 A100s in bfloat16 take 9.94 ms per token and 3 A100s in 8-bit take 9.11.
Implementation notes
The bitsandbytes library implements this as custom GPU kernels (version 0.41.0 requires a CUDA GPU), and Hugging Face Transformers exposes it through a config. Loading looks like this:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_threshold=6.0)
model = AutoModelForCausalLM.from_pretrained(
"bigscience/bloom",
device_map="auto",
quantization_config=config,
)With this setting the 8-bit layers are bnb.nn.Linear8bitLt with has_fp16_weights=False, so the stored weights are int8 and the 16-bit branch reads them back as in the LLM.int8() listing above (the W_O line). The last layer, usually lm_head, stays in its original precision (the Hugging Face integration keeps it for stability). Everything else in the model that has parameters, such as embeddings and layer norms, also stays at its original precision: LLM.int8() replaces the linear layers only.
The setting llm_int8_threshold controls the split. Lowering it sends more columns to the 16-bit branch, more accurate at the cost of more FP16 arithmetic. Raising it leaves some outlier columns in the int8 branch, the failure in Figure 2.
Limits and later work
The paper states four limits. It studies only int8, and the authors suspect 8-bit floats would do better if GPUs supported them. It tests models only up to 175B parameters, and further emergent behaviors may appear beyond that. It leaves the attention function in 16-bit, since that function has no parameters and quantizing it needed additional methods. And it covers inference: Appendix E reports that training with 8-bit attention layers degrades results, and that mixed-precision decomposition recovers only part of the loss (8-bit attention projections raise training perplexity from 9.99 to 10.52 at 1.1B parameters, and putting the attention function in 8-bit too raises it from 16.74 to 19.00 at best at 209M).
Two later methods take a different route on the same problem. SmoothQuant moves the large range from the hidden states into the weights using an exact rescale, then quantizes both in 8 bits with no 16-bit branch. QLoRA, also from Dettmers, stores the model in 4 bits and trains a small adapter at higher precision, which addresses the fine-tuning limitation. LLM.int8() was also developed alongside nuQmm and ZeroQuant, which use finer per-group scaling and custom CUDA kernels; ZeroQuant reaches zero degradation at 20B parameters and GLM-130B later applied the insights to a 130B model, storing weights in 8 bits and multiplying in 16.
Questions you might still have
Do I have to retrain or fine-tune the model?
No. LLM.int8() is applied to a finished 16-bit or 32-bit checkpoint when it loads: the model is loaded, converted to int8 and used immediately, with no further training or tuning. Earlier 8-bit methods for BERT-sized models usually needed quantization-aware fine-tuning.
Why does a threshold of 6 work across models?
The paper reports it as an empirical finding: perplexity degradation stops once features of magnitude 6 or more are treated as outliers. Table 4 notes that ordinary feature dimensions usually stay in [-3.5, 3.5], while outliers are 3-20 times larger. Nothing in the paper derives 6 from theory, and the threshold is exposed as a setting (llm_int8_threshold) in the code.
Is the saving in the hidden states or the weights?
The weights. Hidden states are recomputed on every forward pass and are quantized to int8 only for the duration of each matrix multiplication; the outlier columns stay in 16-bit inside that one operation. The memory halves because the stored weights of the linear layers become int8.
Why not store outliers column-wise in int8 with their own scale?
An inner product sums over the feature index h, so a scale that depends on h stays inside the sum and every term would need its own division. Integer matrix multiplication has one scale per row of X and one per column of W for exactly that reason. A separate 16-bit multiply for those few columns avoids the problem. The SmoothQuant explainer covers a different route, moving the range from the hidden states into the weights with an exact rescale.
Can I train or fine-tune in 8-bit?
Only with difficulty. Appendix E reports that 8-bit feed-forward layers train without degradation, but 8-bit attention projections degrade at 1.1B parameters and on translation, and mixed-precision decomposition recovers only part of the loss. The QLoRA explainer covers the approach people later used: keep the base model frozen in low precision and train a small adapter at higher precision.
What are these outlier dimensions, physically?
The paper measures them and does not explain them. Its related-work section points to Kovaleva et al. (2021), who attribute outlier dimensions in BERT-family models to LayerNorm, and to Puccetti et al. (2022), who tie them empirically to token frequency in the training data. The paper adds the scale result: they appear in all layers once models pass about 6.7B parameters, and removing them halves the attention peak.
Does this make inference faster?
Sometimes, for large models. Table 5 reports LLM.int8() matrix multiplications at 1.22x at 13B and 1.81x at 175B, and slower than 16-bit for everything at 6.7B or below, where it reaches 0.14x at the smallest size. End to end on BLOOM-176B the difference in latency per token is small, and 8-bit lets the model run on fewer GPUs.
Footnotes & further reading
- The paper: Dettmers, Lewis, Belkada & Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022; arXiv August 2022, revised November 2022). Code: bitsandbytes.
- Hugging Face integration: A Gentle Introduction to 8-bit Matrix Multiplication.
- Outlier dimensions: Kovaleva, Kulshreshtha, Rogers & Rumshisky, BERT Busters: Outlier Dimensions that Disrupt Transformers (2021); Puccetti, Rogers, Drozd & Dell'Orletta, Outliers Dimensions that Disrupt Transformers are Driven by Frequency (2022).
- Parallel multi-billion quantization work: Park et al., nuQmm; Yao et al., ZeroQuant; and Zeng et al., GLM-130B (all 2022). Later: Xiao et al., SmoothQuant (2022), covered in the SmoothQuant explainer.
- Row-wise quantization: Khudia et al., FBGEMM (2021). The FP16 format: the binary16 specification and the site's mixed precision training explainer.
How could this explainer be improved? Found an error, or something unclear? I read every message.