QLoRA: Efficient Finetuning of Quantized LLMs

Finetune a 65B model on one GPU by freezing it in 4 bits.

Store the frozen base weights in 4 bits and train a small 16-bit adapter through them. The base shrinks enough to fit one card, and the finished model matches 16-bit finetuning.

Finetuning a 65-billion-parameter model the ordinary way needs more than 780 GB of GPU memory, roughly ten datacenter cards. QLoRA does it in about 40 GB, on one. Same model, same final quality.

Pretrained language models are general. To make one good at a specific job, following instructions, writing in a house style, answering questions about your documents, you finetune it: keep training on data for that job so the weights shift toward it. Finetuning is how a raw pretrained model becomes something like ChatGPT.

That power comes at a cost in memory. Finetuning the standard way means updating every weight, and every weight you update drags a train of extra numbers into memory along with it. For a 65B model that train blows past what any single GPU holds, out of reach for almost everyone. Quantization, storing the weights in fewer bits, had shrunk models for inference, but those tricks fell apart during training. So the largest open models stayed finetunable only by the few labs with a datacenter.

QLoRA closes that gap. It keeps the giant base model frozen and stored in 4 bits, and trains a small set of new weights through it, so it all fits on a single 48 GB card with no loss in quality. The authors used it to finetune a 65B model in 24 hours on one GPU and produced Guanaco, at the time the strongest open chatbot. It is now the default way to finetune large models, built into the Hugging Face stack.

A handful of ideas explain it, and none is hard on its own: where finetuning memory actually goes, what LoRA already fixed and what it left behind, why quantization is done in small blocks, the 4-bit number format (NormalFloat) built for neural-network weights, a second round of quantization on top, how the pieces assemble into one layer, and a memory-paging trick that keeps it from crashing. Take them in order.

Where finetuning memory goes

Start with the wall QLoRA is built to get around. When you finetune a model with the usual optimizer, its weight is far from the only number you have to keep in memory for it. Four separate things pile up, and the weight is the smallest.

First, the weights themselves, 2 bytes each in the 16-bit format finetuning normally uses. Second, the gradient of the loss for each weight, another 2 bytes, computed on the backward pass so you know which way to nudge. Third, the optimizer state. Modern training uses Adam, which for every weight keeps two running averages, a mean and a variance of that weight's recent gradients, stored in full 32-bit precision at 4 bytes each. That is 8 more bytes per weight, and it is the single largest bucket. Add them up and each parameter you train carries about 12 bytes, not 2. For 65B parameters that is $65 \times 10^9 \times 12 \approx 780$ GB, before you count the fourth bucket, the activations, the intermediate outputs saved on the forward pass so the backward pass can use them (their size grows with batch size and sequence length, not with parameter count).

Picture it: training one weight is like shipping one book (2 bytes) with a shipping box (the gradient) and two running tallies (Adam's moments) stapled to it. The packaging outweighs the book five to one. Below, the left bar is full 16-bit finetuning, split into those buckets. Slide the model size up to 65B and watch it run off the chart, past every GPU line:

Two of these buckets, the gradient and the optimizer state, exist only because you are updating the weight. If a weight is frozen and never changes, it needs no gradient and no Adam moments; you only pay the 2 bytes to store it. LoRA is built on exactly that observation.

LoRA: freeze the base, train a patch

LoRA (Low-Rank Adaptation) is the method QLoRA extends, so start there. Its move: leave every pretrained weight matrix $\mathbf{W}$ frozen, and learn a small correction beside it. The correction is factored into two skinny matrices whose product has the same shape as $\mathbf{W}$ but far fewer numbers. A layer that used to compute $\mathbf{Y}=\mathbf{X}\mathbf{W}$ now computes:

with $\mathbf{W}\in\mathbb{R}^{h\times o}$ frozen, and the trainable adapter $\mathbf{L}_1\in\mathbb{R}^{h\times r}$, $\mathbf{L}_2\in\mathbb{R}^{r\times o}$ where the rank $r$ is tiny, 64 in QLoRA's experiments against a hidden size in the thousands. The scalar $s=\alpha/r$ just scales the adapter's contribution. $\mathbf{L}_1$ starts from small random values and $\mathbf{L}_2$ starts at exactly zero, so the product $\mathbf{L}_1\mathbf{L}_2$ is zero at the first step and training begins precisely at the pretrained model, then edits it. Picture a thick reference book you are not allowed to rewrite, with a thin stack of sticky notes: you only ever write on, carry, and swap the notes.

Why does this save so much? Because the two expensive buckets, gradient and optimizer state, are now paid only on the tiny adapter, not on the frozen base. For a 7B model the adapter's parameters take about 26 MB against the base's several GB, and its optimizer state is proportionally small. Since adapters are so cheap, you can afford one on every linear layer rather than a couple, and that matters: QLoRA finds that adapters on all linear layers, not just the attention query and value projections the original LoRA paper used, are what let 4-bit finetuning match full 16-bit quality. With adapters everywhere, the rank $r$ barely affects the final result.

But look again at the middle bar in Figure 1. LoRA deletes the gradient and optimizer buckets, a huge cut, and yet the 65B bar is still 130 GB, more than two 48 GB cards, because of the one thing LoRA does not touch: the frozen base weights still sit in memory at full 16-bit precision, all $65\times10^9$ of them at 2 bytes each. Freezing a weight stops you paying to train it, but you still pay to store it. The book got no lighter; you just stopped photographing every page. Shrinking that resident base, the last big bucket LoRA leaves standing, is what QLoRA adds.

The rest of the method does one thing in three parts: store that frozen base in 4 bits instead of 16, carefully enough that training through it loses nothing. Why 4 and not a safer 8? Because 8-bit would only halve the base to about 65 GB, still over a 48 GB card, and the results below show 4-bit already gives up nothing after finetuning, so there is no reason to stop halfway. Everything from here is about that 4-bit storage.

Quantize in blocks, not all at once

Quantization turns a high-precision number into one of a small set of codes. The simplest scheme, and the base that everything else sits on, is absmax quantization. To pack a tensor into signed 8-bit integers, whose codes run from $-127$ to $127$, you find the largest magnitude in the tensor, call it $\text{absmax}$, and rescale everything by $c = 127/\text{absmax}$ so that largest value lands on the top code:

The number $c$ is called the quantization constant or scale. To read a weight back you just divide by it, $\operatorname{dequant}(c,\mathbf{X}^{\text{Int8}}) = \mathbf{X}^{\text{Int8}}/c$. That round-trip looks lossless, and the algebra is: multiplying by $c$ and dividing by $c$ is exactly invertible. The only thing that actually loses information is the $\operatorname{round}$ to the nearest integer code. (Absmax is symmetric, so it uses $255$ of the 256 available byte values,$-127$ through $127$; the code for $-128$ is left unused.)

A single scale for a whole tensor breaks down as soon as the values are uneven. The scale is pinned to the largest magnitude in the tensor, so one freakishly large value, an outlier, sets it for everyone. When the step between codes is $\text{absmax}/127$ and $\text{absmax}$ is huge, every ordinary small weight gets divided by that huge step and rounds to one of just two or three codes near zero. You have 256 codes and effectively use a handful. It is like setting a camera's exposure off one blinding streetlight: the meter maxes out on that one bright pixel and everything else renders as indistinguishable near-black.

Quantizing in blocks defends against it: instead of one scale for the whole tensor, flatten it and cut it into contiguous blocks (QLoRA uses 64 weights per block), and give each block its own scale. Now an outlier can only stretch the scale of its own 64-weight block; every other block keeps a tight scale fitted to its own values. A tensor of $b\times h$ weights becomes $n=(b\times h)/64$ blocks, each with a constant $c_i$. It is the camera metering each tile of the frame on its own, so one bright spot can only wash out its own tile.

Below, in whole-tensor mode, drag the outlier outward and watch the code lines march apart until the ordinary weights all pile onto two or three codes. Switch to per-block and the grid stays fine, because the outlier is off in some other block:

This outlier is a weight outlier, and weights are fairly well-behaved. The much scarier outliers you may have read about, the ones SmoothQuant and LLM.int8() fight, live in the activations at inference time and are far more extreme. QLoRA never meets those, because it quantizes only the weights and, as the next sections show, always converts them back to 16-bit before any matrix multiply. It is the same "an outlier stretches the scale" mechanism, just on a gentler tensor.

NormalFloat: 4 bits for a bell curve

Absmax quantization decides the scale of the codes. It says nothing about where the 16 codes of a 4-bit type should sit inside the range. Uniform spacing (an integer grid) looks obvious but fits the weights badly, because neural-network weights are not spread uniformly. Trained weights sit in a bell curve, clustered near zero with thin tails. So a uniform 4-bit grid spends half its precious 16 codes out in the empty tails where almost no weights are, and starves the crowded center.

QLoRA's NormalFloat (NF4) spends the codes where the weights actually are. It places the 16 codes at the quantiles of a normal distribution, the cuts that slice a bell-curve histogram into 16 bars each holding the same count of weights rather than the same width. Where weights are dense, near zero, the bars are thin and the codes packed close; out in the sparse tails the bars are wide and the codes few. The paper writes each code as the midpoint of two adjacent quantiles:

Toggle between NF4 and the uniform Int4 grid below and watch where each type lands its 16 codes: NF4 packs them under the bell where the weights are, while Int4 strands its outer codes in the near-empty tails. Each bin is shaded by how much of the weight one code has to represent, so Int4's center code glows while its tail codes go dark. Drop a weight and read the rounding error, smallest for NF4:

The term the paper leans on repays a close look. NF4 is called information-theoretically optimal for normal data, and that phrase means something precise and narrow: for an ideal normal input, every bin holds the same number of weights, so all 16 codes get used equally often and the code carries its full $\log_2 16 = 4$ bits of information. Equal-mass bins maximize the entropy of the code, the same reason a file compressor is happiest when all symbols are equally likely. It does not mean NF4 has the smallest possible rounding error. The quantizer that minimizes rounding error is a different one (Lloyd–Max, which puts a code at the mean of each bin and does not make the bins equal-mass). Entropy-optimal and error-optimal are different targets, and NF4 hits the first.

Two smaller points round out the picture. NF4 is deliberately asymmetric: it uses 7 negative codes, 8 positive codes, and one code that is exactly $0$, 16 in all. The exact zero is on purpose, so that padding and genuine zeros quantize with no error at all, which a symmetric grid could not guarantee. And the equal-mass story is an idealization: because each block is normalized by its own maximum, a follow-up analysis found the bins are not perfectly equiprobable in practice, holding anywhere from 2% to 9% at a block size of 64. That imperfection is arguably a feature, since keeping some fidelity for the rare large weights helps more than forcing the bins to be exactly equal.

The numbers themselves come with a caveat. The 16 exact NF4 values are not produced by equation (3) as literally written, because the very first and last quantiles it asks for are the 0th and 100th percentiles of a normal, which are $-\infty$ and $+\infty$. The released code pulls the tail quantiles inward by a small offset and then rescales by the absolute maximum so the extreme codes land exactly at $\pm 1$. That procedure reproduces the published table to seven decimals:

# the 16 NF4 code points (bitsandbytes create_normal_map,
# not equation (3) verbatim: the tail quantiles are ±infinity)
offset = 0.9677083                     # pulls the tail quantiles finite
pos = norm.ppf(linspace(offset, 0.5, 9)[:-1])   # 8 positive quantiles
neg = -norm.ppf(linspace(offset, 0.5, 8)[:-1])  # 7 negative quantiles
codes = sort([*neg, 0.0, *pos])        # 7 neg + exact 0 + 8 pos = 16
codes = codes / max(abs(codes))        # absmax-normalize: ends -> ±1

The 16 fixed numbers run $-1.0,\ -0.70,\ \dots,\ 0,\ \dots,\ 0.72,\ 1.0$. A stored weight is one 4-bit index into this table; to use it you look up the table value and multiply by the block's scale. What you buy is a weight that costs 4 bits instead of 16, placed by a code that wastes none of them on empty space.

Double quantization

The block-wise scheme has a cost we glossed over. Every block of 64 weights needs its own 32-bit scale, and a 32-bit number spread over 64 weights adds $32/64 = 0.5$ bits to every weight's storage. On top of a 4-bit weight, that is another eighth again, and it is pure overhead. Smaller blocks quantize the weights more precisely but pay this overhead more often, while larger blocks are cheaper but let outliers do more damage.

Double Quantization attacks the overhead by quantizing the scales themselves. There are a lot of them, one per 64-weight block, and they form their own list of numbers that is itself compressible. So QLoRA quantizes that list: it stores each block's scale as an 8-bit float, grouped into super-blocks of 256 scales, with one 32-bit scale-of-the-scales per super-block. (The scale values are all positive, so their mean is subtracted first to center them and let a symmetric quantizer do the work.) It is like zipping the index of a zip file: you already compressed the payload, then you notice the table of offsets is itself worth compressing, and you never touch the payload again. The new overhead per weight is:

down from $0.5$, a saving of $0.373$ bits on every parameter. That sounds like rounding dust until you multiply by 65 billion: $0.373 \times 65\times10^9 / 8 \approx 3$ GB (the $/8$ turns bits into bytes), enough to be the difference between a model that fits a card and one that does not. Crucially, Double Quantization compresses only the scales; the weights stay at 4-bit NF4. The total drops from about 4.5 bits per weight to about 4.127. Drag the block size below and watch the two overhead curves; at block size 64 the gap between them is that 3 GB:

The QLoRA layer, end to end

Now assemble the pieces into one working layer. QLoRA keeps two data types with distinct jobs: a storage type, 4-bit NF4, that the frozen weight lives in on the GPU, and a compute type, 16-bit BFloat16, that every matrix multiply actually runs in. Whenever a weight is needed, it is dequantized from 4-bit back up to BFloat16 on the fly, used, and thrown away; no 16-bit copy is ever stored. The full forward pass for one linear layer is equation (1) with the frozen matrix expanded (the scalar $s$ carries over from (1)):

The $\operatorname{doubleDequant}$ undoes both rounds of quantization from the last two sections. It takes the 4-bit weight $\mathbf{W}^{\text{NF4}}$, the 8-bit block scales $c_2$, and the 32-bit scale-of-scales $c_1$, and reconstructs a BFloat16 weight in two nested steps:

Read the nesting inner-first. The inner $\operatorname{dequant}(c_1, c_2)$ uses the 32-bit $c_1$ to turn the 8-bit block scales back into full-precision numbers; the outer $\operatorname{dequant}$ then uses those recovered scales to turn the 4-bit weight back into BFloat16. (The subscripts are counterintuitive: $c_2$ is the first-level scale, the one on each 64-weight block; $c_1$ is the second-level scale on top of it. The two nested dequants are the two rounds of Double Quantization, not the weight being dequantized twice.)

There is a subtle point in the backward pass. The gradient of the loss arrives at the layer's output and splits at the sum. Part of it updates the adapter: it forms $\partial E/\partial \mathbf{L}_1$ and $\partial E/\partial \mathbf{L}_2$, and the optimizer steps those tiny matrices. The other part has to travel further back, to the adapters in earlier layers, and to do that it must pass through the frozen weight, which means multiplying the incoming gradient by $\mathbf{W}^\top$. That $\mathbf{W}^\top$ forces the 4-bit weight to be dequantized to BFloat16 again, on the backward pass this time. So the frozen weight is read every forward pass and every backward pass, and written never. Its own gradient $\partial E/\partial\mathbf{W}$ is never formed, because the weight is never updated. That is why there is no need to differentiate through the quantizer at all: the gradient goes around the frozen weight to the adapters, not into it.

Toggle the pass below. Forward, the input fans into two paths, the frozen 4-bit base (dequantized for its matmul) and the 16-bit adapter, summed into the output. Backward, the gradient reaches the adapter and updates it, and only passes through the frozen base to reach the input:

To make it fully concrete, take one linear layer of a 7B model, a $4096\times4096$ weight matrix, about 16.8 million numbers. In 16-bit it is 33.6 MB; in 4-bit NF4 with double quantization it is 8.7 MB, with$16.8\text{M}/64 = 262{,}144$ block scales riding along. The adapter at rank 64 is $\mathbf{L}_1\in\mathbb{R}^{4096\times64}$ and $\mathbf{L}_2\in\mathbb{R}^{64\times4096}$, about 524 thousand numbers, roughly a thirtieth of the weight's 16.8 million, and 1 MB in 16-bit against the 4-bit weight's 8.7 MB. And the adapter is the only thing carrying a gradient and optimizer state. In code, one layer is a dozen lines:

# one QLoRA linear layer.  storage: 4-bit NF4.  compute: bf16.
def forward(x, W_nf4, c1, c2, L1, L2, s):     # x, L1, L2 are bf16
    W = double_dequant(c1, c2, W_nf4)         # 4-bit -> bf16, on the fly
    return x @ W + s * (x @ L1) @ L2           # frozen base + adapter

def backward(dy, x, W_nf4, c1, c2, L1, L2, s):
    W = double_dequant(c1, c2, W_nf4)         # dequantize again for dx
    dx  = dy @ W.T + s * (dy @ L2.T) @ L1.T    # passes THROUGH frozen W
    dL1 = s * x.T @ (dy @ L2.T)                # only the adapter is
    dL2 = s * (x @ L1).T @ dy                  # updated; dW is never formed
    return dx, dL1, dL2

Notice double_dequant is called in both forward and backward, and that no dW is ever returned. The base is read twice per step and updated zero times. Nothing else happens: the base is a 4-bit archive on the GPU, decompressed to 16-bit just long enough for each multiply, and your edits go to a small separate patch.

Paged optimizers

A practical gap remains, about crashes rather than steady-state size. The fourth memory bucket, activations, is normally kept small by gradient checkpointing, which throws most activations away on the forward pass and recomputes them during the backward pass, trading compute for memory. But recomputation happens in bursts: when the backward pass reaches a long sequence, it briefly rebuilds a whole layer's activations at once, and that transient spike can shoot past the GPU's memory and kill the run with an out-of-memory error.

QLoRA's third contribution, Paged Optimizers, absorbs those spikes. It uses NVIDIA unified memory, which lets the GPU and CPU share an address space and move pages of memory between them automatically, the way an operating system swaps inactive pages of RAM out to disk when memory runs short. QLoRA allocates the optimizer state in this paged memory. The optimizer state sits idle during the forward and backward passes, so when a spike needs room, those pages are evicted to CPU RAM to make space, then paged back in for the optimizer's update step. It is crash insurance, not a footprint reduction: the state still occupies GPU memory when it is resident, but a momentary spike no longer takes the run down. That is what makes single-GPU 33B and 65B finetuning robust rather than a coin flip. (Because only the adapter is trained, the paged state is only the adapter's, which is small.)

It matches 16-bit finetuning

The whole method is worthless if 4-bit weights make the model worse. The central empirical claim is that they do not, once you finetune through them. Two things have to hold: NF4 has to be a genuinely better 4-bit type than the alternatives, and the finetuned 4-bit model has to reach 16-bit quality.

On the first, the ordering is unambiguous. Measuring language-model perplexity (how surprised the model is by held-out text; lower is better) after quantizing the same base model, NF4 with double quantization comes in at 27.41, ahead of both 4-bit float variants (29.48 and 31.07) and well ahead of 4-bit integers (34.34). The number format built for the weight distribution wins, as the rounding errors in Figure 3 already suggested.

On the second, the finetuning recovers whatever quantization lost. The figure tracks 5-shot accuracy on MMLU (a 57-subject knowledge exam; 5-shot means each question is shown with five worked examples in the prompt, no weights changing) as LLaMA grows from 7B to 65B, for three base data types under LoRA. The NF4 line sits on top of the 16-bit line at every size; the 4-bit float line runs about a point below, most visibly at 7B and 33B. Averaged over all the runs, 16-bit and NF4 both land at about 53, statistically the same:

The abstract claims more than the body proves. Against full 16-bit finetuning, where every weight is trained, QLoRA is shown to match only up to about 3B parameters, on RoBERTa and T5, because running full finetuning at 33B or 65B to compare was too expensive. At 7B through 65B, what the figure shows is that 4-bit QLoRA matches 16-bit LoRA, adapters on a full-precision base. Those are the same LoRA recipe with only the base's data type changed, so it isolates the data type, and NF4 passes. The stronger claim, that 4-bit finetuning equals full finetuning at the largest scales, is a reasonable extrapolation the paper does not directly verify.

Guanaco and its benchmarks

With finetuning a 65B model now a single-GPU job, the authors put it to work and trained Guanaco, a family of chatbots. Guanaco is a LLaMA model QLoRA-finetuned with plain cross-entropy (ordinary supervised learning) on 9,209 examples from OpenAssistant, keeping the top-rated reply at each level of each conversation tree. Notably, there is no reinforcement learning from human feedback here, none of the reward-model machinery of InstructGPT or preference-based RL. They left RLHF out on purpose: the paper's own OpenAssistant baseline is an RLHF model trained on the same data, and supervised Guanaco holds its own against it.

Guanaco 65B fits in 41 GB, trains in 24 hours on one professional GPU, and on the Vicuna benchmark (80 prompts, scored by GPT-4 against ChatGPT) reaches 99.3% of ChatGPT's score. Smaller Guanacos scale down cleanly: 33B fits 21 GB and reaches 97.8%, 13B fits 10 GB, and 7B fits 5 GB, small enough for a phone, while still beating a much larger 16-bit Alpaca model by about 20 points. That same memory saving let the authors finetune over 1,000 models to run this study at all.

The 99.3% number carries a caveat the paper itself flags. 99.3% sits under 100%, so on that metric Guanaco lands just below ChatGPT rather than above it. Guanaco does edge ahead of ChatGPT on the GPT-4-judged Elo tournament, where models play head-to-head and win-rates are converted to chess-style ratings, but it falls behind again on the larger OpenAssistant benchmark. Call it competitive within noise, not a decisive win, and note that the confidence intervals are wide.

The evaluation itself comes with warnings the authors raise and quantify. Using GPT-4 as the judge is cheap, but GPT-4 gives a measurable boost to whichever response it reads first, and it favors its own outputs: as a judge it rates itself at Elo 1348, while human raters put it at 1176, roughly a 20% swing in win probability. Human raters agree with each other only moderately (Fleiss $\kappa=0.42$), and agree with GPT-4 case by case even less ($\kappa=0.25$). None of this sinks the results; it means the ranking is real but the exact percentages should be held loosely.

Data quality beats data quantity, decisively, and that finding survives every caveat above. Guanaco's 9,000-example OpenAssistant set produced a better chatbot than a 450,000-example subset of FLAN v2, fifty times larger. And the benchmarks pull apart: FLAN v2 gives the best MMLU exam scores but the worst chatbot scores, so strong performance on a knowledge test tells you little about whether a model is pleasant to talk to.

QLoRA did not invent quantization or low-rank adapters. What it did was combine a frozen 4-bit base, a number format matched to weights, a second squeeze on the scales, and a paging trick, tightly enough that it loses nothing, and then show that the last barrier to finetuning the biggest open models was memory, not some deeper limit. Finetuning a 65B model went from a datacenter job to an overnight run on one card, and the method shipped into the tools everyone uses within months.