LoRA: Low-Rank Adaptation of Large Language Models

Fine-tune a large model by freezing its weights and training a low-rank update beside them.

On GPT-3 175B, LoRA trains about 10,000 times fewer parameters than full fine-tuning, needs about a third of the GPU memory, matches its accuracy, and folds into the weights after training so the model runs at its original speed.

The paper's hypothesis is that the change fine-tuning makes to a weight matrix spans only a few directions. It builds the method on that hypothesis and tests it on models up to GPT-3 175B.

LoRA (Low-Rank Adaptation) is a way to specialize a pretrained language model to a task, such as SQL generation or dialogue summarization, without retraining all of its weights. It came out of Microsoft in 2021 and is now one of the most widely used ways to fine-tune large models on modest hardware. The page builds it from a few ideas, in this order: what full fine-tuning keeps in memory, the fact that fine-tuning learns a change to each weight matrix, what the rank of a matrix measures, the hypothesis that the change has low rank, the layer that follows from it, how that layer is initialized and scaled, why it costs nothing at inference, and what the experiments show.

What full fine-tuning costs

Full fine-tuning continues training every weight of the pretrained model on task data. For GPT-3, that is 175 billion weights. Stored as 16-bit floats (2 bytes each), the weights alone take 350 GB. Training needs more than the weights. Every trained weight gets a gradient of the same size, and the Adam optimizer the paper uses keeps two running averages per weight, a mean of the gradient and a mean of its square. So a trained weight costs roughly four times its own storage before a single activation is counted.

The paper measures this on GPT-3 175B: full fine-tuning with Adam needs 1.2 TB of GPU memory; LoRA needs 350 GB. The difference comes almost entirely from optimizer state and gradients that LoRA never creates for the frozen weights. Each fine-tuned copy is also another 350 GB checkpoint on disk, so serving 100 tasks as 100 full fine-tunes means about 35 TB of weights. With LoRA the paper puts the same 100 tasks at about 354 GB: one shared base model plus 100 adapters of about 35 MB each.

Figure 1 draws the training-memory bill in units of the weight memory. Slide the model size and watch the ratio of trainable parameters in the corner.

At 175B the readout says about 9,300× fewer trainable parameters: 175 billion against 18.9 million. The paper rounds this to 10,000×. At 125M parameters the same recipe gives about 850×: the adapter shrinks with the width d, but the full matrices shrink with d², so the ratio grows with the width.

Fine-tuning learns an update

A linear layer multiplies its input by a weight matrix. Write the pretrained matrix as $W_0 \in \mathbb{R}^{d\times k}$: it takes a $k$-dimensional input vector $x$ and returns a $d$-dimensional output $h = W_0 x$. Fine-tuning changes the matrix to $W_0 + \Delta W$, so the layer computes

Everything fine-tuning learns is in $\Delta W$. It has the same shape as $W_0$, so a model with 175 billion weights has an update with 175 billion entries, each with its own gradient and optimizer state. The question LoRA asks is whether $\Delta W$ needs that many independent numbers, or whether a much smaller set of numbers can generate it.

Rank and low-rank factors

The rank of a matrix is the number of linearly independent columns it has; equivalently, the number of independent rows. A small example shows what that means for storage. This 3×4 matrix has 12 entries:

Every row is a multiple of $[1,2,3,4]$, so the rank is 1 and the matrix is the product of a 3×1 column and a 1×4 row: 7 numbers instead of 12. A rank-2 matrix is a sum of two such products, which is the product of a 3×2 matrix and a 2×4 matrix. In general, any $d\times k$ matrix of rank $r$ can be written as

and storing $B$ and $A$ costs $r(d+k)$ numbers instead of $dk$. For one GPT-3 attention matrix, $d = k = 12288$, so the full matrix has 150,994,944 entries. At $r = 8$ the factors hold $8 \times (12288 + 12288) = 196{,}608$ numbers, 768 times fewer. That ratio is for a single matrix; the whole-model 10,000× figure is larger because LoRA leaves most matrices without an adapter at all.

Real matrices are rarely exactly low-rank, but many are close to it. The singular value decomposition (SVD) writes any matrix as a sum of rank-1 pieces, sorted by size:

where the $u_i$ are orthonormal output directions, the $v_i$ orthonormal input directions, and the singular value $\sigma_i$ says how much the matrix stretches along that pair. Keeping the first $r$ terms gives the closest rank-$r$ matrix in the Frobenius norm (the square root of the sum of squared entries), a result known as the Eckart–Young theorem, and the squared error is the sum of the dropped $\sigma_i^2$. When the singular values fall off fast, a few terms carry almost all of the matrix. Figure 2 rebuilds a 28×28 matrix whose singular values shrink by a factor of 0.62 per term; drag the rank and read the energy kept (the fraction of the total $\textstyle\sum \sigma_i^2$) under the panels.

Figure 3 plots the two costs of the choice against each other for the same matrix. The number of stored values grows linearly with $r$, one row of $A$ and one column of $B$ per step. The reconstruction error falls quickly at first and then flattens, because each added term carries less than the one before it.

The low-rank hypothesis

The pretrained $W_0$ is typically full-rank, and LoRA does not compress it. The paper's hypothesis concerns only the update: that $\Delta W$ has a low “intrinsic rank”, meaning a rank-$r$ matrix with small $r$ can do the job of the full update during adaptation. It is a hypothesis, not a theorem, and the paper tests it empirically in its Section 7.

It comes from earlier work on intrinsic dimension. Li et al. (2018) trained networks inside a random low-dimensional subspace of their parameters and measured how small the subspace could be before accuracy dropped. Aghajanyan et al. (2020) did the same for fine-tuning pretrained language models: optimizing only 200 parameters, projected back into the full parameter space by a fixed random matrix, brought RoBERTa to 90% of its full fine-tuning performance on the MRPC paraphrase task. If a few hundred numbers can steer a whole model, the per-matrix update plausibly lives in a few directions too.

A reading of why this could hold: pretraining builds general features and uses the full width of each matrix to do it. Adapting to one task mostly changes how strongly some of those existing features are used, and a change concentrated in a handful of input and output directions is, by definition, a low-rank matrix. The paper's Section 7.3 measures something close to this on GPT-3, reported at the end of the page.

Why not use the obvious alternatives? Training only a subset of existing weights (the last layers, or the biases) restricts which weights may change, but LoRA restricts the shape of the change and still lets every adapted matrix move. Random-subspace training, the Aghajanyan method, needs a projection from the subspace to all parameters on every step. Adapters and prefix tuning, the two parameter-efficient methods the paper compares against, each carry a cost at use time: adapters add sequential layers, and prefix tuning reserves part of the input sequence for learned tokens and is hard to optimize.

The LoRA layer

LoRA builds the hypothesis into the layer. It freezes $W_0$, adds a second path next to it that computes $BAx$, and sums the two outputs:

$A$ projects the $k$-dimensional input down to $r$ numbers, and $B$ projects those back up to $d$. The paper's own Eq. (3) omits the gain $\alpha/r$ and introduces it in the next paragraph; (3) above includes it, as the code does. Only $A$ and $B$ are trained. The next section covers the initialization and the gain.

The listing below runs one training step through GPT-3's query projection with $r = 4$, for a batch of $n$ token vectors. The frozen path is an ordinary matmul. The LoRA path first shrinks each token to 4 numbers, then expands it back to 12,288, and adds that to the frozen output. The extra compute is $r(d+k)/(dk)$, about 0.07% of the layer's matmul.

# one training step through one adapted layer (GPT-3 W_q, r = 4)
# x: (n, 12288), n token vectors entering the layer
h = x @ W0.T                  # (n, 12288)  W0 frozen: no grad, no Adam state
z = dropout(x) @ A.T          # (n, 4)      A: (4, 12288), trainable
h = h + (alpha / r) * (z @ B.T)   # (n, 12288)  B: (12288, 4), trainable
loss.backward()   # gradients for A and B only: 4 * (12288 + 12288) = 98,304
opt.step()        # Adam keeps m and v for those 98,304 numbers, nothing else

Figure 4 lays the product $BA$ out like a multiplication table. Every entry of $\Delta W$ is the dot product of one row of $B$ ($r$ numbers) with one column of $A$ ($r$ numbers). Hover a cell to see which row and column produce it, and set $r = 1$ to see a single outer product: every row of $\Delta W$ is then a scaled copy of the same pattern, as in the 3×4 example above.

Figure 5 shows where the path sits in the layer. The same input $x$ goes through the frozen $W_0$ and through $A$ then $B$, and the two outputs are added. Its other two buttons belong to the merging section below.

The corresponding lines of the authors' loralib, abridged:

# loralib/layers.py, class Linear (abridged; line numbers from main)
self.lora_A = nn.Parameter(self.weight.new_zeros((r, in_features)))
self.lora_B = nn.Parameter(self.weight.new_zeros((out_features, r)))
self.scaling = self.lora_alpha / self.r                      # line 112
self.weight.requires_grad = False                            # line 114

nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))        # line 124
nn.init.zeros_(self.lora_B)                                  # line 125

def forward(self, x):                      # while not merged
    result = F.linear(x, T(self.weight), bias=self.bias)
    result += (self.lora_dropout(x) @ self.lora_A.transpose(0, 1)
               @ self.lora_B.transpose(0, 1)) * self.scaling  # line 149
    return result

# model.eval() merges once:                                  # line 141
#   self.weight.data += T(self.lora_B @ self.lora_A) * self.scaling
# model.train() subtracts the same product again (line 135).

The code fixes three details the paper text leaves open. $A$ is initialized with Kaiming-uniform values (uniform in $\pm 1/\sqrt{k}$ for this setting), where the paper says Gaussian. The gain is exactly lora_alpha / r. And dropout is applied to the input of the LoRA path only, never to the frozen path; the paper text does not mention it.

Initialization and the α/r gain

$B$ starts at zero, so $BA = 0$ and the model at step 0 is exactly the pretrained model. The first step still changes the model, because only one of the two factors is zero. With $g$ the gradient of the loss with respect to the layer output $h$, and $s = \alpha/r$, the gradients of the two factors are

At step 0, $A$ is random, so the gradient on $B$ is nonzero and $B$ moves. The gradient on $A$ contains $B$ and is exactly zero, so $A$ starts moving on the second step. If both factors started at zero, both gradients would be zero forever and nothing would train. Zeroing $A$ and randomizing $B$ would also work, by the same argument with the roles swapped.

The gain $\alpha/r$ handles a different problem: how the size of each step depends on $r$. Adam moves every parameter by roughly the learning rate per step, whatever the size of its gradient. Each entry of $BA$ is a sum of $r$ products, so with more columns in $B$ the same learning rate changes $BA$ by more per step. Dividing by $r$ cancels that growth. The paper sets $\alpha$ to the first $r$ it tries and never tunes it, and says the scaling reduces the need to retune hyperparameters when $r$ changes. It also notes that, under Adam, tuning $\alpha$ is roughly the same as tuning the learning rate.

Figure 6 trains a LoRA adapter on a toy problem to show both effects. The target $\Delta W^*$ is a 16×16 matrix with eight directions of decreasing size, and each run fits $(\alpha/r)BA$ to it with Adam (learning rate 0.02), $B = 0$, $A$ uniform as in loralib, and $\alpha = 4$, so the gain is 1 at $r = 4$. Press Play, then compare ranks and switch the gain off.

Every run starts at error 1.000, because the adapter's product is zero. The rank-1 run stalls at 0.313, the rank-2 run at 0.094 and the rank-4 run at 0.009. Those are the Eckart–Young floors for this target: the fraction of its squared norm outside its best 1, 2 or 4 directions, so no matrix of that rank can do better. Ranks 8 and 16 reach zero, because the target has eight directions; the extra capacity at $r = 16$ buys nothing. With the gain on, ranks 2 through 16 sit between 0.49 and 0.56 at step 10. With it off, the same learning rate leaves rank 1 at 0.92 and takes rank 16 to 0.13 by step 10: raising $r$ acts like raising the learning rate, and a learning rate tuned at one rank would be wrong at another.

Merging: no inference latency

The LoRA path is linear and runs in parallel with $W_0$, so after training it can be added into the weight matrix once:

$W$ has the same shape as $W_0$. The deployed model is an ordinary model with ordinary weights, and it runs one matmul per layer, the same as the base model, so LoRA adds no inference latency by construction. In loralib this happens when you call model.eval(), and model.train() subtracts the product again. Switching a deployed model to another task works the same way: subtract $BA$, add a different $B'A'$. Click LoRA, merged in Figure 5 to see the single path.

Adapters (Houlsby et al., 2019), the method LoRA is mainly compared against, insert small bottleneck modules after existing layers: project down to a few dimensions, apply a nonlinearity, project back up, add a skip connection. They also train few parameters, but the nonlinearity means they cannot be folded into the weights, and each one is an extra step that must run after the layer before it. Click adapter (2019) in Figure 5 to see the serial path. The paper measures GPT-2 medium on one GPU (its Table 1). At batch size 32 and sequence length 512, adapters add 2.2% to 3.0% to a 1,449 ms forward pass. At batch size 1 and sequence length 128, the setting of online serving, they add 20.7% to 30.3% to a 19.8 ms pass. The paper attributes this to the adapter layers running sequentially, which the GPU's parallelism cannot hide when the batch is small.

A merged model serves one task. The paper notes that once the adapters are absorbed into the weights, batching requests for different tasks (each with its own $A$ and $B$) through one forward pass is no longer straightforward. A multi-task deployment keeps the adapters unmerged and pays for the small extra path on every request.

Which matrices, which rank

A Transformer layer has four weight matrices in self-attention, the query, key, value and output projections $W_q, W_k, W_v, W_o$, each $d_{\text{model}}\times d_{\text{model}}$, and two in the MLP. The paper adapts only attention matrices and freezes the MLP, for simplicity and parameter efficiency; it says LoRA could go on any subset of weight matrices. GPT-3 175B has 96 layers and $d_{\text{model}} = 12288$. The standard setting of the paper, $r = 4$ on $W_q$ and $W_v$ in every layer, trains

which is about 36 MiB at 2 bytes each. The paper calls it roughly 35 MB and compares it with the 350 GB checkpoint of a full fine-tune: the roughly 10,000× smaller checkpoint in the abstract.

With a fixed budget of about 18M parameters, the paper tried seven ways to spend it (Table 5): all of it on one matrix at $r = 8$, on two matrices at $r = 4$, or on all four at $r = 2$. Putting it all on $W_q$ or $W_k$ scores clearly lower on WikiSQL, a benchmark of turning English questions into SQL queries (70.4 and 70.0), than splitting it over $W_q$ and $W_v$ (73.7). The paper concludes that a budget is better spread over more matrices at a lower rank, though $W_o$ alone at $r = 8$ (73.2) does beat $W_q$ and $W_k$ together at $r = 4$ (71.4). Table 6 then sweeps the rank for three choices of matrices. For $\{W_q, W_v\}$, WikiSQL accuracy runs 73.4, 73.3, 73.7, 73.8 and 73.5 at $r$ = 1, 2, 4, 8 and 64: flat within the ±0.5 spread the paper reports between runs. $W_q$ alone needs a larger rank and still scores lower. Figure 7 plots both tables.

Rank 1 on $W_q$ and $W_v$ already reaches 73.4 on WikiSQL and 91.3 on MultiNLI, a benchmark of deciding whether one sentence implies, contradicts, or is neutral to another. The paper checks why with a subspace comparison: it trains adapters at $r = 8$ and $r = 64$, takes the singular vectors of each learned $A$, and measures how much the top directions of one span the top directions of the other, on a scale from 0 (no overlap) to 1 (the same subspace). The top direction is shared, with similarity above 0.5; the remaining directions of the $r = 64$ adapter overlap little with the $r = 8$ one, and the paper reads them as mostly noise accumulated during training. That result is specific to these tasks. The paper warns that a task far from the pretraining data, such as one in a different language, may need a much larger $r$, up to retraining the whole matrix.

Results and limits

On GPT-3 175B (Table 4), full fine-tuning trains 175,255.8M parameters and scores 73.8 on WikiSQL, 89.5 on MultiNLI-matched, and 52.0/28.0/44.5 ROUGE-1/2/L on SAMSum dialogue summarization (ROUGE counts word and word-pair overlap with reference summaries). LoRA with 4.7M trainable parameters scores 73.4, 91.7 and 53.8/29.8/45.9; LoRA with 37.7M scores 74.0, 91.6 and 53.4/29.2/45.1. The 4.7M configuration is $r_q = r_v = 1$ (or $r_v = 2$ alone), and the 37.7M one is $r_q = r_v = 8$ or $r = 4$ on all four attention matrices. LoRA matches or beats full fine-tuning on these three tasks, with the 4.7M version 0.4 points behind on WikiSQL, inside the ±0.5 run-to-run spread.

On RoBERTa, DeBERTa and GPT-2 the comparison comes out the same way as on GPT-3: LoRA on par with or better than full fine-tuning. On GLUE, a suite of eight language-understanding tasks, RoBERTa-base with 0.3M LoRA parameters averages 87.2 against 86.4 for full fine-tuning of all 125M; DeBERTa XXL with 4.7M averages 91.3 against 91.1 for all 1.5B. On E2E, which asks for a sentence describing a restaurant from a list of its attributes, GPT-2 medium with 0.35M LoRA parameters reaches 70.4 BLEU (n-gram overlap with reference sentences) against 68.2 for full fine-tuning.

Training is also faster. On GPT-3 175B the paper measures 43.1 tokens per second per V100 GPU with LoRA against 32.5 for full fine-tuning, which it reports as a 25% speedup: the backward pass skips the weight gradients of every frozen matrix. Memory drops from 1.2 TB to 350 GB, the “3×” of the abstract, and the paper puts the general saving at up to two-thirds of training VRAM when $r \ll d_{\text{model}}$.

Section 7.3 of the paper looks at what the learned update does. For $W_q$ at $r = 4$, the update has Frobenius norm 6.91, against 61.95 for $W_q$ itself. Projecting $W_q$ onto the four directions of the update gives a norm of only 0.32, compared with 21.67 on $W_q$'s own top four directions and 0.02 on four random directions. So the update points at directions that already exist in the pretrained matrix (0.32 is well above random) but that the matrix barely uses (0.32 is far below 21.67), and it scales them up by about 6.91 / 0.32 ≈ 21.5. The paper's reading is that LoRA amplifies features learned in pretraining that matter for the task but were not emphasized.

You still choose which matrices to adapt and at what rank, and the paper's guidance comes from a few tasks on a few models. A merged model serves one task. The frozen base stays fully in memory, so LoRA reduces the training overhead and the per-task storage, never the 350 GB of base weights. Activations for the backward pass are still stored, because the gradient has to pass through every frozen layer to reach the adapters below it. The most-used follow-up, QLoRA, shrinks the base: it stores the frozen weights in 4 bits and trains LoRA adapters on every linear layer on top of them.