QLoRA: Efficient Finetuning of Quantized LLMs

Finetune a 65B model on one 48 GB GPU by storing the frozen model in 4 bits.

The pretrained weights stay frozen in NormalFloat, a 4-bit format fitted to bell-shaped weight distributions, and a small 16-bit LoRA adapter trains through them.

Ordinary 16-bit finetuning of LLaMA 65B needs more than 780 GB of GPU memory, about ten 80 GB cards. QLoRA needs less than 48 GB, and the authors used it to train a 65B chatbot in 24 hours on a single GPU.

A pretrained language model becomes useful for a particular job (following instructions, answering in a house style, chatting) by finetuning: more gradient steps on data for that job. In May 2023, finetuning LLaMA 65B required a multi-GPU server. Quantization, storing weights in fewer bits, already shrank models for inference, but methods such as LLM.int8() and SmoothQuant were built for the forward pass of a finished model and, in the paper's words, "break down during training". The one earlier study of backpropagating through quantized weights beyond 1B parameters, SwitchBack (Wortsman et al., 2023), used 8 bits.

QLoRA (Dettmers, Pagnoni, Holtzman and Zettlemoyer) keeps the pretrained model frozen and stored in 4 bits per weight, and trains a small set of new 16-bit weights, a LoRA adapter, by backpropagating through the 4-bit model. On LLaMA 7B to 65B, the result matches the same adapters trained on a 16-bit model. The memory saving let the authors finetune more than 1,000 models for the study, and train Guanaco, a chatbot family whose 65B member took 24 hours on one GPU.

The page follows the method in the order its parts depend on each other: where finetuning memory goes, what LoRA removes and what it leaves, block-wise quantization, the NormalFloat (NF4) data type, double quantization of the scales, one complete layer forward and backward, paged optimizers, the training recipe, and then the results and the Guanaco evaluation.

Where finetuning memory goes

Finetuning with the Adam optimizer keeps four kinds of numbers in GPU memory. Three of them scale with the number of trained parameters:

• the weight itself, 2 bytes in 16-bit;
• its gradient, another 2 bytes, computed on the backward pass;
• the optimizer state: Adam keeps a running mean and a running mean of squares of each weight's gradients, both in 32-bit, so 8 bytes.

That is 12 bytes per parameter, and for 65 billion parameters it reproduces the paper's number exactly:

The fourth kind, activations, are the intermediate outputs saved on the forward pass for use in the backward pass. They grow with batch size and sequence length rather than parameter count, and the 780 GB does not include them. Of the 12 bytes, 10 exist only because the weight is being updated: the gradient and the two Adam moments outweigh the weight five to one. (Mixed-precision setups that also keep a 32-bit master copy of every weight, the accounting in ZeRO, need 16 bytes per parameter, which would be 1,040 GB here.)

Figure 1 splits the memory three ways for each LLaMA size. Slide to 65B and the full-finetuning bar runs off the chart; the other two bars are the subject of the next section.

What LoRA saves, and what it leaves on the GPU

LoRA (Low-Rank Adaptation, Hu et al., 2021) freezes every pretrained weight matrix $\mathbf{W}$ and learns a correction next to it, factored into two thin matrices. A linear layer that computed $\mathbf{Y}=\mathbf{X}\mathbf{W}$ now computes

with $\mathbf{W}\in\mathbb{R}^{h\times o}$ frozen and only $\mathbf{L}_1\in\mathbb{R}^{h\times r}$ and $\mathbf{L}_2\in\mathbb{R}^{r\times o}$ trained. The rank $r$ is small; QLoRA uses $r=64$. The scalar $s$ is $\alpha/r$, and with QLoRA's $\alpha=16$ it is $16/64=0.25$. By the convention LoRA and its PEFT implementation use, $\mathbf{L}_1$ starts random and $\mathbf{L}_2$ starts at zero, so $\mathbf{L}_1\mathbf{L}_2=0$ at step zero and training starts exactly at the pretrained model.

For a concrete size, take the query projection of LLaMA 7B, a $4096\times4096$ matrix of 16.8 million weights. At $r=64$, $\mathbf{L}_1$ is $4096\times64$ and $\mathbf{L}_2$ is $64\times4096$, together 524,288 numbers, one thirty-second of the matrix. Gradients and Adam moments are needed only for those 524,288. QLoRA puts an adapter on every linear layer of every transformer block (the four attention projections and the three MLP matrices), because the paper finds that 4-bit finetuning matches 16-bit when every linear layer has an adapter, whatever the rank: the original LoRA recipe adapted only the query and value projections, and with adapters on all layers the choice of $r$ "is unrelated to final performance". By our count from LLaMA 65B's layer sizes, adapters on all 80 blocks come to about 800 million parameters, 1.2% of the model. The paper's own memory example uses a smaller adapter: for LLaMA 7B, an adapter sized at the common 0.2% of the model's weights takes only 26 MB, while the input gradients flowing through it take 567 MB.

Freezing removes the gradient and optimizer state for the 65 billion base weights, but the base still has to sit in GPU memory for every forward pass. In 16-bit that is $65\times10^9\times2$ bytes = 130 GB, more than two 48 GB cards, which is the middle bar of Figure 1. QLoRA stores that frozen base in 4 bits instead (about 33.5 GB at 65B, counting the scales) without hurting the finetuned model.

Why 4 bits and not a more cautious 8? An 8-bit base at 65B is 65 GB, still over a 48 GB card. The results section shows that 4-bit NF4 already matches 16-bit after finetuning, so stopping at 8 would cost memory and buy nothing measurable.

How block-wise absmax quantization works

Quantization maps each high-precision number to one of a small set of codes. The scheme everything here builds on is absmax quantization. For signed 8-bit integers with codes $-127$ to $127$, find the largest magnitude in the tensor and scale so that it lands on the top code:

The number $c=127/\text{absmax}$ is the quantization constant, or scale, and it is stored next to the codes. Dequantization divides by it: $\mathbf{X}^{\text{Int8}}/c$. Multiplying and dividing by $c$ cancel, so the rounding is the only step that loses information. Codes run $-127$ to $127$, so one of the 256 byte values ($-128$) goes unused.

A worked example: $\mathbf{X}=[0.62,\,-1.18,\,0.033,\,2.54]$ has absmax 2.54, so $c=50$ and the codes are $[31,\,-59,\,2,\,127]$. Dividing by 50 gives back $[0.62,\,-1.18,\,0.04,\,2.54]$, with an error only on the third value. Now replace 2.54 with an outlier, 25.4. The scale drops to $c=5$, the step between codes grows to 0.2, and the decoded tensor is $[0.6,\,-1.2,\,0,\,25.4]$: the 0.033 has rounded to zero. One large value sets the step size for every other value in the tensor, and in a large tensor most values end up on the few codes nearest zero.

Block-wise quantization limits the damage. Flatten the tensor, cut it into contiguous blocks of $B$ values, and give each block its own scale $c_i$. A tensor of $b\times h$ values becomes $n=(b\times h)/B$ blocks, and an outlier stretches only the scale of its own block. QLoRA uses $B=64$ for the weights. A camera that sets one exposure for the whole frame from a single streetlight renders everything else black; metering each tile of the frame separately confines the problem to the tile that holds the light.

Figure 2 zooms in on ordinary weights near zero and the 16 code positions of a 4-bit grid (the 8-bit case looks the same with more lines). In whole-tensor mode, drag the outlier outward and count how many codes the ordinary weights still reach. Then switch to per-block mode, where the outlier sits in some other block.

These are weight outliers. The more extreme outliers that SmoothQuant and LLM.int8() deal with live in the activations. QLoRA never quantizes activations: it stores only the weights in 4 bits and converts them back to 16-bit before every matrix multiply, so activation outliers never pass through its quantizer.

NormalFloat: a 4-bit type for normally distributed weights

Absmax fixes the scale of the codes but not where the 16 codes of a 4-bit type sit inside $[-1,1]$. Trained weights are roughly normal: crowded near zero with thin tails. The paper checks this on LLaMA 7B with a Shapiro-Wilk test per hidden unit and finds 7.5% of units non-normal at a 5% significance level, about 2.5 points above the false-positive rate. A uniform grid (Int4) puts as many codes in the sparse tails as in the dense center, so a few center codes absorb most of the weights.

NormalFloat (NF4) places the codes at quantiles of a standard normal: cut the bell into 16 slices of equal probability mass rather than equal width, and put one code in each. Near zero the slices are narrow and the codes close together; in the tails the slices are wide. NF4 specializes the block-wise quantile quantization of Dettmers et al. (2022) to a fixed normal, so the codes are computed once and every block reuses them. The paper writes each code as the midpoint of two adjacent quantiles:

where $Q$ is the inverse CDF (quantile function) of $\mathcal{N}(0,1)$ and $k=4$ bits. Figure 3 compares where NF4, Int4 and FP4 (a 4-bit float with 2 exponent bits and 1 mantissa bit) put their codes on the same bell. The shading of each bin shows how much of the weight mass that one code has to represent. Move the pointer across the axis to drop a weight and read its rounding error.

On this bell (standard deviation 0.36 after normalization, so the block maximum sits about 2.8 standard deviations out) the RMS rounding errors are 0.0343 for NF4, 0.0385 for Int4 and 0.0393 for FP4.

The paper calls NF4 information-theoretically optimal for normal data. The phrase has a narrow meaning: with equal-mass bins, every one of the 16 codes is used equally often, so a stored code carries the full $\log_2 16 = 4$ bits of information. That maximizes the entropy of the codes. Minimum rounding error is a different objective, and the quantizer that achieves it (Lloyd-Max) puts each code at the mean of its bin and does not make the bins equal-mass.

Two details of the type. It is asymmetric: 7 negative codes, an exact $0$, and 8 positive codes, 16 in all, with the exact zero there so that padding and true zeros are stored without error. And the bins are equal-mass only in the idealization. Each block is divided by its own maximum, so the normalized values are not exactly $\mathcal{N}(0,1)$ scaled to $[-1,1]$, and Yoshida (2023) measured per-code usage between 2% and 9% at block size 64 instead of a uniform 6.25%. Yoshida argues this departure helps, because it keeps more precision for the rare large weights.

The 16 values themselves do not come from equation (3) as written: its first and last terms are $Q(0)=-\infty$ and $Q(1)=+\infty$. The released code takes quantiles between 0.5 and an offset of 0.9677083 instead, and rescales so the extreme codes are exactly $\pm1$. This reproduces the published table to about seven decimal places:

# the 16 NF4 values (bitsandbytes create_normal_map, functional.py:169)
# equation (3) taken literally asks for Q(0) = -inf and Q(1) = +inf
offset = 0.9677083                              # = ((1-1/30) + (1-1/32)) / 2
pos = norm.ppf(linspace(offset, 0.5, 9)[:-1])   # 8 positive quantiles
neg = -norm.ppf(linspace(offset, 0.5, 8)[:-1])  # 7 negative quantiles
codes = sort([*neg, 0.0, *pos])                 # 7 neg + exact 0 + 8 pos = 16
codes = codes / max(abs(codes))                 # absmax-normalize: ends at -1, +1

The result runs −1.0, −0.6962, …, −0.0911, 0, 0.0796, …, 0.7230, 1.0. A stored weight is a 4-bit index into this table, and decoding multiplies the table entry by the block's scale.

One block, encoded and decoded

Encoding a block of 64 weights takes four steps. (1) Compute the block's absmax and keep it as the scale. (2) Divide every weight by it, which puts the block in $[-1,1]$. (3) Replace each normalized value by the index of the nearest NF4 value; bitsandbytes orders the table ascending, so index 0 is $-1$, index 7 is the exact zero and index 15 is $+1$. (4) To decode, look up the table entry and multiply by the scale. Storage for the block is 64 four-bit indices, which is 32 bytes, plus the scale.

Figure 4 does this for a block drawn from a normal with standard deviation 0.02, about the spread of LLaMA's weights. Each teal bar is an original weight and the amber tick on it is its decoded value. Pick a weight with the slider or by tapping its bar and follow it through the four steps; "new block" draws another 64.

Two things to read off the figure. The weight with the largest magnitude always decodes exactly, because it normalizes to $\pm1$ and both are codes. And the RMS error stays a few percent of the block's absmax whatever block you draw, because the codes move with each block's own scale.

Double quantization

Each block of 64 weights carries its own 32-bit scale, which adds $32/64=0.5$ bits to every weight, so a "4-bit" model actually costs 4.5 bits per weight. Smaller blocks track outliers better but pay this overhead more often; larger blocks are cheaper and let an outlier degrade more weights.

Double Quantization quantizes the scales. LLaMA 65B has $65\times10^9/64$, about 1.02 billion, block scales, and those numbers form a tensor of their own. QLoRA stores each one as an 8-bit float, in blocks of 256 scales, with one 32-bit scale per block of 256. The scales are all positive, so their mean is subtracted first to center them around zero and allow a symmetric quantizer. The weights themselves stay 4-bit NF4; only the scales change. The overhead per weight becomes

bits per parameter, down from 0.5, a saving of 0.373 bits on every weight. At 65B that is $0.373\times65\times10^9/8\approx3$ GB (dividing by 8 converts bits to bytes). Counted directly: the 1.02 billion scales take 4.06 GB at 32 bits, and 1.02 GB at 8 bits plus 16 MB for the 4 million second-level scales. The total storage drops from about 4.5 to about 4.127 bits per weight, which is the 33.5 GB base in Figure 1. Figure 5 plots both overheads against block size.

At $B=32$ plain scales would cost a full extra bit per weight and double quantization brings that to 0.254; at $B=4096$ both are under 0.01 bits but each scale covers 4,096 weights, so a single outlier coarsens all of them. At $B=64$, double quantization cuts the scale overhead from 0.5 to 0.127 bits per weight. The paper reports no performance degradation from 8-bit scales in blocks of 256.

How QLoRA works: one layer, forward and backward

QLoRA uses two data types. The storage type, 4-bit NF4, holds the frozen weights in GPU memory. The compute type, 16-bit BFloat16 (BF16), is what every matrix multiply runs in. When a layer needs its weight, the weight is dequantized from 4-bit to BF16, used, and discarded; no 16-bit copy of the base is ever stored. Writing equation (1) with that conversion made explicit gives the forward pass of one linear layer:

The scalar $s$ is the LoRA scale from (1). The paper prints this equation without it, but its own equation for LoRA has it and the released code applies $\alpha/r$. The function $\operatorname{doubleDequant}$ undoes both rounds of quantization:

Read it from the inside. The inner $\operatorname{dequant}(c_1, c_2)$ uses the 32-bit second-level scales $c_1$ to recover the block scales from their 8-bit form $c_2$. The outer $\operatorname{dequant}$ then uses those recovered scales to turn each 4-bit index into a BF16 weight, as in step 4 of Figure 4. The subscripts run opposite to the order of quantization: $c_2$ holds the first-level scales, one per 64 weights, and $c_1$ holds the second-level scales, one per 256 of those. The two nested dequants are the two rounds of double quantization; the weight is dequantized once.

The backward pass has two jobs at each layer. It computes the gradients for this layer's adapter, $\partial E/\partial \mathbf{L}_1$ and $\partial E/\partial \mathbf{L}_2$, which the optimizer uses to update them. And it computes the gradient with respect to the layer's input, $\partial E/\partial \mathbf{X}$, which the layer below needs for its own adapter. That input gradient is $\partial E/\partial \mathbf{Y}$ multiplied by $\mathbf{W}^\top$ (plus the adapter's term), so the 4-bit weight is dequantized again on the backward pass. The weight gradient $\partial E/\partial\mathbf{W}$ is never formed, because $\mathbf{W}$ is never updated. This is also why no straight-through estimator or other trick for differentiating through rounding is needed: the rounding was applied once, when the model was loaded, and no later computation needs its derivative.

Figure 6 shows the two paths. Toggle to the backward pass to see the gradient update the adapter and pass through, but not into, the frozen weight.

Continuing the LLaMA 7B example: the $4096\times4096$ query projection is 33.6 MB in 16-bit and 8.7 MB in NF4 with double quantization, with 262,144 block scales (one per 64 weights). Its rank-64 adapter is 1 MB in BF16 and is the only part of the layer with a gradient and optimizer state. In code, with the gradient written out by hand:

# one QLoRA linear layer.  storage: 4-bit NF4.  compute: bf16.
def forward(x, W_nf4, c1, c2, L1, L2, s):     # x, L1, L2 are bf16
    W = double_dequant(c1, c2, W_nf4)         # 4-bit -> bf16, on the fly
    return x @ W + s * (x @ L1) @ L2           # frozen base + adapter

def backward(dy, x, W_nf4, c1, c2, L1, L2, s):
    W = double_dequant(c1, c2, W_nf4)         # dequantize again for dx
    dx  = dy @ W.T + s * (dy @ L2.T) @ L1.T    # passes THROUGH frozen W
    dL1 = s * x.T @ (dy @ L2.T)                # only the adapter gets
    dL2 = s * (x @ L1).T @ dy                  # gradients; dW is never formed
    return dx, dL1, dL2

double_dequant is called in both forward and backward, and backward returns no dW, so per training step the base is read twice and never written.

Paged optimizers

With the base at 33.5 GB and the adapter's weights, gradients and Adam state at about 9.6 GB (our estimate for 800 million adapter parameters at 12 bytes each), a 65B QLoRA run on a 48 GB card has roughly 5 GB left for activations. That is enough on average, because gradient checkpointing keeps only a few saved activations on the forward pass and recomputes the rest during the backward pass. The recomputation comes in bursts, though, and a mini-batch with a long sequence produces a large burst. One such burst past 48 GB ends the run with an out-of-memory error, even if every other step fits.

Paged optimizers handle the bursts with NVIDIA unified memory, which lets the GPU and CPU share an address space and moves pages between them on demand, as an operating system swaps RAM pages to disk. QLoRA allocates the optimizer state in this paged memory. Adam's moments are not touched during the forward and backward passes, so when the GPU runs out, those pages are evicted to CPU RAM; they are paged back in for the optimizer update step, by which time the activations have been freed. The footprint during normal steps is unchanged; during a burst, optimizer pages move to the CPU instead of the run crashing.

Figure 7 runs eight training steps on a 48 GB card with the sizes above. Step 6 holds a long sequence. Play it with the regular optimizer, then with the paged one, and watch the CPU lane under the axis during step 6.

The paper does not measure paging overhead in detail, because paging happens only on rare long-sequence mini-batches; for 65B models on 48 GB GPUs at batch size 16 it found the same training speed as with regular optimizers. It states that paged optimizers are needed for 33B on a single 24 GB GPU and 65B on a single 48 GB GPU; its memory appendix notes that the 33B model "does not quite fit" into 24 GB without them.

The training recipe

The paper's settings (App. B.2) are the same for every dataset apart from batch size and step count: NF4 with double quantization, BF16 compute, LoRA $r=64$ and $\alpha=16$ on all linear layers, LoRA dropout 0.1 up to 13B and 0.05 for 33B and 65B, Adam with $\beta_2=0.999$, gradient norm clipped at 0.3, and a constant learning rate of $2\times10^{-4}$ for 7B and 13B and $1\times10^{-4}$ for 33B and 65B. Group-by-length batching puts examples of similar length together, which reduces padding and makes the loss curve oscillate. The Guanaco runs on OASST1 used batch size 16 for 1,875 steps at every model size. In the Hugging Face stack, which ships bitsandbytes and PEFT support for all three pieces, the recipe is a few configuration objects:

# QLoRA with transformers + peft + bitsandbytes (paper App. B.2 settings, 65B)
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",               # NormalFloat, not FP4
    bnb_4bit_use_double_quant=True,          # quantize the block scales too
    bnb_4bit_compute_dtype=torch.bfloat16,   # dequantize to bf16 for matmuls
)
model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb)
model = prepare_model_for_kbit_training(model)   # turns on grad checkpointing
lora = LoraConfig(r=64, lora_alpha=16, lora_dropout=0.05,   # 0.1 up to 13B
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                  "gate_proj", "up_proj", "down_proj"])
model = get_peft_model(model, lora)              # s = 16 / 64 = 0.25
args = TrainingArguments(optim="paged_adamw_32bit",   # paged optimizer
                         learning_rate=1e-4,          # 2e-4 for 7B and 13B
                         lr_scheduler_type="constant", max_grad_norm=0.3,
                         adam_beta2=0.999, group_by_length=True, bf16=True)

How 4-bit finetuning compares with 16-bit

Two claims need to hold: NF4 has to be a better 4-bit type than the alternatives, and a model finetuned on a 4-bit base has to reach 16-bit quality.

For the first, the paper quantizes pretrained OPT, BLOOM, LLaMA and Pythia models from 125M to 13B parameters and measures perplexity on Pile Common Crawl, a measure of how well the model predicts held-out text where lower is better. Averaged over models: Int4 34.34, FP4 with 2 exponent bits (E2M1) 31.07, FP4 with 3 exponent bits (E3M0) 29.48, and NF4 with double quantization 27.41. NF4 comes out best here, as it did in Figure 3. Int4 and FP4 swap places, though: on Figure 3's bell Int4 rounds slightly more accurately than FP4, but in the full models both FP4 variants give lower perplexity than Int4.

For the second, Figure 8 plots 5-shot accuracy on MMLU, a multiple-choice exam over 57 subjects where "5-shot" means five solved examples are placed in the prompt, for LLaMA 7B to 65B finetuned with LoRA on Alpaca and on FLAN v2. Only the data type of the frozen base differs between the three lines: BF16, FP4, or NF4 with double quantization, with identical adapters.

Over all eight runs the means are 53.0 for BF16, 53.1 for NF4 with double quantization and 52.2 for FP4. The paper summarizes FP4 as about one percentage point behind and NF4 as fully recovering 16-bit LoRA. The FP4 gap is largest at 7B (1.2 points on Alpaca, 1.6 on FLAN v2) and 33B (1.8 and 2.0); at 65B on FLAN v2 FP4 scores 63.3 against BF16's 62.5.

The comparison in Figure 8 is against 16-bit LoRA, not against full finetuning. Full 16-bit finetuning, where every weight is trained, was replicated only on RoBERTa-large (GLUE) and on T5 models from 80M to 3B parameters (Super-NaturalInstructions); there, 16-bit, 8-bit and 4-bit adapters all matched it. Full finetuning at 11B and above needs more than one server of GPUs, so at 7B to 65B the paper tests 4-bit QLoRA against 16-bit LoRA, and its limitations section states that it did not establish parity with full 16-bit finetuning at 33B and 65B. The abstract's "while preserving full 16-bit finetuning task performance" is stronger than what the body shows at those sizes.

Guanaco and the chatbot evaluation

Guanaco is LLaMA finetuned with QLoRA on OASST1, the OpenAssistant conversation dataset, keeping only the top-ranked reply at each level of each conversation tree: 9,209 examples. Training is ordinary supervised learning with a cross-entropy loss. There is no reinforcement learning from human feedback (the reward-model stage of InstructGPT and of learning from human preferences). The paper includes an RLHF model for comparison: Open Assistant, a LLaMA 33B trained with RLHF on the same OASST1 data.

The main benchmark is the Vicuna set of 80 prompts. GPT-4 scores ChatGPT's answer and the model's answer out of ten each, and the model's score is reported as a percentage of ChatGPT's, averaged over both orders of presentation. Table 6 of the paper:

• Guanaco 65B, 41 GB of memory: 99.3% (±4.4), the best open model and second only to GPT-4;
• Guanaco 33B, 21 GB: 97.8%, three points above Vicuna 13B (94.9% at 26 GB in 16-bit);
• Guanaco 13B, 10 GB: 90.4%;
• Guanaco 7B, 5 GB, small enough for a phone: 87.0%, against 69.4% for Alpaca 13B.

99.3% is below ChatGPT, since the relative score can exceed 100% when a model outscores ChatGPT. In the Elo tournament judged by GPT-4 (models play pairwise matches and win rates become chess-style ratings), Guanaco 65B is ahead of ChatGPT on the Vicuna prompts, 1022 to 966. On the Open Assistant (OA) benchmark, 953 prompts built from every user message in the OASST1 validation conversations, it is behind, 1008 to 1015. The confidence intervals are wide and many models overlap, so "close to ChatGPT" describes the result better than "beats ChatGPT".

The paper also measures the evaluation itself. GPT-4 favors whichever response appears first in the prompt: for Guanaco 65B the score is 96.7% with ChatGPT's answer first and 101.9% with Guanaco's first. It favors its own outputs, rating GPT-4 at Elo 1348 where human raters give 1176, about 20% more win probability. Human raters agree with each other moderately (Fleiss $\kappa=0.42$), and GPT-4 agrees with the human majority less at the level of single examples ($\kappa=0.25$). At the level of whole systems the rankings agree better: Kendall $\tau=0.43$, Spearman 0.55.

On data, a 9k-example dataset (OASST1) produced a better chatbot than a 450k-example subsample of FLAN v2. FLAN v2 gave the best MMLU scores and the worst Vicuna scores: Guanaco 65B reaches 99.3% where the FLAN v2 65B finetune reaches 48.4%. The paper concludes that strong MMLU performance does not imply strong Vicuna performance, or the reverse.