LLaMA: Open and Efficient Foundation Language Models

Train a small model far too long, so it is cheap to run.

LLaMA (Meta AI, 2023) is a family of language models from 7B to 65B parameters, trained only on public data. It picks model size by what the model will cost to serve, then trains each size on more text than a compute-optimal recipe would use.

LLaMA-13B, trained on a trillion tokens, scores above the 175B GPT-3 on most benchmarks and runs on one GPU. This page works through the cost arithmetic behind that choice, the 1.4T-token public dataset, the three architecture changes LLaMA borrowed (RMSNorm, SwiGLU, rotary embeddings), how the 65B was trained in 21 days, and what the results do and do not show.

By 2022 the standard way to size a language model came from a scaling law. Chinchilla (Hoffmann et al., 2022) trained more than 400 models and asked: for a fixed amount of training compute, what split between model size and training data gives the lowest loss? The answer was to grow both together, about 20 training tokens for every parameter. By that rule GPT-3, with 175B parameters trained on 300B tokens, had far too little data for its size.

LLaMA starts from that result and changes the question. Chinchilla prices only training. A model that gets deployed is trained once and then run on every request for as long as it is in use, and the per-request cost depends on the parameter count alone. The LLaMA introduction puts it this way: "given a target level of performance, the preferred model is not the fastest to train but the fastest at inference." So the team trained models of fixed sizes (7B, 13B, 33B and 65B) on 1.0T to 1.4T tokens, far past the 20-per-parameter point for the smaller ones. The architecture and training setup are borrowed from earlier models; the paper documents them closely enough to reproduce.

Training cost versus serving cost

Compute for a Transformer can be counted in floating-point operations (FLOPs) with two rules of thumb. A forward pass costs about $2N$ FLOPs per token for a model with $N$ parameters: each weight takes part in one multiply and one add. The backward pass costs about twice the forward, so one training token costs about $6N$. Training on $D$ tokens and then generating $T$ tokens over the model's deployed life costs, in total,

Both terms leave out the attention scores, whose cost grows with context length and adds a few percent at these model sizes. Equation (1) is this page's summary, not an equation from the paper. Chinchilla fixes the first term and minimizes loss; it has no $T$ in it. The second term has no $D$ in it: once the model exists, serving a token costs the same whether it was trained on 200B tokens or 2T.

Put real sizes into (1). GPT-3 trained 175B parameters on 300B tokens: $6 \times 175\text{B} \times 300\text{B} \approx 3.2 \times 10^{23}$ FLOPs. LLaMA-13B trained 13.0B parameters on 1.0T tokens: $7.8 \times 10^{22}$ FLOPs, a quarter of GPT-3's training compute. Per served token the gap is wider: GPT-3 costs $3.5 \times 10^{11}$ FLOPs per generated token and LLaMA-13B costs $2.6 \times 10^{10}$, 13.5 times less, on every token, for as long as each model is in use. And LLaMA-13B scores higher than GPT-3 on most of the paper's benchmarks, so the smaller model is cheaper on both terms.

The second comparison in the figure is the one the paper's introduction raises. Chinchilla's rule would train a 10B model on 200B tokens ($1.2 \times 10^{22}$ FLOPs). LLaMA-7B was trained on 1.0T tokens ($4.0 \times 10^{22}$), 3.35 times as much training compute. Each token the 7B serves saves $2 \times (10 - 6.7)\text{B} = 6.6 \times 10^{9}$ FLOPs, so the extra training is repaid after$\;2.8 \times 10^{22} / 6.6 \times 10^{9} \approx 4.3$ trillion served tokens, about 4.3 billion answers of 1,000 tokens each. The paper reports that the 7B was still improving at 1T tokens; the figure prices compute only, so the break-even holds if the 7B is at least as good as the 10B.

At the left edge of the plot, a billion served tokens, each curve is flat at its training cost. Past about 10¹² served tokens the serving term dominates and both curves rise with slope 1, separated by the ratio of the parameter counts. At 100T served tokens, training is under 3% of LLaMA-13B's lifetime compute.

Training past the Chinchilla point

Choosing a small model and training it longer only works if loss keeps falling with more data. The paper's Figure 1 shows that it does. Its training-loss curves for the four sizes are all still descending at the last token, and the introduction states the case for the smallest model directly: although Hoffmann et al. "recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens."

Divide tokens by parameters for each model. The 7B saw 1.0T tokens for 6.7B parameters, about 149 tokens per parameter, 7.5 times Chinchilla's 20. The 13B saw about 77 per parameter. The 33B and 65B were trained on 1.4T tokens, about 43 and 21 per parameter, so the 65B sits near the compute-optimal point and the smaller models are the deliberately overtrained ones. At the end of training the 7B's loss is still falling, just slowly: each additional token lowers it by less than the last.

This does not refute Chinchilla. The paper says its training approach "is inspired by the Chinchilla scaling laws." For a fixed training budget, the overtrained 7B reaches a higher loss than a compute-optimal model would, as Chinchilla predicts. It wins on a quantity Chinchilla never measured: loss reached per FLOP of inference. The two laws minimize different things. Chinchilla answers "what is the cheapest way to train a model to this loss?" and LLaMA answers "what is the cheapest way to train and then serve a model at this loss?" Sardana et al. (2023) later wrote the combined objective down and solved it; for a model expected to serve a large number of tokens, the optimum is smaller and trained on more data than Chinchilla's, the direction LLaMA took by judgment.²

A dataset from public sources

The other half of the title is "open." The strongest models of 2022 were trained on data that nobody outside their labs could inspect; the LLaMA paper notes that such data is "either not publicly available or undocumented," described only by labels like "Social media conversations." LLaMA restricted itself to publicly available sources, so another group could in principle rebuild the training set, and so the trained weights could be shared with researchers. The mixture comes to about 1.4 trillion tokens after tokenization.

Most of it is web text. CommonCrawl, five dumps from 2017 to 2020, goes through a pipeline that deduplicates at the line level, drops non-English pages with a fastText language classifier, and removes low-quality text with an n-gram language model. A further linear classifier, trained to separate pages cited as references on Wikipedia from random pages, discards pages not classified as reference-like. C4, a second cleaned crawl, is added for diversity. Together the two crawls are 82% of the sampled tokens. The rest is curated: GitHub projects under Apache, BSD or MIT licenses; Wikipedia in 20 languages that use Latin or Cyrillic script; Project Gutenberg and the Books3 section of the Pile; arXiv LaTeX source with everything before the first section, the bibliography and the comments removed; and the 28 largest Stack Exchange sites, answers sorted by score.

The sampling proportion and the epoch count answer different questions. The proportion is the fraction of training tokens drawn from a source; the epoch count is how many times the run passes over that source. Take Wikipedia: 4.5% of 1.4T is about 63B sampled tokens, and at 2.45 epochs that means the Wikipedia source holds about 26B tokens, each read two or three times. CommonCrawl is 67% of the samples at 1.10 epochs, so almost every web token is seen once. Wikipedia and Books are the only sources read more than about once (2.45 and 2.23 epochs; the paper's text says "approximately two epochs"), and GitHub is read at 0.64, so about a third of the code is never seen. The small models, trained on 1.0T tokens, use the same proportions.

The tokenizer is byte-pair encoding, run through the SentencePiece library, with two choices that affect what the model can learn. Every number is split into single digits, so "2023" becomes the four tokens 2, 0, 2, 3 rather than one or two arbitrary chunks; the usual rationale, which the paper does not state, is that arithmetic generalizes better when every number is spelled from the same ten symbols. And any UTF-8 character the vocabulary lacks is decomposed into its raw bytes, so no input is ever out of vocabulary.

"Publicly available" meant downloadable, and that is different from licensed for this use. Books3 contains copyrighted books; it later drew takedowns and lawsuits. The paper does not discuss licensing for any of its sources.

Three borrowed architecture changes

The network is a decoder-only Transformer, the stack from Attention Is All You Need with only the decoder half and a causal mask. Table 2 of the paper gives the shapes. The 7B has model width $d = 4096$, 32 layers and 32 attention heads; the 65B has $d = 8192$, 80 layers and 64 heads. In all four sizes each head is 128 wide. Every model trains with a batch of 4M tokens and a learning rate of $3.0 \times 10^{-4}$ (7B, 13B) or $1.5 \times 10^{-4}$ (33B, 65B).

The optimizer is AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1 and gradient clipping at 1.0. The learning rate warms up over 2,000 steps and then follows a cosine schedule down to 10% of its peak.

The paper lists three departures from the original Transformer, each with the model it was taken from:

1. Pre-normalization, from GPT-3: normalize the input of each sublayer instead of its output, using RMSNorm.
2. SwiGLU, from PaLM: a gated feed-forward block in place of the ReLU one, at a reduced width.
3. Rotary embeddings, from GPT-Neo: encode position by rotating queries and keys at every layer, in place of a position vector added at the input.

Pre-normalization with RMSNorm

Each Transformer sublayer (attention, then the feed-forward block) sits inside a residual connection: the sublayer computes an update and adds it to its input. The original Transformer normalized after the addition, a layout called post-normalization. Deep post-norm stacks are hard to train unless the learning rate is ramped up slowly at the start. Pre-normalization moves the normalizer inside the branch, onto the sublayer's input, so the residual path from the first layer to the last carries the input unchanged plus the sum of every sublayer's update. Xiong et al. (2020) showed that the pre-norm layout has well-behaved gradients at initialization and can train with little or no warmup.³ LLaMA still uses 2,000 warmup steps; it adopted pre-norm, as the paper says, "to improve the training stability."

The normalizer itself is RMSNorm. LayerNorm does two things to a vector: it subtracts the mean of the components and divides by their standard deviation, then applies a learned gain and bias. RMSNorm keeps only the division, by the root mean square, and the gain:

Take a four-component input, $x = (1, 2, 3, 6)$. Its RMS is $\sqrt{(1 + 4 + 9 + 36)/4} = \sqrt{12.5} \approx 3.54$, so RMSNorm (with gain 1) returns $(0.28, 0.57, 0.85, 1.70)$. LayerNorm first subtracts the mean, 3, and divides by the standard deviation, $\sqrt{3.5} \approx 1.87$, giving $(-1.07, -0.53, 0, 1.60)$. Now add 10 to every component. LayerNorm's output does not change, because the mean absorbs the shift. RMSNorm returns $(0.84, 0.91, 0.99, 1.22)$: the shift survives, and the four values are squeezed toward 1.

Removing the mean is a real change to the function, as the figure shows, so dropping it has to be justified by experiment. Zhang and Sennrich hypothesized that LayerNorm's benefit comes from re-scaling (the output does not change when the input is multiplied by a positive constant) and that invariance to re-centering is dispensable. They supported this empirically: RMSNorm matched LayerNorm's quality on their tasks and cut running time by 7% to 64% depending on the model, since it computes one statistic per vector instead of two and has no bias to add. In the released code RMSNorm is one line, x * rsqrt(mean(x**2) + eps) * weight, and each block applies it twice, once before attention and once before the feed-forward block.

SwiGLU in the feed-forward block

After attention, every Transformer block runs a feed-forward network on each position independently. The standard version multiplies by a matrix that widens the vector from $d$ to $4d$, applies ReLU, and multiplies by a second matrix back down to $d$. ReLU passes positive values unchanged and sets negative values to zero. SwiGLU changes two things: the activation function, and the number of projections.

The activation is Swish, also called SiLU: $\operatorname{Swish}(z) = z\,\sigma(z)$, where $\sigma$ is the logistic sigmoid. For large positive $z$, $\sigma(z)$ is near 1 and Swish is close to $z$; for large negative $z$, it approaches 0. Between those limits it has a minimum of about $-0.28$ at $z \approx -1.28$, so small negative inputs pass a small negative value, and unlike ReLU it has a derivative at every point, including 0.

The "GLU" part, gated linear unit, is the second projection. The block computes two projections of the input, passes one through Swish, and multiplies the two elementwise before projecting back down:

Read one hidden coordinate at a time. If that coordinate of $xW$ is $-3$, Swish gives $-3 \times \sigma(-3) \approx -0.14$, and the matching coordinate of the value $xV$ is multiplied by $-0.14$: almost shut off. If the coordinate of $xW$ is 2, Swish gives about 1.76 and the value passes through scaled up. The gate is computed from the same input as the value, so the network learns, per coordinate and per token, how much of each value to let through, where a ReLU block can only zero a coordinate or pass it unchanged.

Nobody has a derivation showing why this works better. Shazeer (2020) compared GLU variants with ReLU and GELU blocks matched for parameters and computation; SwiGLU and GEGLU had the lowest held-out log-perplexity. His conclusion: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence."⁴ LLaMA uses SwiGLU because it measured better, in Shazeer's experiments and in PaLM.

The gate costs a matrix. A standard block has two, $W_1$ ($d \times 4d$) and $W_2$ ($4d \times d$): $8d^2$ parameters. SwiGLU has three: the gate $W$, the value $V$, and the down-projection $W_2$. At a hidden width of $m \cdot d$, that is $3m\,d^2$ parameters. PaLM used $m = 4$, which makes the block 1.5 times larger than a standard one. LLaMA uses $\tfrac{2}{3} \cdot 4d$, so $m = 8/3$ and $3m = 8$: the same $8d^2$ as the ReLU block. Shazeer made the same two-thirds cut for his comparison (a hidden width of 2,048 in place of 3,072). Drag the width and watch where the two rows line up.

The released code does not use $8d/3$ exactly. It computes int(2 * 4d / 3) and rounds up to a multiple of 256, for GPU-friendly matrix shapes. For the 7B, $8 \times 4096 / 3$ is 10,922.7, which becomes 11,008 (2.6875$d$). The other sizes get 13,824, 17,920 and 22,016.

Rotary position embeddings

Attention on its own has no notion of order. The score between two tokens is a dot product of a query and a key, and nothing in it depends on where the tokens sit, so without extra information "dog bites man" and "man bites dog" produce the same set of scores. The original Transformer added a fixed position vector to each token embedding once, at the bottom of the network. LLaMA removes those absolute embeddings and uses rotary position embeddings (RoPE, Su et al., 2021), which encode position by rotating the query and key vectors inside attention.

The construction splits each 128-wide query and key vector into 64 two-dimensional pairs. For a token at position $m$, pair $i$ is rotated by the angle $m\,\theta_i$, with a frequency that falls geometrically across the pairs, $\theta_i = 10000^{-2i/d}$ for $i = 0, \dots, d/2 - 1$ and $d$ the head width:

Each vector is rotated by an amount set by its own absolute position, and the score ends up depending only on the difference. A rotation preserves lengths and angles, and undoing a rotation by $m\theta$ and then rotating by $n\theta$ is a single rotation by $(n - m)\theta$ (the matrices satisfy $R_m^{\top} R_n = R_{n-m}$). So the dot product of a rotated query at position $m$ and a rotated key at position $n$ equals the dot product of the unrotated query with the key rotated by the offset alone:

A worked case on the fastest pair, $\theta_0 = 1$ radian per token. A query at position 10 is turned by 10 radians and a key at position 7 by 7 radians; the angle between them changes by 3 radians from their content angle. Move both tokens 100 positions later, to 110 and 107: each vector turns much further, and the angle between them still changes by exactly 3 radians. The score depends on the gap $m - n$, and the absolute positions drop out. Figure 7 shows this for any shift.

With 64 pairs at different frequencies, different pairs track position at different scales. The fastest pair turns 1 radian per token and completes a cycle every $2\pi \approx 6.3$ tokens, so it separates neighbors. The slowest turns by $10000^{-126/128} \approx 1.2 \times 10^{-4}$ radians per token and takes about 54,000 tokens to complete a cycle, so it changes little across a paragraph and separates distant parts of the text. RoFormer also shows an upper bound on the score that shrinks as the offset grows, a gradual decay of attention with distance built into the encoding.

Three details matter for an implementation. RoPE rotates $q$ and $k$ but not $v$: position enters the scores and nothing else. It is applied inside every attention layer, not once at the input. And it is a multiplication, where the original sinusoidal encoding was an addition to the embedding. The released code treats each pair as a complex number and multiplies it by $e^{\,i m \theta_i}$, which is the same rotation.

One block in code

With the three changes in place, a LLaMA block normalizes its input by RMS, runs causal attention with rotated queries and keys, adds the result back, normalizes again, runs the SwiGLU block, and adds again. The pseudocode follows the released model.py:

# one LLaMA block (after llama_v1/llama/model.py). x: (batch, seq, d)
h   = x + attention(rms_norm(x, g_attn))   # pre-norm: normalize the input only
out = h + swiglu_ffn(rms_norm(h, g_ffn))   # same shape out as in
#
# rms_norm(z, g):   z * rsqrt(mean(z**2, axis=-1) + eps) * g    # no mean, no bias
#
# attention(z):     heads of size 128; Wq, Wk, Wv, Wo are d x d, no biases
#   q, k, v = z @ Wq, z @ Wk, z @ Wv
#   q, k    = rope(q, pos), rope(k, pos)   # rotate q and k, never v
#   s       = softmax(q @ k.T / sqrt(128) + causal_mask)
#   return (s @ v) @ Wo
#
# swiglu_ffn(z):    W, V are d x f, W2 is f x d, f = 11008 when d = 4096
#   return (silu(z @ W) * (z @ V)) @ W2

The shapes in Table 2 are enough to recover the parameter count. Per layer, attention has four $d \times d$ matrices ($4d^2$, 67.1M for the 7B) and the SwiGLU block has three $d \times 11008$ matrices (135.3M), so the feed-forward block holds two thirds of each layer. Add the input embedding and the output layer, one row per vocabulary entry each, and the RMSNorm gains:

# LLaMA-7B parameter count from Table 2 (d = 4096, 32 layers)
d, L, V = 4096, 32, 32000              # V: the released tokenizer's vocabulary
f = 256 * ceil(int(2 * 4 * d / 3) / 256)   # 10922 -> 11008
attn = 4 * d * d                       #  67,108,864 per layer
ffn  = 3 * d * f                       # 135,266,304 per layer
embed_and_head = 2 * V * d             # 262,144,000
norms = (2 * L + 1) * d                #     266,240
total = L * (attn + ffn) + embed_and_head + norms   # 6,738,415,616

That is 6.74B, the paper's "6.7B." The same arithmetic with the other widths gives 13.02B, 32.53B and 65.29B against Table 2's 13.0B, 32.5B and 65.2B. There are no bias vectors anywhere; every linear layer in the code is created with bias=False.

Training the 65B in 21 days

The 65B model trained on 2,048 A100 GPUs with 80GB of memory each, processing about 380 tokens per second per GPU. That is $380 \times 2048 \approx 778{,}000$ tokens per second, and 1.4T tokens at that rate take 1.8 million seconds, about 21 days. In FLOPs, each GPU sustained about $6 \times 65.2\text{B} \times 380 \approx 1.5 \times 10^{14}$ per second, roughly 48% of an A100's 312 TFLOP/s peak for 16-bit matrix math. Two memory savings made that throughput possible, one in attention and one in the backward pass.

The first is in attention. A direct implementation computes the full table of query-key scores, applies the softmax, and stores the resulting attention weights for the backward pass. For a 2,048-token sequence that table has 4.2 million entries per head, 8.4MB in 16-bit floats, and a 7B-sized model has 32 heads in each of 32 layers: about 8.6GB for one sequence. LLaMA uses the memory-efficient causal attention in Meta's xformers library, which the paper says is inspired by Rabe and Staats (2021) and uses the backward pass of FlashAttention (Dao et al., 2022). It saves in two separate ways. It does not store the attention weights, which removes that memory. And it does not compute the scores the causal mask would hide, which removes about half of the score computation, since a token may attend only to itself and earlier tokens.

The second saving is activation checkpointing. The backward pass needs activations from the forward pass. Storing all of them costs memory in proportion to depth; storing none means recomputing the whole forward pass. LLaMA stores the activations that are expensive to recompute, such as the outputs of the linear layers, and recomputes the cheap ones (the elementwise operations around them) during the backward pass. To control exactly which activations are kept, the team wrote the backward function of the Transformer layers by hand instead of relying on PyTorch's autograd. Keeping those activations needs memory, so the paper also cuts per-GPU memory with the model and sequence parallelism of Korthikanti et al. (2022), covered in the activation recomputation explainer, and overlapped the GPUs' all-reduce communication with computation.

Results against GPT-3, Chinchilla and PaLM

The figure shows seven zero-shot commonsense benchmarks from the paper's Table 3 and the 5-shot MMLU average from Table 9, for LLaMA-13B and LLaMA-65B against GPT-3 175B, Chinchilla-70B and PaLM-540B. Step through them and read the line under the bars.

LLaMA-13B scores above GPT-3 on five of the seven commonsense benchmarks and on MMLU (46.9 against 43.9). It trails on PIQA (80.1 against 81.0) and OpenBookQA (56.4 against 57.6), so the abstract's "most benchmarks" is accurate and "all" would not be. GPT-3 is also a weak baseline for its size: it predates Chinchilla and saw about 1.7 tokens per parameter (300B tokens for 175B parameters), against 77 for LLaMA-13B. The paper notes that the 13B runs inference on a single V100 GPU, a 2017 data-center card.

The comparison with the best models falls to the 65B. On the commonsense benchmarks it scores above Chinchilla-70B on all five that Chinchilla reports (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande), and above PaLM-540B on all except BoolQ (85.3 against 88.0) and WinoGrande (77.0 against 81.1). On MMLU it trails both: 63.4 against 67.5 for Chinchilla and 69.3 for PaLM. The abstract calls it "competitive with" them, which fits these numbers. On closed-book question answering (NaturalQuestions and TriviaQA) the 65B has the best scores among the compared models in both zero-shot and few-shot settings.

Where it falls short

The MMLU gap is the largest. The paper attributes it to data: its books and academic papers (ArXiv, Gutenberg and Books3) add up to 177GB, while Gopher, Chinchilla and PaLM trained on up to 2TB of books, and MMLU is an exam over academic subjects. A brief instruction finetuning run, following the protocol of Chung et al. (2022), raises the 65B's MMLU from 63.4 to 68.9, which the paper reports as LLaMA-I.

Toxicity on RealToxicityPrompts increases with model size, most clearly for prompts that ask the model to be respectful. On CrowS-Pairs the model is most biased in the religion category, 79.0 against OPT-175B's 68.6. On TruthfulQA it does better than GPT-3, but the paper notes that its rate of correct answers is still low.

The training has an energy cost. The 65B alone took 1,022,362 GPU-hours, 449 MWh and an estimated 173 tonnes of CO₂-equivalent. Developing all the models, about five months of work on 2,048 A100s, used about 2,638 MWh and 1,015 tonnes. The authors argue that releasing the models reduces future emissions, since other groups can start from the trained weights instead of training their own.

The paper released all four models "to the research community." Later open models such as Mistral 7B use the same block design with RMSNorm, SwiGLU and RoPE, and QLoRA (2023) finetuned LLaMA models up to 65B on a single 48GB GPU.