Chinchilla: Training Compute-Optimal Large Language Models

At a fixed training budget, a smaller model trained on more text reaches a lower loss.

DeepMind trained over 400 language models to find how a compute budget should be split between parameters and training tokens. They found that both should grow at the same rate, about 20 tokens per parameter. A 70B model trained that way outperformed the 280B Gopher at the same cost.

GPT-3, Jurassic-1, Gopher and Megatron-Turing NLG were each trained on roughly 300 billion tokens. By this paper's analysis, Gopher's compute would have been better spent on a model a quarter of its size reading more than four times as much text.

In 2020, Kaplan and colleagues at OpenAI fit power laws to language-model loss (covered in the Scaling Laws explainer) and concluded that when the training budget grows, most of it should go into a larger model: the optimal parameter count grows as $C^{0.73}$ and the optimal token count as $C^{0.27}$. By that rule a budget ten times larger buys a model 5.4 times larger and only 1.9 times more tokens. The largest models of the next two years were built that way:

The first four rows are from the paper's Table 1, including the GPT-3 row. Hoffmann et al. at DeepMind trained over 400 models, from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and asked one question of them: for a fixed amount of training compute, which model size and token count give the lowest loss? Three different estimation methods gave the same answer, that parameters and tokens should grow in about equal proportion. They then tested it at full scale by training Chinchilla, the last row.

The page goes in this order: how compute is counted and why a fixed budget forces a trade; the three estimation methods (training-curve envelope, IsoFLOP profiles, a fitted loss formula) and the closed-form optimum the third one gives; why the earlier study got a different answer; the Chinchilla-against-Gopher test; and a 2024 replication which showed that the constants printed for the third method point to a different ratio than the paper's conclusion.

How training compute is counted

A training budget is measured in floating-point operations (FLOPs): the multiplies and adds the accelerators perform over the whole run. The paper uses the approximation from Kaplan et al. for a dense Transformer:

where $C$ is total training FLOPs, $N$ the number of parameters, and $D$ the number of training tokens (a token is a chunk of text, a word or a piece of one, as cut up by the tokenizer). The 6 comes from two passes over the network. In the forward pass each parameter takes part in one multiply-add per token (a weight multiplies an activation and the product is added into a running sum), which is 2 FLOPs, so one token costs about $2N$ FLOPs. The backward pass does two matrix products per layer where the forward pass did one: one computes the gradient of the loss with respect to that layer's weights, the other computes the gradient passed down to the layer below. That is about $4N$ FLOPs per token, and the run total is $(2N + 4N)\,D$.

For Gopher, 280B parameters on 300B tokens, (1) gives $6 \times 280\text{B} \times 300\text{B} = 5.04\times10^{23}$. For Chinchilla, 70B on 1.4T tokens, it gives $5.88\times10^{23}$. The Gopher paper reported its training compute as $5.76\times10^{23}$, and that is the budget the Chinchilla paper holds fixed throughout. The paper's Appendix F also counts every term separately (embeddings, attention including the terms that grow with the square of the sequence length, the dense blocks, the output logits), which puts Gopher at $6.3\times10^{23}$; for models from 73M to 6.8B parameters that full count comes out between 0.99 and 1.10 times $6ND$. (1) leaves out attention's sequence-length terms, which are small at these model widths.

The two buttons in Figure 1 load Gopher and Chinchilla. The bar splits the total into its forward and backward parts, and the line under the total gives it as a multiple of the $5.76\times10^{23}$ budget:

# training compute, the paper's approximation (App. F checks it)
def train_flops(N, D):        # N parameters, D training tokens
    fwd = 2 * N * D           # one multiply-add per parameter per token
    bwd = 2 * fwd             # backward pass ~ twice the forward
    return fwd + bwd          # = 6 * N * D

train_flops(70e9, 1.4e12)     # Chinchilla -> 5.88e23
train_flops(280e9, 300e9)     # Gopher     -> 5.04e23 (reported: 5.76e23)

A fixed budget forces a trade between model size and data

Once the budget is fixed, (1) fixes the product $N D$, so choosing a model size also chooses the token count:

The paper's Appendix D.4 runs this choice at $10^{21}$ FLOPs. Kaplan's rule recommends a 4.74B model there, which by (2) can read 35.2B tokens, about 7 per parameter. Approach 1 of this paper recommends 2.80B, which reads 59.5B tokens, about 21 per parameter. The authors trained both, and the 2.80B model ended with the lower loss.

The two ends of the range fail in different ways. A model much too small for its budget reads an enormous corpus but has too few parameters to fit it, so its loss flattens out well above what a larger model reaches. A model much too large reads so few tokens that training ends while its loss is still falling steeply. Between the two sits one size with the lowest final loss for that budget, which the paper calls compute-optimal and estimates in three ways.

All three use the same Transformer family, the same corpus (MassiveText, Gopher's training data), and cosine learning-rate schedules that lower the rate by 10× over the planned length of a run. How that length is chosen matters for the comparison with Kaplan's result further down.

Approach 1: the lowest training curve at each budget

The first method reuses training curves. The authors fix a set of model sizes from 70M to over 10B parameters and train each size four times, with cosine schedules whose lengths span a factor of 16. Plot each run's loss against the FLOPs spent so far ($6N$ times the tokens read) and each size traces a falling curve. At any FLOP count, the lowest curve is the best loss that much compute could buy, and its model size is the compute-optimal size for that budget. The paper smooths and interpolates every curve, repeats the selection at 1500 log-spaced FLOP values, and fits power laws to the winning sizes and token counts.

Figure 2 draws seven sizes across that range. Move the budget line, with the slider or the pointer, and read off which curve is lowest. Small models win at small budgets; each larger model takes over once the budget gives it enough tokens to get below the smaller ones:

With only seven sizes the winner changes in steps, so its tokens-per-parameter readout jumps between about 9 and 65 as the line moves. The paper's power-law fit over many sizes and 1500 budgets smooths those steps out and gives $N_{\text{opt}} \propto C^{0.50}$ and $D_{\text{opt}} \propto C^{0.50}$. Every point the selection picked lies in the last 15% of its run's schedule, which the authors turn into practical advice: set the cosine cycle to decay over about the number of tokens the model will actually be trained on.

Approach 2: IsoFLOP profiles

The second method fixes the budget first. For each of 9 budgets from $6\times10^{18}$ to $3\times10^{21}$ FLOPs, the authors train a family of model sizes, each for the token count (2) allows and with its cosine schedule set to that token count, and record only the final loss of each run. The name IsoFLOP means equal FLOPs: every run in one family cost the same. Plotted against model size, each family forms a valley, with the too-small models on the left and the too-large ones on the right. The paper fits a parabola to each valley and takes its minimum as the compute-optimal size for that budget.

In Figure 3 the budget slider picks one of nine valleys. The amber dots mark the minimum of each, and the readout in the corner gives the tokens-per-parameter ratio at the highlighted one:

Fitting the minima across budgets gives two power laws,

with $a = 0.49$ and $b = 0.51$. With both exponents near one half, 100 times more compute means about 10 times more parameters and 10 times more tokens. Kaplan's exponents turn the same 100× into 29× more parameters and 3.5× more tokens. Extrapolated to Gopher's budget, the IsoFLOP fit projects an optimal size of 63B.

Approach 3: a formula for the whole loss surface

The third method models the final loss of every run from Approaches 1 and 2 at once, as a function of both model size and token count:

The paper gives each term a meaning. $E$ is the loss of an ideal model of the text, which the paper equates with the entropy of natural text: no model size or amount of data goes below it. $A/N^{\alpha}$ is how far a perfectly trained Transformer with $N$ parameters falls short of that ideal; it shrinks as $N$ grows, at a rate set by $\alpha$. $B/D^{\beta}$ is the extra loss from stopping after $D$ tokens instead of training to convergence; it shrinks with more data at a rate set by $\beta$.

The fit minimizes a Huber loss (squared error for small residuals, absolute error for large ones, with threshold $\delta = 10^{-3}$) between predicted and observed log loss, using the L-BFGS optimizer from a grid of starting points. The Huber loss keeps a few badly fitting runs from dragging the fit; in practice the smallest runs, at $C \le 10^{21}$ FLOPs, have the largest residuals and are treated as outliers. The paper reports

As a worked example, put Chinchilla and Gopher into (4) with these constants. Chinchilla (70B, 1.4T) gets $1.69 + 0.083 + 0.163 = 1.94$. Gopher (280B, 300B) gets $1.69 + 0.052 + 0.251 = 1.99$. Gopher's four-times-larger size lowers the parameter term by 0.031, and its 300B tokens raise the data term by 0.088, so at about the same compute the smaller model is predicted to reach the lower loss.

The formula also gives the compute-optimal split in closed form. Substitute (2) into (4): the data term becomes $B\,(6N/C)^{\beta}$, which grows with $N$, while the parameter term shrinks with it. Differentiate in $N$ and set the result to zero:

The condition on the right has a direct reading. Increasing $\ln N$ by a small amount lowers the parameter term by $\alpha$ times its current value per unit of $\ln N$, and because $ND$ is fixed the same move lowers $\ln D$ by the same amount, raising the data term by $\beta$ times its value per unit. The best split sets those two rates equal. Solving for $N$ gives $N^{\alpha+\beta} = \frac{\alpha A}{\beta B}\,(C/6)^{\beta}$, which is the paper's Eq. (4):

The exponent on model size, $a$, comes from the data exponent $\beta$, and vice versa. They always sum to one, because $N_{\text{opt}} D_{\text{opt}}$ has to equal $C/6$, and they are equal exactly when $\alpha = \beta$: when an extra doubling of parameters and an extra doubling of tokens reduce loss at the same rate, the budget goes to both equally. With the printed $\alpha = 0.34$ and $\beta = 0.28$, $a = 0.28/0.62 = 0.45$ (the paper's Table 2 lists 0.46, from the unrounded fit) and $b = 0.55$.

Figure 4 draws the surface (4) over the plane of model size and tokens, in log-log axes, using the constants of a 2024 re-fit by Epoch AI instead of the printed ones; the section on the constants explains why. Each faint diagonal is one budget, because $ND = C/6$ is a straight line of slope −1 in these axes. The amber line is the compute-optimal frontier (6). The last slider stop is Gopher's budget, with rings on Chinchilla and Gopher:

The frontier is a straight line of slope about one: under the re-fit, $a = 0.513$ and $b = 0.487$, so every doubling of parameters on the frontier comes with slightly less than a doubling of tokens. At Gopher's budget, (6) with the re-fit constants gives 72B parameters and 1.33T tokens, and the balance condition checks out at Chinchilla itself: $\alpha A/N^{\alpha} = 0.028$ and $\beta B/D^{\beta} = 0.027$. The listing below computes the same optimum:

# compute-optimal split for a budget C, from the fitted loss (4)
E, A, B, alpha, beta = 1.8172, 482.01, 2085.43, 0.3478, 0.3658  # Epoch re-fit

def loss(N, D):
    return E + A / N**alpha + B / D**beta

def optimal_split(C):
    # closed form (6)
    G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
    N = G * (C / 6) ** (beta / (alpha + beta))
    D = (C / 6) ** (alpha / (alpha + beta)) / G
    return N, D

N, D = optimal_split(5.76e23)   # Gopher's budget
# N ~ 7.2e10, D ~ 1.33e12, D / N ~ 18, loss(N, D) ~ 1.97
# brute-force check: scan N, set D = C / (6N), keep the lowest loss

Why compute-optimal training uses about 20 tokens per parameter

The paper's Table 2 puts the three estimates next to Kaplan's:

The ratio of 20 tokens per parameter is a reading of the paper's Table 3 (projections from Approach 1), not a phrase the paper prints: 1B parameters go with 20.2B tokens, 10B with 205.1B, and 67B with 1.5T. Chinchilla, at 70B and 1.4T, is exactly 20.

Combined with (1), the ratio gives a rule for sizing a run. Setting $D = 20N$ in $C = 6ND$ gives $C = 120N^2$, so

At Gopher's $5.76\times10^{23}$ FLOPs this gives 69B parameters and 1.39T tokens. At $10^{21}$ FLOPs it gives 2.9B parameters and 58B tokens, close to the 2.80B model of the head-to-head comparison above.

Figure 5 starts both rules from the same compute-optimal model, 400M parameters on 8B tokens, and multiplies the budget. The number under each pair of bars is its tokens per parameter:

The paper names two reasons Kaplan et al. reached a different answer. The first is the learning-rate schedule. Kaplan et al. trained their models with one fixed cosine schedule to 130B tokens and used losses read partway through those runs as stand-ins for shorter training runs. A cosine schedule lowers the learning rate gradually over the planned run, and a large share of the loss reduction happens late, as the rate approaches its minimum. A run stopped at 20B tokens of a 130B-token schedule still has a high learning rate, so its loss is higher than that of a run whose schedule was set to end at 20B. The overestimate is largest for the shortest horizons, which makes training on fewer tokens look less effective than it is and, in the paper's words, “contributes to the conclusion” that model size should grow faster than data.

The second reason is the range of model sizes. The frontier of best loss per FLOP bends slightly as models grow (the paper's Appendix E), so fits to small models extrapolate differently from fits to large ones. Most of Kaplan's runs were small, many under 100M parameters; most of the Chinchilla runs have more than 500M, and they go up to 16B.

Chinchilla against Gopher at the same compute

The three approaches put the optimal model for Gopher's $5.76\times10^{23}$ FLOPs between 40B and 70B parameters. The authors trained a model at the larger end: 70B parameters on 1.4T tokens. By (1) that is $5.88\times10^{23}$ FLOPs against 6ND's $5.04\times10^{23}$ for Gopher, the same budget to within the accuracy of the approximation, split the other way: a quarter of the parameters and about 4.7 times the tokens (1.4T / 300B; the paper rounds to 4×). Both models have 80 layers, so the size difference is in the width:

Halving $d_\text{model}$ divides the weight matrices by four. A Transformer layer holds about $12\,d_\text{model}^2$ weights (four $d \times d$ attention projections and a feed-forward block of width $4d$), so 80 layers give about 258B for Gopher and 64B for Chinchilla, before embeddings.

The split is not the only change. Chinchilla uses AdamW instead of Adam, a tokenizer that skips NFKC unicode normalisation (94.15% of its vocabulary is shared with Gopher's), a float32 copy of the weights in the sharded optimizer state (the scheme from ZeRO), and a slightly different mix of MassiveText subsets to supply the extra tokens. The paper's Appendix G trains 417M, 680M and 1.4B models with Adam and with AdamW, and AdamW ends with the better loss independent of the learning-rate schedule.

On the results: Chinchilla reaches 67.5% on MMLU, 57 tasks of multiple-choice exam questions on academic subjects, against Gopher's 60.0%, more than 7 points higher; it is better on 51 of the 57 tasks, equal on 2 and worse on 4. It has lower bits-per-byte than Gopher on every subset of the Pile evaluated, and a WikiText-103 perplexity of 7.16 against 7.75, though the authors caution that a model that read four times more text is more likely to have seen test documents during training. The abstract states that Chinchilla outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) on a large range of downstream tasks; the conclusion scopes it as “nearly every measured evaluation task”.

The size difference also matters after training. Generating one token costs about $2N$ FLOPs in the forward pass, so Chinchilla needs about a quarter of Gopher's compute per generated token; the paper notes it uses substantially less compute for fine-tuning and inference.

The Approach 3 constants that do not reproduce the headline

In 2024 Besiroglu, Erdil, Barnett and You at Epoch AI reconstructed the data points from the paper's plots and re-fit (4). Their fit gives $E = 1.8172$, $A = 482.01$, $B = 2085.43$, $\alpha = 0.3478$ and $\beta = 0.3658$, and through (6) it recommends about 20 tokens per parameter, in line with Approaches 1 and 2. The constants printed in (5) do not: Epoch summarizes the policy they imply as about 70 tokens per parameter, and at Gopher's budget (6) with the printed values gives 32B parameters and 93 tokens per parameter.

The ratio is this sensitive because of how (6) combines the constants. Dividing the two halves of (6) gives

The exponent depends on $\alpha - \beta$, a difference between two numbers of similar size, and $G$ raises $\alpha A/(\beta B)$ to the power $1/(\alpha+\beta) \approx 1.6$ before (7) squares it. With the printed $\alpha = 0.34$ and $\beta = 0.28$ the exponent is +0.097, so the recommended ratio climbs with the budget, from 26 at $10^{18}$ FLOPs to about 150 at $10^{26}$. Changing $\beta$ alone from 0.28 to 0.30 brings the ratio at Gopher's budget from 93 down to 19.

Two causes have been identified. The printed exponents are rounded: a comment in the paper's TeX source keeps the fit's raw output, with $\alpha = 0.33917$ and $\beta = 0.28491$. Those unrounded values give 40B parameters at Gopher's budget, the Approach 3 projection the paper reports, and $a = 0.457$, the 0.46 in Table 2, but still 59 tokens per parameter. The rest of the gap, according to one of the paper's authors as reported in Epoch's updated paper, came from the fit itself: the Huber losses were averaged over data points instead of summed, a choice of loss scale that made L-BFGS stop before it converged. The same early stop explains the reported confidence interval for $a$, 0.454 to 0.455. Epoch estimates that an interval that narrow would take over 600,000 training runs, where the paper had fewer than 500.

Figure 7 plots the recommended tokens per parameter against compute for the printed constants, the unrounded ones, and the re-fit. The $\beta$ slider changes only the printed data exponent; the rows under the plot give each set's optimum at Gopher's budget:

The correction is confined to Approach 3's printed numbers. The conclusion that parameters and tokens should grow at about the same rate rests on Approaches 1 and 2, which do not use (4), and on Chinchilla outperforming Gopher. The paper itself remarks that Approach 3 predicts smaller optimal models than the other two. Both problems push in that direction: at Gopher's budget the printed constants give 32B, the unrounded ones 40B, and the re-fit 72B.

Limitations and later practice

The authors list the limits in their conclusion. There are only two comparable runs at large scale, Chinchilla and Gopher, and no tests at intermediate scales. The analysis assumes the frontier is a power law, yet they observe some concavity in $\log N_{\text{opt}}$ at high compute, which suggests the optimal size of large models may still be overestimated, i.e. the best models at large budgets may be smaller still. Every run in the analysis trained on less than one epoch of its data, so the paper says nothing about repeating data. And a 20:1 ratio at large scale needs trillions of tokens, which makes collecting large, high-quality datasets central; the paper stresses that more data helps only when it is high quality.

Compute-optimal is a statement about training compute only. A model that will serve many requests can justify a smaller size trained on more tokens than 20 per parameter, paying more at training time to spend less per generated token. Sardana et al. (2024) add inference cost to the objective and get higher ratios, and LLaMA trained its 7B model on 1T tokens, about 150 per parameter.