Kaplan scaling laws: Scaling Laws for Neural Language Models

Test loss falls as a power law in model size, data and compute, so small training runs predict large ones.

Kaplan and colleagues at OpenAI trained decoder-only Transformers from 768 to 1.5 billion non-embedding parameters, about a million-fold range of sizes, and fit the test loss with a few power laws. The same equations say how much data a model needs, how to split a compute budget, and where the trends have to stop.

A January 2020 paper whose advice on spending compute shaped the largest language models for two years, until a 2022 re-measurement changed the split between parameters and data.

Before this paper, deciding how big to make a language model was mostly a matter of trying it. Kaplan and colleagues measured instead. They trained Transformer language models across six orders of magnitude in size, more than two orders of magnitude in dataset size and eight orders of magnitude in compute, and recorded the loss on held-out text, text the model never trained on. Plotted against size, data or compute on logarithmic axes, the loss falls along straight lines, and none of the lines bends at the top of the range they tested.

The loss is the cross-entropy per token in nats: the model's average negative log-probability of the true next token. It depends on three quantities: the number of parameters $N$, the number of training tokens $D$, and the training compute $C$. When one of the three limits training and the other two are plentiful, the loss is a power law in that one.

Those fits are useful in two ways. You can predict: fit the line on small models, then read off the loss of a model a hundred times larger before paying for it. And you can allocate: with the loss written as a function of size, data and compute, you can ask which split of a fixed budget reaches the lowest loss. Kaplan's answer was to spend most of it on model size and stop training early. The field followed that answer until Chinchilla re-measured the split in 2022.

The page follows the paper's order: the power-law form and how to read it; what exactly $N$, $D$ and $C$ count; the joint law for size and data, which says how much data a model needs; learning curves and why larger models need fewer steps; the critical batch size; the compute-efficient allocation; the point where the trends contradict each other; and what later work corrected.

What a power law is, and why log-log axes show it as a line

A power law says the loss is a fixed power of some quantity $X$, written here as $L = (X_c/X)^{\alpha}$. Taking logarithms of both sides gives

which is a straight line in $\log X$ with slope $-\alpha$. On axes that plot both quantities by their logarithms, a power law is a line, and the exponent $\alpha$ is how steeply it falls.

The quantities here span orders of magnitude, so the choice of axes decides what you can see. On linear axes the loss drops sharply at the far left and then runs along a nearly flat tail for the rest of the range, and a power law is hard to tell apart from an exponential or a logarithm. Pick a quantity in Figure 1 and drag the slider from linear to log-log. The curve straightens onto the dashed chord; the slope you end on is $-\alpha$.

The paper fits three such laws. Against non-embedding parameters:

This one is measured on models trained to convergence on enough data that data is not the limit. Against dataset size in tokens, for large models trained with early stopping (training halted once the test loss stops improving):

And against compute in PF-days, when model size and batch size are chosen to use that compute efficiently:

The subscript on $C_{\min}$ is defined in the batch-size section below. Until then, $C_{\min}$ is the compute of a run that wastes none of it on an oversized batch.

The constants $N_c$, $D_c$ and $C_c$ depend on the tokenizer and dataset, and the paper says they have no fundamental meaning. The exponents carry the content. With $\alpha_N \approx 0.076$, doubling the model multiplies the loss by $2^{-0.076}$, about 0.95, a 5% relative drop. Every further doubling multiplies it by the same 0.95, so the absolute gain shrinks as the loss shrinks. Ten doublings, a model 1024 times larger, multiply the loss by $2^{-0.76} \approx 0.59$. Data is a little steeper (a doubling of $D$ multiplies the loss by 0.94) and compute is shallower (a doubling multiplies it by 0.97; ten times the compute, by 0.89).

A loss of $L$ nats per token corresponds to a perplexity of $e^{L}$: on average the model is as uncertain as a uniform guess among that many tokens. One nat is about 1.44 bits. A 1-billion-parameter model in (1.1) has $L = (8.8\times 10^{4})^{0.076} \approx 2.38$ nats, a perplexity of about 10.8.

What N, D and C count, and why model shape barely matters

Depth, width, number of attention heads and the ratio of width to depth appear nowhere in (1.1). Within the Transformer family the authors varied each of them at a fixed parameter count and found the loss moved by a few percent at most. A model with $(n_{\text{layer}}, d_{\text{model}}) = (6, 4288)$ reaches a loss within 3% of the $(48, 1600)$ model used in GPT-2, which has about the same parameter count, and the ratio of width to depth can vary by a factor of 40 with only a small effect on the loss. Across model families the architecture does matter: in the same paper, LSTMs matched Transformers on tokens early in the context but fell behind on later tokens.

$N$ is the number of non-embedding parameters: the attention and feed-forward weights, without the token-embedding matrix ($n_{\text{vocab}}\, d_{\text{model}}$ parameters) or the positional embeddings ($n_{\text{ctx}}\, d_{\text{model}}$). Each layer has $4d^2$ attention weights (queries, keys, values and the output projection) and $8d^2$ feed-forward weights (two matrices between width $d$ and $4d$), so with the standard shape

The paper leaves the embeddings out because, counted with them, its loss curves depended on depth as well as on parameter count; counted without them, models of different depths fell on one line. Figure 2 draws the shape as a stack of layers, puts $N$ and the embedding count side by side on one bar, and prints the loss (1.1) predicts. Compare the (6, 4288) and (48, 1600) presets, which look nothing alike and get nearly the same $N$, then set a 2-layer, 128-wide model and look at the bar.

At GPT-3's shape, (96, 12288), the embeddings are 0.36% of the parameters and the choice of count changes almost nothing. At (2, 128) the model has 0.39 million non-embedding parameters and 6.6 million embedding parameters, 94% of the total. The paper's size range starts at 768 non-embedding parameters, so many of its fitted points are models like this one, and the last section returns to what that did to its compute recommendation.

Compute is counted from $N$ as well. A forward pass costs about $2N$ floating-point operations per token, one multiply and one add per weight, plus an attention term ($2 n_{\text{layer}} n_{\text{ctx}} d_{\text{model}}$) that is small when $d_{\text{model}}$ is well above $n_{\text{ctx}}/12$. The backward pass costs about twice the forward pass, so a training token costs about $6N$ operations, and a run of $S$ steps at batch size $B$ tokens costs

The paper quotes compute in PF-days: $10^{15}$ operations per second for a day, $8.64\times 10^{19}$ operations. The paper's default run trains for $2.5\times 10^{5}$ steps at 512 sequences of 1024 tokens ($2^{19}$ tokens) per batch with Adam (Adafactor, a lower-memory variant, for models above a billion parameters), a 3000-step linear warmup and a cosine decay to zero. With the definitions fixed, (1.1) can be evaluated directly:

# the single-variable fits (Kaplan et al., WebText2, loss in nats/token)
aN, Nc = 0.076, 8.8e13         # L(N) = (Nc / N) ** aN, N = non-embedding params
aD, Dc = 0.095, 5.4e13         # L(D) = (Dc / D) ** aD, D = tokens

def loss_from_params(N):
    return (Nc / N) ** aN

loss_from_params(1e9)          # 2.38 nats/token, perplexity e**2.38 = 10.8
loss_from_params(1.5e9)        # 2.30
loss_from_params(2e9) / loss_from_params(1e9)   # 2 ** -0.076 = 0.949

How much data a model needs: the joint law L(N, D)

Laws (1.1) and (1.2) each assume the other resource is unlimited. With a fixed amount of text and a growing model, the model eventually fits the particular text it was given better than the language that produced it, and the test loss stops improving. The paper fits one equation over both variables, the early-stopped test loss:

With unlimited data, $D_c/D$ goes to zero and (1.5) reduces to $(N_c/N)^{\alpha_N}$, which is (1.1). With an unlimited model, the first term goes to zero and (1.5) reduces to $(D_c/D)^{\alpha_D}$, which is (1.2). In between, the larger of the two terms inside the bracket sets the loss. The joint fit has its own constants ($\alpha_N = 0.076$, $\alpha_D = 0.103$, $N_c = 6.4\times 10^{13}$, $D_c = 1.8\times 10^{13}$), slightly different from the single-variable fits because it is fit to a different set of runs.

The paper measures overfitting as the relative excess loss over the infinite-data value, $\delta L = L(N,D)/L(N,\infty) - 1$. Substituting (1.5):

$N$ and $D$ enter only through the combination $N^{\alpha_N/\alpha_D}/D$. With the joint-fit exponents that is $0.076/0.103 = 0.74$, so overfitting depends on $N^{0.74}/D$. Dividing the single-variable exponents instead (0.076/0.095) would give 0.80, a different number from a different fit. Seed-to-seed variation in the loss is about 0.02, so the paper asks for$\delta L$ below that, which gives

In Figure 3 the dashed line is (1.1), the infinite-data limit, and the bright curve is (1.5) for the dataset size on the slider. The amber marker sits at the largest model (4.4) allows for that data. Past it, the bright curve leaves the dashed line and flattens toward the floor $(D_c/D)^{\alpha_D}$ set by the data alone.

The exponent 0.74 is below 1, so data has to grow with the model but more slowly. A model eight times larger needs $8^{0.74} \approx 4.7$ times the data to stay within the threshold; the paper rounds this to "roughly 5x".

Take $N = 10^9$. Law (1.1) predicts 2.38 nats if data is not the limit. Rule (4.4) asks for $5\times 10^{3} \times (10^{9})^{0.74}$. The power is $10^{6.66} \approx 4.6\times 10^{6}$, and times 5000 that is about $2.3\times 10^{10}$ tokens, 23 billion. WebText2, the paper's training set, has $2.29\times 10^{10}$ tokens in total, of which $6.6\times 10^{8}$ are held out for testing, so a billion-parameter model is about the largest this dataset supports. The paper says the same: models below about $10^9$ parameters train with minimal overfitting on WebText2, and its largest models see mild overfitting. Train the billion-parameter model on a tenth of that data, 2.3 billion tokens, and (4.3) puts its loss about 13% above the infinite-data value.

Larger models learn faster: the learning-curve law L(N, S)

The second pairing is model size against training time. A larger model does more arithmetic per step, and it is natural to expect it to need more steps as well. The paper finds the opposite: larger models reach any given loss in fewer optimization steps and after seeing fewer tokens. After an initial transient, learning curves with unlimited data fit

In (1.6), $S_{\min}$ is the number of steps the run would need at a very large batch, a correction the next section defines; for a run at a fixed batch below the critical size it is smaller than the raw step count. The fit's own $\alpha_N$ and $N_c$ are 0.077 and $6.5\times 10^{13}$.

The first term is the loss the model converges to, lower for a larger model. The second term decays with training and has the same form for every size. Each model descends the same curve shape onto its own floor, and a larger model's floor is lower, so its whole curve is lower. At a target of 3.8 nats, (1.6) gives a 300M-parameter model 1.6 thousand steps and a 3M-parameter model 31 thousand. A 300k-parameter model has a floor of 4.38 nats and never reaches 3.8, however long it trains.

Two consequences follow for a fixed compute budget. A large model reaches a given loss in fewer steps, so it can be stopped early. A small model trained to convergence ends at its floor, which a larger model passes partway through its own run. Whether stopping a large model early beats converging a small one at equal compute is the question the compute section answers.

The critical batch size

Batch size sets a trade between time and compute. A larger batch gives a less noisy gradient, so fewer steps reach the same loss, but each step processes more tokens. The paper uses the empirical model of McCandlish et al. (2018): to reach a fixed loss, the number of steps $S$ and the number of training tokens $E$ satisfy

where $S_{\min}$ is the fewest steps that can reach the loss (with a very large batch) and $E_{\min}$ the fewest tokens (with a very small batch). Their ratio defines the critical batch size, $B_{\text{crit}} = E_{\min}/S_{\min}$. Solving (5.1) at batch $B$ gives $S = S_{\min}(1 + B_{\text{crit}}/B)$ and $E = E_{\min}(1 + B/B_{\text{crit}})$. Well below $B_{\text{crit}}$, doubling the batch nearly halves the steps and costs almost no extra tokens. Well above it, doubling the batch nearly doubles the tokens and saves almost no steps. At $B = B_{\text{crit}}$ a run takes twice the minimum steps and twice the minimum tokens.

Kaplan's measurement is that $B_{\text{crit}}$ does not depend on model size, only on the loss being targeted:

With $1/\alpha_B \approx 4.8$, the critical batch grows quickly as the loss falls. At $L = 4$ it is about 270 thousand tokens; at $L = 3$, $2\times 10^{8}/3^{4.76} \approx 1.1$ million. The paper's default batch of $2^{19}$ = 524,288 tokens sits below it at a loss of 3, so those runs took about three times the minimum steps and about 1.5 times the minimum tokens. In Figure 5 the teal curve is $S/S_{\min}$, the cost in time, and the amber curve $E/E_{\min}$, the cost in tokens and therefore in compute at fixed model size. They cross at $B_{\text{crit}}$, both at 2. Lower the target loss and watch the crossing move right.

The trade-off defines the corrected compute used in (1.3). A run at batch $B$ spends compute $C$; the same run at a batch far below $B_{\text{crit}}$ would process only $E_{\min}$ tokens and spend

and $S_{\min} = S/(1 + B_{\text{crit}}/B)$ is the matching correction for steps, the one used in (1.6). The paper's runs used a fixed batch, which was too large for early, high-loss parts of training and too small late in training; converting to $C_{\min}$ and $S_{\min}$ removes that and gives cleaner fits. A run at exactly $B_{\text{crit}}$ spends $C = 2C_{\min}$.

How to spend a compute budget: the compute-efficient frontier

With a fixed compute budget, $C \approx 6NBS$ makes model size and training length compete: a larger model costs more per step, so it gets fewer steps. The question is which model size turns the budget into the lowest loss.

Each model size traces a curve of loss against compute spent as it trains, falling and then flattening at its converged loss. Draw the curves for all sizes on one plot and take the lowest loss reached at each compute value; that lower envelope is the compute-efficient frontier. In Figure 6, move the budget and look at where the highlighted model sits when it touches the frontier.

The frontier touches each model's curve while that curve is still falling, above the model's converged loss. Near convergence each extra step lowers the loss by very little, so the last stretch of a long run buys less loss per FLOP than the same FLOPs spent on a larger model that is still early in training. The compute-efficient choice is to train a large model and stop well short of convergence. The optimal size grows with compute as

The batch grows as $C_{\min}^{0.24}$, because it tracks $B_{\text{crit}}$, which grows as the loss falls, and the number of steps as $C_{\min}^{0.03}$, almost flat. The three exponents add to 1.00, as they must: $C_{\min} = 6NBS$ with $B$ at the critical batch and $S = S_{\min}$, so the growth of the three factors has to multiply to the growth in compute.

The paper also derives the allocation from (1.6) and (1.4) instead of reading it off the frontier. Substituting $S_{\min} = C_{\min}/(6NB)$ into (1.6) and minimizing over $N$ at fixed compute gives

Equation (1.8) is the condition that makes the three exponents in (1.7) add to 1. The paper's values give $\alpha_C^{\min} \approx 0.054$ and $N \propto C_{\min}^{0.71}$, within a few percent of the measured 0.050 and 0.73. The exponent on $N$ is large because $\alpha_N$ is the smallest of the three: the loss responds least to $N$, so $N$ has to grow fastest.

The paper's Appendix A gives the prefactors. At $C_{\min} = 1$ PF-day the recommended run is a 1.3-billion-parameter model, a batch of 2 million tokens and 5,400 steps; $6NBS$ for those numbers is $8.4\times 10^{19}$ operations, one PF-day. Trained at $B_{\text{crit}}$, the run sees about $2\times 10^{10}$ tokens, each once.

# Kaplan's compute-efficient allocation (Table 6), C_min in PF-days
def allocate(C_min):
    N = 1.3e9 * C_min ** 0.73   # non-embedding parameters
    B = 2.0e6 * C_min ** 0.24   # batch size in tokens, tracks B_crit
    S = 5.4e3 * C_min ** 0.03   # minimum steps S_min
    D = 2.0e10 * C_min ** 0.27  # tokens, one epoch at B = B_crit
    return N, B, S, D

allocate(1)      # 1.3e9 params, 2.0e6-token batch, 5.4e3 steps, 2.0e10 tokens
allocate(10)     # N x5.4, B x1.7, S x1.07: almost all of it goes into N
6 * 1.3e9 * 2.0e6 * 5.4e3 / 8.64e19   # = 0.97, so C_min = 6 N B S checks out

At $C_{\min} = 1000$ PF-days the same formulas give a 200-billion-parameter model trained on 130 billion tokens, fewer tokens than parameters. The paper notes that in practice hardware constraints lead people to train smaller models for longer than this. The allocation was the paper's practical recommendation and the part later work revised; the last section covers how.

Where the trends must stop

Section 6.3 of the paper shows that its own trends contradict each other at large enough compute. Rule (4.4) says data should grow as $N^{0.74}$ to avoid overfitting, and with $N \propto C_{\min}^{0.73}$ that is $D \propto C_{\min}^{0.54}$. Compute-efficient training uses much less: one pass over data that grows as about $C_{\min}^{0.27}$. So at large compute a compute-efficient run is data-limited, and a data-limited run follows (1.2): the loss can fall no faster than $D^{-0.095} \propto C_{\min}^{-0.03}$. The compute law (1.3) falls as $C_{\min}^{-0.05}$, faster than that floor. The compute trend starts above the floor, and a steeper line eventually crosses a shallower one.

Past the crossing, (1.3) would predict a lower loss than the data can support, so the trends have to break at or before it. The paper puts the crossing at

and says these values are uncertain by an order of magnitude in either direction, depending on the exact exponents. As a check, (1.3) at $10^{4}$ PF-days gives $(3.1\times 10^{4})^{0.05} \approx 1.68$ nats. Drag the compute in Figure 7 toward the crossing and watch the gap between the two lines close.

The authors then offer a conjecture and label it as one. If a model cannot grow past $N^{*}$ without needing qualitatively more data, perhaps at that point it has extracted all the reliable information in the text, and $L^{*}$ would be a rough estimate of the entropy per token of natural language. Using the paper's figure of 4.3 characters per WebText2 token, 1.7 nats per token is 2.45 bits per token, or about 0.57 bits per character. The paper presents this as an interpretation of the crossing, and its measured trends showed no sign of levelling off in the range it tested.

What Chinchilla corrected, and what held up

GPT-3 followed in May 2020, four months after this paper. At 175 billion parameters it was, in its authors' words, ten times larger than any previous non-sparse language model, and about a hundred times larger than GPT-2. Its authors report that the loss-versus-compute trend from this paper continued for two more orders of magnitude, with small deviations from the predicted curve. The authors of the scaling paper compare their relations to the ideal gas law, which relates pressure, volume and temperature without reference to how individual molecules move; here loss is related to size, data and compute with little dependence on most architectural details.

The large models of the next two years were trained with the compute-efficient recipe's proportions: GPT-3, Jurassic-1, Gopher and Megatron-Turing NLG each saw about 300 billion training tokens, with 175 to 530 billion parameters. In 2022 DeepMind's Chinchilla paper re-measured the split and found parameters and tokens should grow about equally, each as roughly $C^{0.5}$, which works out to about 20 tokens per parameter. By that measure those models were undertrained: too large for the data they were shown. Figure 8 puts the two prescriptions side by side for the same compute. At $C_{\min} = 1000$ PF-days, Kaplan's formulas give 200 billion parameters on 130 billion tokens; 20 tokens per parameter with the same total FLOPs gives 38 billion parameters on 760 billion tokens.

The two recipes agree near $C_{\min} \approx 0.6$ PF-days, where both give a model of about 900 million parameters, and diverge from there. That agreement point lies inside the compute range the paper measured, and a small disagreement in a fitted exponent grows into a large one over several orders of magnitude of extrapolation.

Chinchilla's main explanation for the gap was Kaplan's learning-rate schedule. Every run used a cosine decay set for a fixed length, so a checkpoint taken partway through a long run still had a high learning rate and a loss above what a run tuned to end there would reach. Chinchilla also pointed out that most of Kaplan's runs were small, many under 100 million parameters. Two 2024 papers took the gap apart. Pearce and Song found the main cause was the non-embedding count from the second section, combined with fitting at small scale: where embeddings are a large share of all parameters, counting them or not changes the fitted exponent. Porian and colleagues identified the rest: leaving out the compute of the final output layer, a 3000-step warmup that was too long for the smallest models, and not re-tuning the optimizer at each scale. They reproduce Chinchilla's exponent (0.497 to 0.518) with a constant learning rate, so the schedule was not essential.

None of this changed the power laws. Later models moved further in Chinchilla's direction for a different reason: LLaMA trained its 7-billion-parameter model on 1.0 trillion tokens, about 150 per parameter, accepting more training compute for a smaller model that is cheaper to run. Later work kept the paper's forms (loss as a smooth power law in scale, a joint law over size and data, a critical batch size set by the loss, and an allocation read off a fitted frontier) and replaced its allocation exponent, 0.73, with about 0.5.