Language Models are Few-Shot Learners

Make it big enough and the prompt becomes the program.

GPT-3 is a 175-billion-parameter next-token predictor. Show it a few examples in the prompt and it does a new task, with no training. The surprise is that this ability barely exists in small models and arrives with scale.

What if adapting a language model to a new task meant writing a prompt instead of training a model?

By 2020 the recipe for a new language task was settled. Take a pretrained model, collect thousands to hundreds of thousands of labeled examples for your task, and fine-tune: keep training until the weights drift to fit it. It worked well. It also meant a fresh labeled dataset and a fresh model for every task you cared about, and the paper notes a subtler cost. A big model fine-tuned on a narrow dataset can latch onto quirks of that dataset and look better on the benchmark than it really is, because the comparison was never quite fair.

GPT-3 (from OpenAI) takes a different route. There is one model. You never change its weights. To do a task you write a prompt: a few worked examples followed by the question you want answered, all as plain text. The model reads it and continues it. The paper calls this in-context learning, and the headline finding is that it gets dramatically better as the model gets bigger. A small model barely does it. A 175-billion-parameter model does it well enough to rival fine-tuned systems on some tasks, having seen the task only in its prompt.

To see why that is surprising and how it works, we build up in order: what a language model computes, what it means to learn from the prompt, why scale was the lever, and what the giant could and could not do.

The cost of a model per task

Start with the thing GPT-3 is reacting against. The dominant paradigm, pre-train then fine-tune, gives you a task-agnostic architecture and then specializes it with gradient descent on task-specific data. The paper lays out three reasons that grates.

First, practicality. There is a wide range of useful language tasks, from fixing grammar to critiquing a short story, and most of them have no large labeled dataset sitting around. Collecting one for every new task is expensive, and you do it again each time.

Second, generalization. The more expressive the model and the narrower the fine-tuning set, the more room there is to exploit spurious correlations that hold on the benchmark and nowhere else. A model that scores at human level on a dataset may be much worse on the actual underlying task.

Third, the comparison to people. A human picks up a new language task from a short instruction or a couple of examples. We do not hand someone ten thousand labeled sentences to teach them to spot sarcasm. If the goal is broadly useful systems, the fine-tuning recipe is a strange fit for how the target ability is supposed to look.

GPT-3's answer is to push everything into the prompt and leave the weights alone. To understand what that buys, we first need to be precise about what the model is.

A language model predicts the next token

Strip away the mystique and a language model does one thing: given a stretch of text, it predicts what comes next. Text is first chopped into tokens (sub-word pieces, via byte-level byte-pair encoding inherited from GPT-2), so a sequence is a list of token ids $x_1, x_2, \dots, x_T$. The model is autoregressive: it factorizes the probability of the whole sequence into a product of next-token probabilities, each conditioned on everything before it.

Read it left to right. To score a sentence, score the first token, then the second given the first, then the third given the first two, and multiply. Each factor $p_\theta(x_t \mid x_{<t})$ is a full probability distribution over the vocabulary. The network emits one raw score per vocabulary token (a logit), and a softmax turns those scores into probabilities that sum to one. Training maximizes this likelihood on a giant pile of text, which is the same as minimizing the average cross-entropy, the negative log-probability the model assigned to each true next token:

That quantity is measured in nats per token. A loss near $\log V$ (for vocabulary size $V$) means the model is guessing uniformly at random. Driving it down means the model puts more of its probability on the token that actually came next. There is no task here and no labels beyond the text itself. The objective is "predict the next token," full stop.

Stripped to its core, the entire training objective is a few lines:

# the autoregressive objective: maximize the next-token likelihood
# x is a sequence of tokens [x_1, ..., x_T]
loss = 0
for t in range(1, T):
    logits = model(x[:t])          # read the prefix x_1..x_{t-1}, in one pass
    p = softmax(logits)            # distribution over the whole vocabulary
    loss += -log(p[x[t]])          # penalize surprise at the true next token
loss = loss / (T - 1)              # average cross-entropy (nats per token)

One detail to keep. The model conditions on a fixed-length window of recent tokens, called the context. GPT-3's context window is $n_\text{ctx} = 2048$ tokens for every model size. Anything you want the model to use, instructions, examples, the question, has to fit inside those 2048 tokens. That constraint is about to become load-bearing.

In-context learning: the prompt is the program

The rest of the paper turns on one move. Because the model continues whatever text you give it, you can specify a task inside the text and let the forward pass do the rest. Want translation? Write a few lines of the form English: ... French: ..., then a final English line with the French left blank, and let the model complete it. No gradients, no fine-tuning. The demonstrations are conditioning, not training data.

The paper names points on a spectrum by how many demonstrations $K$ the prompt contains:

• Zero-shot ($K=0$): only a natural-language instruction, no examples.
• One-shot ($K=1$): an instruction plus a single demonstration. This is closest to how you would brief a person.
• Few-shot ($K$ in the tens): as many demonstrations as fit the window, typically 10 to 100.

All three are the same operation: build a string, run one forward pass, read the completion. The only thing that changes is how much of the 2048-token window is spent on examples. This is a different thing from fine-tuning, where you would run backpropagation and change $\theta$. In-context learning never touches $\theta$.

The whole mechanism, in code, is string concatenation:

# few-shot prompt = K demonstrations, then the query (no completion)
prompt = ""
for (ctx, completion) in demos[:K]:        # K examples drawn from the task
    prompt += ctx + " " + completion + "\n"
prompt += query_ctx + " "                   # the example we want answered

answer = model.generate(prompt)            # forward pass only; weights frozen
# K = 0 is zero-shot, K = 1 is one-shot, K in 10..100 is few-shot

Why would this work at all? The framing the paper offers is meta-learning. During pre-training, predicting the next token across a trillion words forces the model to pick up a broad set of skills and patterns. Among those patterns are tasks that recur inside a single passage, like a list of translations or a Q-and-A format. The model learns to recognize and continue such patterns. At test time, your prompt is one more such pattern, and the model continues it. The paper calls the slow accumulation of skills during pre-training the outer loop, and the fast recognition-and-continuation inside a single forward pass the inner loop, the part they name in-context learning.

Why scale: loss follows a power law

In-context learning was not new. GPT-2 had shown flickers of it the year before, and the results were far behind fine-tuning. GPT-3 bets the ability grows with scale rather than staying a small-model curiosity. The reason to believe that comes from a separate line of work on how loss behaves as you scale up.

Kaplan et al. (2020) found that the validation loss of a Transformer language model falls as a clean power law in scale. Hold data and compute generous, and loss versus the number of (non-embedding) parameters $N$ obeys

and an analogous law holds for training compute $C$, with $L(C) = (C_c/C)^{\alpha_C}$, $\alpha_C \approx 0.050$, $C_c \approx 3.1\times 10^8$ petaflop/s-days. A power law is a straight line on log-log axes. The exponents are small, so loss falls slowly, but it keeps falling, predictably, with no sign of a wall. GPT-3's first result is that this line holds for two more orders of magnitude of compute past where Kaplan fit it, with only slight deviation.

Why does a loss curve matter for tasks? Because lower next-token loss tracks better performance on real language tasks. The paper's wager is that since in-context learning means absorbing skills into the weights and then deploying them at inference, it should ride the same smooth trend. Spend more compute, get lower loss, get a better in-context learner. The only way to test that was to build the biggest model anyone had built and measure.

The bet: few-shot ability emerges with scale

Across the 42 benchmarks the paper aggregates, accuracy in all three settings rises with model size, which is expected. The shape is the telling part. The few-shot curve climbs faster than the zero-shot curve, so the gap between them widens as the model grows. A small model gains almost nothing from having examples in its prompt. A large model gains a lot. That widening gap is the paper's evidence that larger models are better at learning in context, not only better across the board.

Concretely, on closed-book TriviaQA the full model scores 64.3% zero-shot, 68.0% one-shot, and 71.2% few-shot, the last of which beat the fine-tuned state of the art in the same closed-book setting at the time. On CoQA it reaches 81.5 / 84.0 / 85.0 F1 across the three settings. The examples in the prompt are worth real points, and they are worth more the bigger the model.

The model: 175 billion parameters

The architecture is deliberately boring. GPT-3 reuses GPT-2's Transformer almost unchanged: the same pre-normalization, the same modified initialization, the same reversible byte-level tokenizer. The one structural change is that GPT-3 alternates dense attention with locally-banded sparse attention in its layers, following the Sparse Transformer. The point of the paper is scale, not a new architecture, so they hold the design fixed and turn the size dial.

They train eight models spanning three orders of magnitude, from 125 million to 175 billion parameters, so they can watch behavior as a function of size. The widths and depths grow together. The feed-forward layer is always four times the model width, $d_\text{ff} = 4\,d_\text{model}$, and the per-head dimension stays near 128. Every model uses the same 2048-token context window and trains on the same 300 billion tokens.

The full model has 96 layers, a width of 12,288, and 96 attention heads of dimension 128. Its training cost, from the paper's own accounting, was about 3,640 petaflop/s-days (one petaflop/s-day is a day of computing at $10^{15}$ operations per second), which is roughly $3.1\times 10^{23}$ floating-point operations. That figure has a clean back-of-envelope form: training takes about $6$ flops per parameter per token,

where $N$ is parameters and $D$ is training tokens. The factor of 6 is 2 for the multiply-and-add in the forward pass times a factor of 3 for the backward pass. One detail the paper flags about its own data: the 300 billion training tokens were not sampled in proportion to corpus size. Higher-quality sources were upweighted, so filtered Common Crawl (410 billion tokens) was seen less than once on average while Wikipedia (3 billion tokens) was seen about 3.4 times.

What it actually does

The most striking results are the synthetic tasks the authors invented specifically to test on-the-fly reasoning, things very unlikely to sit verbatim in the training data. Arithmetic is the cleanest. They asked GPT-3 questions like Q: What is 48 plus 76? A: and scored exact-match on 2,000 random instances per task.

In the few-shot setting the full 175B model gets 100% on 2-digit addition, 98.9% on 2-digit subtraction, and 80.4% on 3-digit addition. Accuracy falls as the digits grow (about 25 to 27% on 4-digit, 9 to 10% on 5-digit) and 2-digit multiplication sits at 29.2%. None of this is memorization: they searched the training set for the test problems and found matches for under 1% of them, and the model's wrong answers look like arithmetic slips, such as forgetting to carry a 1.

The arithmetic also shows the scale effect at its sharpest. There is a large jump from the second-largest model (13B) to the full 175B. The 13B model solves 2-digit addition and subtraction only about half the time and scores under 10% on everything else. The capability appears near the top of the size ladder, which is exactly the pattern the paper is pointing at.

The same story repeats elsewhere. GPT-3 does word unscrambling, uses a freshly-defined nonsense word in a sentence after seeing it once, and translates, with one-shot and few-shot well ahead of zero-shot. And it generates news articles people cannot reliably flag as machine-written. In the paper's human evaluation, participants distinguished GPT-3 175B's ~200-word articles from real ones with mean accuracy of about 52%, where 50% is chance, down from about 86% on a deliberately weak control model. Longer ~500-word articles gave the same result, still near chance.

Contamination and limits

A model trained on much of the web raises an obvious worry: maybe it does well on a benchmark because the benchmark was in the training data. The paper takes this seriously and runs a systematic contamination study, building a "clean" version of each benchmark with train-test overlaps removed and re-scoring on it. They are candid that a filtering bug let some overlaps through and that retraining the model was too expensive to redo. For most benchmarks the clean-subset performance barely moved, so contamination had little effect. A few were flagged: results on PIQA and Winograd carry an asterisk in the paper, and several Wikipedia language modeling benchmarks were dropped entirely because they were almost fully contained in the training set.

The limitations are real and the authors list them plainly. Generated text still loses coherence and contradicts itself over long passages. Some task types stay near chance even few-shot, in particular "comparison" tasks like deciding whether one sentence entails another (ANLI) or whether a word is used the same way in two sentences (WiC). The authors suspect the autoregressive, one-direction-only objective hurts here, since these tasks reward looking back and forth, and conjecture a bidirectional model at this scale would do better when fine-tuned. The model is expensive to run, its sample efficiency during pre-training is poor (it sees far more text than a human ever does), and a fundamental open question remains: whether few-shot prompting teaches a genuinely new task at inference time or just surfaces one already learned during pre-training. The paper does not claim to know.

Set the caveats aside and the through-line is simple. There is no new architecture and no clever training objective. There is the same next-token prediction, run at a scale nobody had tried, and an ability that small models lack shows up at the top of the curve. Adapting the model stopped meaning "collect a dataset and train" and started meaning "write a prompt." That shift, more than the parameter count, is what the years since have been built on.