GPT-3: Language Models are Few-Shot Learners

Scale a language model to 175 billion parameters and it can do a new task from a few examples in its prompt.

GPT-3 is a next-token predictor built on GPT-2's design and about 117 times larger. The paper never trains it on a task. It writes a few solved examples into the prompt, reads what the model writes next, and measures how well that works at eight model sizes from 125 million to 175 billion parameters.

In 2020, getting a language model to do a task meant training it on thousands of labeled examples of that task. The paper tests how much of that a prompt can replace, and the answer depends on model size.

Why not fine-tune

The standard recipe in 2020 was pre-train, then fine-tune. You start from a model pre-trained on generic text, collect a labeled dataset for your task (the paper says thousands to hundreds of thousands of examples), and keep training on it until the weights fit the task. BERT and T5 work this way, and fine-tuned models held most benchmark records.

The paper gives three reasons to want something else:

1. Every task needs its own dataset. Useful language tasks range from correcting grammar to critiquing a short story, and most of them have no large labeled dataset. You would have to build one per task.
2. Narrow data invites shortcuts. In the paper's words, the potential to exploit spurious correlations in training data grows with the expressiveness of the model and the narrowness of the training distribution. A large model fine-tuned on one benchmark's training split can pick up patterns that hold on that benchmark and nowhere else, so it can reach human-level scores while being worse at the underlying task.
3. People do not learn this way. A person picks up most language tasks from a one-line instruction ("tell me whether this sentence describes something happy or sad") or a couple of examples.

GPT-3 takes the other route. There is one pre-trained model, and its weights never change for a task. From the abstract: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." The rest of this page covers what the model computes, how a task is written into a prompt, why the authors expected size to matter, the eight models they trained, what the results look like at each size, and where the approach fails.

What the model computes

GPT-3 reads text as tokens, sub-word pieces produced by the byte-level byte-pair encoding it inherits from GPT-2. The vocabulary has 50,257 tokens; a common word is one token, and a rare word is split into several. A piece of text becomes a list of token ids $x_1, \dots, x_T$.

The model is autoregressive: at every position it outputs a probability distribution over all 50,257 tokens for what comes next, given everything before that position. The probability of a whole sequence is the product of those next-token probabilities:

The symbol $\theta$ stands for the weights, and $x_{<t}$ is shorthand for $x_1, \dots, x_{t-1}$. The network produces one raw score (a logit) per vocabulary token, and a softmax turns the 50,257 scores into probabilities that sum to 1. Pre-training maximizes (1) on a large corpus, which is the same as minimizing the average negative log-probability of each true next token, the cross-entropy:

With natural logarithms the unit is nats per token. Take the sentence in Figure 1, "The cat sat on the mat", as six tokens, and suppose the model gives the true tokens probabilities 0.41, 0.22, 0.34, 0.46, 0.63 and 0.31. The six losses $-\ln p$ are 0.89, 1.51, 1.08, 0.78, 0.46 and 1.17, and their mean is 0.98 nats per token. A model that spread its probability evenly over the vocabulary would score $\ln 50{,}257 \approx 10.83$ nats on every token. There are no labels in (2) other than the text itself: the target at each position is the next token.

In code, one pre-training step on one sequence looks like this. The causal mask in the attention layers stops each position from seeing later tokens, so a single forward pass computes all $T-1$ predictions at once (the code takes the first token as given):

# next-token cross-entropy on one training sequence x of T token ids
# a causal mask lets one forward pass produce every position at once
logits = model(x[:-1])              # (T-1, 50257): one row per prefix
logp = log_softmax(logits, dim=-1)  # each row: a distribution over the vocab
nll = -logp[range(T - 1), x[1:]]    # surprise at each true next token
loss = nll.mean()                   # eq. (2), in nats per token
loss.backward()                     # pre-training: the gradient reaches every weight

The model conditions on at most $n_\text{ctx} = 2048$ tokens, its context window, and this is the same for every GPT-3 size. Anything the model should use when it answers (an instruction, examples, the question) has to fit in those 2048 tokens.

Putting the task in the prompt

A model trained on (2) continues whatever text it is given. If the text is a list of solved problems followed by an unsolved one, the most probable continuation is the solution. The paper uses the arithmetic format "Q: What is 48 plus 76? A: 124." A few-shot prompt with $K=3$ demonstrations looks like this:

Q: What is 48 plus 76?
A: 124

Q: What is 17 plus 58?
A: 75

Q: What is 93 plus 9?
A: 102

Q: What is 65 plus 29?
A:

The model reads the whole string in one forward pass and the evaluation takes its continuation, here ideally "94", as the answer. No gradient is computed and no weight changes. The paper calls this in-context learning and names three settings by the number of demonstrations $K$:

• Zero-shot ($K=0$): a natural-language instruction only, such as "Translate English to French:" followed by the word to translate.
• One-shot ($K=1$): the instruction plus one demonstration. The paper singles it out because it is how tasks are often explained to people, for example to crowd workers.
• Few-shot: as many demonstrations as fit in the window, typically $K$ from 10 to 100.

All three are the same computation on different strings, and the only resource they compete for is the 2048-token window. Each arithmetic demonstration above is about a dozen tokens, so more than a hundred would fit. A reading comprehension demonstration carries a passage and runs to hundreds of tokens, so only a handful fit. The paper's choices follow that: K=5 for CoQA and QuAC, which come with passages, and K=64 for closed-book TriviaQA.

Reading an answer off the model takes one more decision per task type (Section 2.4). For multiple-choice tasks the model does not generate anything: each option is appended to the prompt, the log-probabilities of its tokens are averaged, and the highest-scoring option is the prediction. Averaging per token keeps long options from losing just because they have more tokens. For free-form answers the model generates with beam search:

# evaluation: weights frozen, one forward pass per candidate, no .backward()
prompt = "".join(ctx + target + "\n\n" for ctx, target in demos[:K]) + query

# multiple choice (HellaSwag, PIQA, ...): score each option, keep the best
def score(option):
    lp = token_logprobs(model, prompt + option)[-n_tokens(option):]
    return lp.sum() / len(lp)       # per-token average, so length is fair
pred = max(options, key=score)

# ARC, OpenBookQA, RACE: subtract how likely the option is anyway
#   log P(option | prompt) - log P(option | "Answer: ")
# free-form (TriviaQA, arithmetic, translation): beam search, width 4,
#   length penalty 0.6, then exact match, F1 or BLEU

Why should next-token training produce this ability? The paper frames it as meta-learning with two loops. Pre-training is the outer loop: over hundreds of billions of tokens, gradient descent builds up broad skills and pattern recognition in the weights. Web text is full of repeated formats (question-and-answer pages, lists, translations side by side, code with comments), and predicting the fifth item of such a list is easier once the first four have been used to work out the format. The inner loop is what happens inside one forward pass on one prompt, where the model adapts to the format in its activations while the weights stay fixed. The paper reserves the name in-context learning for this inner loop.

That framing suggests, as an interpretation and not a finding of the paper, why the ability should grow with size: a larger model has absorbed more of the formats that recur in its training text, so a few examples are more often enough to identify which one the prompt is using. The paper itself leaves open whether few-shot prompting teaches a new task or identifies one learned in pre-training; the limits section returns to this.

Why the authors expected scale to help

In-context learning was not new. GPT-2 had tried it a year earlier, and the paper describes its results as far inferior to fine-tuning. GPT-3 is a test of a hypothesis: that this ability, like pre-training loss, improves steadily with scale.

The reason to expect steady improvement came from Kaplan et al. (2020), who found that the test loss of Transformer language models falls as a power law in scale. For a model with $N$ non-embedding parameters trained on enough data,

and for compute $C$ spent on the compute-efficient frontier (the best model size for each budget),

A PF-day, or petaflop/s-day, is $10^{15}$ operations per second for one day, which is $8.64\times 10^{19}$ operations. The exponents are small. By (3), doubling the parameter count multiplies the loss by $2^{-0.076} \approx 0.949$, a 5.1% drop. By (4), ten times the compute multiplies it by $10^{-0.05} \approx 0.891$, a 10.9% drop. Taking the logarithm of (4) turns it into a straight line with slope $-\alpha_C$ on log-log axes, which is how Figure 3 draws it:

The eight GPT-3 models span a factor of about 1,400 in compute (3,640 / 2.60), about three orders of magnitude. Along the dashed line that factor is worth $1400^{-0.05} \approx 0.70$, a 30% lower loss for 175B than for 125M. Counting six extra-small models down to 100,000 parameters that appear only in its loss plot, the paper reports that Kaplan's trend continues for two more orders of magnitude of compute with only small deviations.

Loss only matters here if it predicts task performance. The paper's argument for the link is empirical: the results sections show that the improvements in loss come with consistent gains across a broad range of tasks, rather than only from modeling details of the training corpus. A way to see why the two should be linked, which is our gloss and not an argument the paper makes: every gap in grammar, facts or reasoning costs probability on some next token somewhere, so a lower loss requires more of those skills.

Eight models, one recipe

The architecture is GPT-2's, a stack of Transformer decoder layers with the same modified initialization, pre-normalization (layer normalization at the input of each sub-block rather than after it) and reversible tokenizer. The one change is that layers alternate between dense attention and locally banded sparse attention, as in the Sparse Transformer. Holding the design fixed makes size the only variable. The authors trained eight models spanning three orders of magnitude:

Every model has a 2048-token context window and trains on the same 300 billion tokens. The feed-forward layer inside each block is always four times the model width, $d_\text{ff} = 4\,d_\text{model}$, so 49,152 for the 175B model. The per-head dimension ranges from 64 to 128 (128 in most of the larger models, 80 in 2.7B). The paper prints 5140 for the 13B width; the table shows 5120, and the Provenance panel at the end explains why.

The 175B row is enough to recover the parameter count. Each layer has four $d \times d$ attention matrices (query, key, value, output) and two feed-forward matrices of size $d \times 4d$, so about $12d^2$ weights. With $d = 12{,}288$ and 96 layers:

The token embedding adds $50{,}257 \times 12{,}288 \approx 0.62$ billion and the position embedding $2048 \times 12{,}288 \approx 0.03$ billion, for a total of about 174.6 billion, the parameter count (174,600M) that Table D.1 uses for its compute estimate.

Training compute follows from a rule of thumb stated in the caption of Table D.1. In the forward pass each token costs one multiply and one add per parameter, 2 operations; the backward pass costs about twice the forward, which the paper accounts for with a multiplier of 3 on the forward cost. That is 6 operations per parameter per token:

In (5), $N$ is the parameter count and $D$ the number of training tokens. Dividing by $8.64\times 10^{19}$ gives 3,640 PF-days, the value in Table D.1: a machine sustaining one petaflop/s would need about ten years. With the rounded 175 billion, (5) gives $3.15\times 10^{23}$. The same rule gives 2.60 PF-days for the 125M model, and it ignores the attention operation itself, which the paper estimates at under 10% of the total for these models. Generating one token with the 175B model costs the forward-pass share, 2 operations per parameter, or about $3.5\times 10^{11}$ operations.

The optimizer is Adam with $\beta_1 = 0.9$, $\beta_2 = 0.95$ and $\epsilon = 10^{-8}$, gradient norm clipped at 1.0 and weight decay 0.1 (Appendix B). The learning rate warms up linearly over the first 375 million tokens, then follows a cosine decay to 10% of its peak over 260 billion tokens; the peak is $0.6\times 10^{-4}$ for 175B and $6\times 10^{-4}$ for Small. The batch size starts at 32 thousand tokens and ramps up to its full value over the first 4 to 12 billion tokens. At 3.2 million tokens per batch, 300 billion tokens is roughly 94,000 optimizer steps.

The 300 billion training tokens come from five sources (Table 2.2). The largest is Common Crawl, 45 TB of compressed text from 2016 to 2019 ("nearly a trillion words") that filtering reduced to 570 GB, counted in Table 2.2 as 410 billion tokens. The authors add four smaller datasets they consider higher quality: WebText2, two book corpora and English Wikipedia. The sources hold 499 billion tokens between them, and the authors did not sample them in proportion to their size.

The paper describes the trade as accepting a small amount of overfitting on the repeated sources in exchange for higher-quality training data. Even the most repeated source, Wikipedia, is seen fewer than four times, and most Common Crawl text is never seen.

Few-shot gains by model size

The paper evaluates all eight models in all three settings on over two dozen datasets. Its aggregate over 42 benchmarks (Figure 1.3, which the authors say should not be read as a rigorous benchmark in itself) has one main feature: zero-shot accuracy improves steadily with size, few-shot accuracy improves faster, so the gap between them widens. The authors read that as larger models being better at in-context learning, not only better overall.

The aggregate is a chart without a table, but Appendix H lists every task at every size in every setting. Figure 5 plots five of those rows. Look at the few-minus-zero gap as you move the size slider, and at how differently the tasks behave.

On WebQs, a question-answering set, the gap grows from 3.7 points at 125M to 27.1 at 175B (41.5% few-shot against 14.4% zero-shot). TriviaQA is closed-book here: the model answers trivia questions with no retrieved documents. The 175B model scores 64.3% zero-shot, 68.0% one-shot and 71.2% few-shot, and the few-shot number was higher than the fine-tuned state of the art in the same closed-book setting. The TriviaQA gap is largest at 13B (15.7 points) and narrower at 175B (6.9), because zero-shot jumps from 41.8% to 64.3% in that last step.

LAMBADA asks for the last word of a paragraph. Zero-shot, a language model does not know the answer must be a single word that ends the sentence, so it spreads probability over other plausible continuations. The few-shot prompt frames it as fill-in-the-blank ("Alice was friends with Bob. Alice went to visit her friend ____. → Bob"). That framing cuts the smallest model's accuracy from 42.7% to 22.0% and raises the 175B model's from 76.2% to 86.4%. The paper notes that one-shot is always worse than zero-shot on this task and suggests the models need several examples to recognize the pattern.

HellaSwag, choosing the most plausible ending of a short story or set of instructions, improves with size, but the three settings stay within 1.4 points of each other at every size (78.9, 78.1 and 79.3 at 175B). ANLI Round 3, a three-way inference task (does one sentence follow from, contradict, or neither, another), stays within a few points of the 33% chance level for every model and setting except 175B few-shot, which reaches 40.2%.

On CoQA, conversational reading comprehension scored in F1, the 175B model reaches 81.5, 84.0 and 85.0 in the three settings. The aggregate's widening gap is an average over rows like these, which range from a gap that grows to 27 points (WebQs), to one that changes sign (LAMBADA), to none (HellaSwag).

Arithmetic

To test tasks that are unlikely to appear verbatim in the training data, the authors built ten arithmetic tasks: addition and subtraction with 2 to 5 digits (2D+ to 5D−), 2-digit multiplication (2Dx), and composite one-digit expressions such as "What is (6+2)*4?" (1DC). Each task has 2,000 random problems, phrased as in the prompt above, and a problem counts as solved only if the answer matches exactly.

Few-shot, the 175B model gets 100% on 2-digit addition, 98.9% on 2-digit subtraction, 80.4% on 3-digit addition and 94.2% on 3-digit subtraction. Accuracy falls with more digits: 25.5% and 26.8% for 4-digit addition and subtraction, 9.3% and 9.9% for 5-digit. 2-digit multiplication reaches 29.2% and the composite expressions 21.3%.

Model size matters more than the prompt here. Few-shot, the 13B model solves 2-digit addition 55.5% of the time and 2-digit subtraction 52.4%, about half, and every other task less than 10% of the time; below 13B no few-shot score exceeds 14%. The 175B model with no demonstrations at all beats the few-shot score of every smaller model on nine of the ten tasks (76.9% zero-shot on 2-digit addition against 55.5% for 13B few-shot); the exception is 1DC, 9.75% against 9.95%. The paper reports this jump without proposing a mechanism. An interpretation, which is ours and not the paper's, is capacity: the smaller models never form a procedure for multi-digit carrying, and the largest one does, so the ability looks sudden even though the loss curve in Figure 3 is smooth.

The authors checked for memorization on the 3-digit problems, searching the training data for both <NUM1> + <NUM2> = and <NUM1> plus <NUM2>. They found 17 of the 2,000 addition problems (0.8%) and 2 of the 2,000 subtraction problems (0.1%). The wrong answers are informative too: the model often makes mistakes such as not carrying a 1, which suggests it is attempting the computation rather than looking answers up.

The same design produced word-manipulation tasks. Few-shot, the 175B model removes random characters inserted into a word 66.9% of the time, undoes a cyclic shift of a word's letters 38.6% of the time, and solves anagrams that keep the first and last letters fixed 15.1% of the time. No model can spell a word backwards.

Generated news articles

The authors also measured whether people can tell GPT-3's writing from human writing. They took 25 article titles and subtitles from newser.com (the real articles average 215 words), had each model write the article, and showed around 80 US-based participants per model a mix of real and generated articles. Accuracy is the fraction of each participant's non-neutral judgments that were correct, so 50% is chance. The authors also generated articles with a deliberately bad control model, to show the score for text that is easy to spot.

Articles from the control model were identified 86% of the time and those from the 175B model 52% of the time, with a confidence interval of 49% to 54% that includes chance. Participants also spent more time per article as the model got larger. A follow-up with longer articles (12 Reuters stories, about 500 words each) found the same: 88% for the control model and 52% for 175B.

Contamination and limits

A model trained on a large share of the web may have seen the benchmarks. The authors tried to remove every overlap between the training data and the development and test sets before training, but a bug in the filter let some through, and retraining was too expensive. Instead, for each benchmark they built a clean subset with every overlapping example removed and re-scored the model on it. Performance on most benchmarks changed negligibly. PIQA and Winograd showed some evidence of contamination and carry an asterisk in the results. The four Wikipedia language-modeling benchmarks from GPT-2 and the Children's Book Test were almost entirely in the training data and are not reported. LAMBADA had substantial real overlap, yet its clean subset scored within 0.5% of the full set.

Section 5 lists the limitations:

• Long text. Samples repeat themselves at the document level, lose coherence over long passages and contradict themselves.
• Comparison tasks. Tasks that compare two sentences or snippets stay weak even in the one- and few-shot settings: WiC (is a word used with the same meaning in two sentences) and ANLI (does one sentence imply another). The authors attribute this to the autoregressive objective, which reads left to right only, and conjecture from past work that a large bidirectional model would be stronger at fine-tuning than GPT-3.
• Sample efficiency. Pre-training uses much more text than a person sees in a lifetime.
• Cost. A 175-billion-parameter model is expensive and inconvenient to run for inference.
• What few-shot does. The paper cannot say whether few-shot prompting learns a task from scratch at inference time or identifies a task learned in pre-training, and places tasks on a spectrum: unscrambling words and defining nonsense words seem likely to be learned on the spot, while translation must have been learned in pre-training.

Two follow-ups on this site answer questions the paper left open. Chinchilla (2022) argued that GPT-3 and the other large models of its time, most trained on about 300 billion tokens, were significantly undertrained for their compute budgets. InstructGPT (2022) fine-tuned GPT-3 on human-written demonstrations and human preference rankings so that it follows instructions, and chain-of-thought prompting (2022) put worked reasoning steps into the few-shot demonstrations.