InstructGPT: Training language models to follow instructions with human feedback

Fine-tuned on human feedback, a 1.3B model's answers are preferred to the 175B GPT-3's.

GPT-3 was trained to continue internet text. InstructGPT fine-tunes it in three steps: imitate answers written by labelers, learn a reward model from their rankings, then optimize the model against that reward with reinforcement learning.

The recipe used about 13k written demonstrations, 33k ranked prompts, a 6B reward model, and PPO with a penalty (β = 0.02) on drifting from the fine-tuned starting point, plus a pretraining term (γ = 27.8) that limits lost benchmark accuracy.

What GPT-3 does with an instruction

GPT-3 is trained on one objective: given a stretch of internet text, predict the next token. A question typed into it is treated as the start of a document, and the model writes whatever usually comes next in documents that start that way. Figure 8 of the paper shows what that looks like. The prompt is a short Python function that computes a binomial coefficient, followed by "What is the purpose of the list C in the code below?" The 175B GPT-3 replies:

A. to store the value of C[0]
B. to store the value of C[1]
C. to store the value of C[i]
D. to store the value of C[i - 1]

That is a plausible continuation (quiz questions on the internet are followed by answer options), and it answers nothing. The 175B InstructGPT, given the same prompt, writes a paragraph explaining that C stores binomial coefficient values as the loop runs; its explanation is not quite right either, which the paper points out. The prompts in that figure were picked to show the behavior, the outputs were not, and the paper notes GPT-3 does answer this particular question about half the time.

The authors call the language-modeling objective misaligned: predicting the next token on a web page is a different objective from "follow the user's instructions helpfully and safely." They make "aligned" concrete with three properties. A helpful model does the task the user intended, including tasks given implicitly by a few examples or a pattern like Q: ... A:. An honest model does not make up facts. A harmless model does not produce output that causes harm. Human labelers judge all three, and the rest of the method converts their judgments into a training signal.

The prompts come from what people sent to early InstructGPT models on the OpenAI API (over 96% English), plus prompts the labelers wrote to cover gaps. The paper's Table 1 breaks the API prompts down by use case:

• generation 45.6%, open QA 12.4%, brainstorming 11.2%, chat 8.4%, rewrite 6.6%
• summarization 4.2%, classification 3.5%, other 3.5%, closed QA 2.6%, extraction 1.9%

Open-ended generation and brainstorming together are about 57% of the prompts; classification and question answering, the tasks academic benchmarks are built around, are about 18%, a mismatch that shows up again when the results compare InstructGPT with models trained on benchmark-style instructions.

The training method was already published. Section 3.1 says it follows Ziegler et al. (2019) and Stiennon et al. (2020), who fine-tuned language models from human preferences on stylistic continuation and summarization. Both build on Christiano et al. (2017), which learned a reward function from human comparisons for Atari games and simulated robots. InstructGPT applies the recipe to the broad, messy distribution of API prompts, and reports that a 1.3B model trained this way produces outputs labelers prefer to those of the 175B GPT-3, despite having over 100× fewer parameters. The two models have the same architecture and differ only in the fine-tuning on human data.

Three steps, three datasets

"Be helpful" cannot be written as a loss function, so the method learns it in three steps. Each step produces a model the next step starts from. Click through them:

Each step has its own dataset, drawn from the same prompt pool (Section 3.2):

• SFT set, about 13k training prompts (API and labeler-written), each with a demonstration a labeler wrote.
• RM set, about 33k training prompts (API and labeler-written), each with several model outputs that a labeler ranked.
• PPO set, about 31k training prompts, from the API only, with no labels. The reward model supplies the signal.

The labels come from about 40 contractors hired through Upwork and Scale AI. Given the same pair of outputs, two of the training labelers pick the same winner 72.6 ± 1.5% of the time, and held-out labelers (people who produced no training data) agree with each other 77.3 ± 1.3% of the time. Roughly one pairwise judgment in four is disputed between two labelers, which sets a rough ceiling on how well any model can predict a single labeler's choice.

Step 1: supervised fine-tuning

Supervised fine-tuning (SFT) is ordinary supervised learning. Each training example is a prompt followed by the labeler's demonstration, and GPT-3 is trained on that text with the same next-token cross-entropy loss it was pretrained with. After training, given a prompt, the model writes something shaped like a labeler's answer.

The paper trains for 16 epochs with a cosine learning-rate decay and residual dropout of 0.2 (learning rate 9.65e-6 at batch size 32 for the 1.3B and 6B models, 5.03e-6 at batch size 8 for 175B). The SFT models overfit on validation loss after one epoch. Training longer still raised both the reward model's score and human preference ratings, so the authors kept going and selected the final checkpoint by reward-model score on the validation set instead of by validation loss.

Appendix C.3 specifies the model step 3 starts from: a separate SFT variant, GPT-3 fine-tuned for 2 epochs on the demonstrations with 10% pretraining data mixed in, which is also the reference the policy is held near. The 16-epoch model is the baseline the paper reports as "SFT".

Why not stop after step 1 and collect more demonstrations? A demonstration is expensive (a labeler writes a complete, good answer) and it only teaches the model to imitate what labelers write. To push past imitation, the model needs a signal saying one answer is better than another, including answers no labeler wrote, and steps 2 and 3 build that signal.

Step 2 data: rankings

Judging is cheaper than writing. A labeler can read two answers and pick the better one in a fraction of the time it takes to write the better one, and can recognize quality in an answer they could not have produced. InstructGPT exploits this: for each prompt in the RM set, the model generates between $K=4$ and $K=9$ responses and a labeler ranks them from best to worst.

One ranking of $K$ items fixes the winner of every pair among them, so it yields $\binom{K}{2}$, or $K(K-1)/2$, pairwise comparisons: 6 at $K=4$, 15 at $K=6$, 36 at $K=9$. Drag $K$ to see how fast the count grows:

The obvious way to use these pairs is to shuffle them all into one dataset of independent examples. The authors tried that, and the reward model overfit within a single pass over the data. The pairs from one prompt are highly correlated: at $K=9$, each response appears in 8 of the 36 pairs, so after shuffling the model takes up to $K-1$ separate gradient steps on the same response within one epoch, which amounts to repeating data.

The paper instead treats all $\binom{K}{2}$ comparisons from one prompt as a single batch element. The reward model runs one forward pass per response ($K$ passes, 9 at $K=9$) and scores every pair from those same outputs, where the shuffled version needs $\binom{K}{2}$ passes (36). It is cheaper, and because it stops the overfitting it reaches much better validation accuracy and log loss. Appendix C.2 gives the rest of the recipe: batches of 64 prompts (so up to 64 × 36 = 2,304 comparisons per batch), ties dropped, and one epoch over the RM set at learning rate 9e-6 with a cosine schedule, because a second epoch quickly overfit the training data.

Step 2 model: the reward model

A pile of "A beat B" judgments cannot be optimized against directly, because the policy in step 3 will write new responses that appear in no judgment. So step 2 trains a network to predict the judgments. The reward model is a GPT-3 network with its final unembedding layer (the layer that turns the last hidden state into next-token logits) replaced by a projection to a single number. It reads a prompt $x$ and a response $y$ and outputs a scalar $r_\theta(x, y)$, meant to be high when a labeler would like the response.

Section 3.5 describes starting this network from the SFT model. Appendix C.2 says the reward model actually used was initialized from a 6B GPT-3 fine-tuned on a set of public NLP datasets, "mostly for historical reasons", and that starting from GPT-3 or the SFT model gave similar results.

To connect a scalar score to a pairwise choice, the paper uses the model Stiennon et al. used: the difference in rewards is the log odds that a labeler prefers one response to the other. With $y_w$ the preferred ("winning") response and $y_l$ the other one:

The function $\sigma$ is the logistic, $1/(1+e^{-z})$. This is the Bradley-Terry model (1952) for paired comparisons, although the paper never uses that name. Only the gap between the two scores matters. If the reward model scores response A at 5 and response B at 3, the gap is 2 and the predicted preference for A is $\sigma(2) \approx 0.88$, odds of $e^2 \approx 7.4$ to 1. Equal scores give 0.5, a coin flip. A gap of 4 gives 0.98. Drag the gap and watch the predicted vote split:

Training minimizes the negative log-likelihood of the labelers' choices under that model, averaged over the $\binom{K}{2}$ pairs of each prompt. This is equation (1), a loss to minimize:

$D$ is the dataset of human comparisons. Each pair contributes $-\log\sigma(r_w - r_l)$: 0.13 when the gap is +2 (right order, confident), 0.69 when the gap is 0, and 2.13 when the gap is −2 (wrong order, confident). A backwards pair costs about 17 times as much as a correctly ordered one at the same distance.

The next figure works equation (1) on one prompt. A labeler ranked four responses A > B > C > D, and the reward model's current scores put B above A and D above C, so three of the six pairs are in the wrong order and the loss is 0.678. Press Train to run gradient descent on the loss; drag a dot to start from other scores.

Two things are visible in the figure. First, the wrong pairs move fastest. The derivative of $-\log\sigma(g)$ with respect to the gap $g$ is $-(1-\sigma(g))$, close to −1 for a badly wrong pair and close to 0 for a pair already ordered with a large gap, so the gradient concentrates on the mistakes. Second, the dashed mean never moves. Each pair pushes its winner's score up by exactly the amount it pushes its loser's score down, so the gradients on the four scores sum to zero.

The zero-sum gradient follows from a general property of equation (1): adding the same constant to every reward leaves every gap, and so the loss, unchanged. The loss therefore never determines where zero is, and the paper fixes the offset by hand. Before reinforcement learning, it adds a bias to the reward model so that the labeler demonstrations score a mean of 0. A positive reward then means "better than a typical labeler-written answer, as far as the reward model can tell".

With a single consistent ranking, as in the figure, the loss keeps falling as the gaps widen and never reaches zero. Real labels disagree. If labelers prefer $y_w$ to $y_l$ a fraction $p$ of the time, the expected loss on that pair is smallest where $\sigma(\text{gap}) = p$, so disagreement keeps the fitted gaps finite. In code, one batch element looks like this:

# reward-model loss for ONE prompt (one batch element), equation (1)
resps  = ranked_responses(prompt)      # K in 4..9, labeler order, best first
scores = reward_model(prompt, resps)   # K forward passes -> K scalars
pairs  = [(w, l) for w in range(K) for l in range(w + 1, K)]  # C(K,2)
loss   = 0
for w, l in pairs:                     # every pair reuses the same K scores
    loss += -log_sigmoid(scores[w] - scores[l])
loss = loss / len(pairs)               # the 1 / C(K,2) average
loss.backward()                        # gradient flows into the RM only

How well does it predict people? The paper split the labelers into 5 groups and trained reward models on 4 of them at a time. Those models predicted the preferences of their own training labelers 72.4 ± 0.4% of the time and of the held-out group 69.6 ± 0.9% of the time, close to the 72.6% rate at which two training labelers agree with each other.

The reward model is 6B parameters for every policy size, including the 175B one, and Appendix C.2 gives two reasons for not using a 175B reward model. Training 175B reward models was more unstable, which made them less suitable as the starting point for the PPO value function (a second network PPO trains to estimate expected reward, initialized here from the reward model). And a 175B reward model plus a 175B value function would have greatly increased the compute cost of step 3. In preliminary experiments 6B reward models were stable across a wide range of learning rates and led to equally strong PPO models.

Step 3: PPO against the reward

With a reward model in hand, step 3 is reinforcement learning. The language model is now called the policy, $\pi^{\mathrm{RL}}_\phi$, with weights $\phi$. The environment is a bandit: it shows the policy a random API prompt, the policy writes one complete response, the reward model scores it, and the episode ends. There is one decision per episode and no follow-on state, unlike a game where each move changes the board. The optimizer is PPO (Schulman et al., 2017), used, like the rest of the recipe, "following Stiennon et al. (2020)".

Maximizing $r_\theta$ alone would be a mistake, for reasons the next section covers. The objective InstructGPT maximizes is equation (2):

The first expectation is over prompts $x$ from the PPO set and responses $y$ sampled from the current policy. Inside it, the reward is added and a log-ratio term is subtracted with coefficient $\beta$. The second expectation, weighted by $\gamma$, is over pretraining text and is added; the alignment-tax section covers it, and for the models the paper calls plain "PPO", $\gamma = 0$. $\pi^{\mathrm{SFT}}$ is the frozen SFT-init model from Appendix C.3.

The log-ratio is charged per token. A response is a sequence of tokens, so $\log \pi(y \mid x)$ is a sum of per-token log-probabilities, and the penalty splits into one term per token. Token $t$ costs

where $y_{<t}$ is the tokens before $t$. Averaged over responses the policy samples, the summed log-ratio equals the KL divergence $\mathrm{KL}(\pi^{\mathrm{RL}} \,\|\, \pi^{\mathrm{SFT}})$, which is zero when the two models assign the same probabilities and grows as the policy moves probability onto text the SFT model found unlikely. One token's log-ratio is not a KL and can be negative: when the policy has made a token less likely than the SFT model did, that token's penalty is a small bonus, and only the average over many samples is guaranteed non-negative.

PPO receives this as a reward sequence. Every token gets its $-\beta$ times log-ratio, and the last token also gets the reward model's score. Appendix C.4 says no discount is applied, so the return credited to the response is $r_\theta(x,y)$ minus $\beta$ times the summed log-ratios. The figure shows two responses to the prompt "Write a story about a wise frog": one sampled early in training, close to the SFT model, and one sampled late, which gets a higher reward-model score but sits far from SFT.

At the paper's $\beta = 0.02$, the late sample's return is 1.90 − 0.02 × 11.00 = 1.68 and the early sample's is 0.60 − 0.02 × 0.42 = 0.59, so PPO pushes the policy toward responses like the late one. The two returns tie at $\beta$ ≈ 0.123 (1.30 / 10.58); above that, the reward model's extra 1.3 points no longer cover the penalty for 10.58 extra nats (natural-log units) of summed log-ratio, and PPO favors staying near SFT. The step in code:

# step 3, one episode: a prompt in, a response out, one reward per token
x      = sample(api_prompts)           # the bandit's only "state"
y      = policy.generate(x)            # pi_RL, starts from the SFT-init model
lp_rl  = policy.logprobs(x, y)         # one log-prob per response token
lp_sft = sft_ref.logprobs(x, y)        # frozen copy of the SFT-init model
reward = -beta * (lp_rl - lp_sft)      # per-token KL penalty, beta = 0.02
reward[-1] += reward_model(x, y)       # the RM score lands on the last token
# PPO (clip 0.2, no discount) turns these rewards into a policy update.
# PPO-ptx only: an extra step on pretraining text, weighted by gamma = 27.8
loss_ptx = -gamma * policy.logprobs(pretrain_batch).mean()

PPO then trains the policy on these rewards. It compares each token's return with a value network's estimate of the return expected at that point, raises the probability of tokens that did better than expected, lowers it for tokens that did worse, and clips each update so the policy cannot change much in one step. The value network starts from the 6B reward model, since that network already maps a prompt and response to a reward-scaled number. Appendix C.4 lists the rest: 256k episodes over about 31k unique prompts, batches of 512 episodes split into 8 minibatches of 64 with one pass over each batch, sampling temperature 1, and PPO clip ratio 0.2. Appendix C adds that prompts longer than 1k tokens were filtered out, responses were capped at 1k tokens, and every model was trained with Adam ($\beta_1 = 0.9$, $\beta_2 = 0.95$), a different pair of betas from the KL coefficient.

The next figure draws the two terms of the first expectation against a single schematic axis, "how far the policy has moved from SFT". The reward rises and flattens; the penalty grows faster the further the policy goes. Drag $\beta$ and watch where their sum peaks:

PPO has a second, separate limit on movement. Its clipped surrogate caps the ratio between the new policy and the policy from the previous iteration at 1 ± 0.2, so a single update cannot move far. The KL penalty is measured against the fixed SFT model and enters through the reward, while the clip acts inside the update rule. A thousand small, individually clipped updates can still carry the policy a long way from where it started; the clip bounds only the ratio to the previous policy, so it places no limit on that cumulative distance, and the KL penalty charges for exactly that distance.

Why the KL penalty is there

The reward model is a 6B network trained on rankings for about 33k prompts, and it agrees with held-out labelers about 70% of the time. Its scores are informative for responses like the ones it was trained on: outputs of the SFT model and its close neighbors. A policy optimized hard against it will find responses that score high for reasons a labeler would not endorse, in regions of text the reward model never saw labeled. The score keeps rising while the quality people would assign stops rising and then falls. This is Goodhart's law (when a measure becomes a target, it stops being a good measure), and in this setting it is called reward model over-optimization. Section 3.5 gives it as the reason for the per-token KL penalty.

The figure below draws the shape with distance from SFT, measured as KL, on the horizontal axis. The proxy reward the reward model reports keeps climbing, while the true quality rises, peaks and falls. Drag $\beta$ and watch where the optimum of reward minus $\beta$ × KL lands relative to the peak:

The KL term is independent of the reward; it depends only on how far the policy's token probabilities have moved from the SFT model's. Adding it caps how far optimization can go, and $\beta$ sets the cap. If $\beta$ is too large, the policy barely moves from SFT and gains little. If it is too small, the policy runs into the region where reward-model scores and human judgments disagree.

InstructGPT itself publishes no turnover curve, but Appendix E.7 reports a sweep of $\beta$ against human Likert ratings (with the pretraining mix on): both $\beta = 0$ and $\beta = 2$ perform poorly, and the best values are around 0.01 to 0.02. The canonical over-optimization curve is in Stiennon et al. (2020). Gao, Schulman, and Hilton (2022) later measured it as scaling laws, plotting true reward against the square root of the KL distance from the initial policy.

The alignment tax

RLHF makes the model far more helpful on API prompts, and it costs something elsewhere. During RLHF fine-tuning, plain PPO models regress relative to GPT-3 on several public NLP benchmarks: SQuAD v2 and DROP (reading comprehension), HellaSwag (commonsense sentence completion), and WMT 2015 French-to-English translation. The paper calls this an alignment tax, a loss of capability paid for alignment. A large tax would push users toward the unaligned model for tasks where capability matters, which defeats the purpose.

The $\gamma$ term in equation (2) pays it back. The regressions indicate the policy is losing competence it learned in pretraining, so during RL the paper keeps training on pretraining text: for each minibatch it computes the PPO gradient and a pretraining gradient (the ordinary next-token loss on text drawn from GPT-3's training set) in consecutive steps and accumulates both, multiplying the pretraining gradient by $\gamma = 27.8$. It uses 8 times as many pretraining examples as RL episodes. These models are called PPO-ptx ("ptx" for pretraining mix), and unless the paper says otherwise, "InstructGPT" means PPO-ptx. Appendix E.6 reports that $\gamma \geq 20$ recovers the regressions on the 1.3B model, and one value, 27.8, worked from 1.3B to 175B. Toggle between the two model types:

Adding the pretraining mix mitigates the regressions on all these datasets and surpasses GPT-3 on HellaSwag, but PPO-ptx still lags behind GPT-3 on DROP, SQuAD v2 and translation. The paper also tested the simpler fix of raising $\beta$ to keep the policy closer to its starting point. Raising the KL coefficient cost significant validation reward and never fully recovered DROP and SQuAD, while in the same 1.3B sweep some value of the pretraining coefficient reversed those two regressions with minimal loss of validation reward.

The two coefficients in equation (2) have different jobs. $\beta$ limits how far the policy drifts from SFT, which keeps it inside the region where the reward model is reliable. $\gamma$ keeps the policy practicing the pretraining task, which preserves benchmark ability that neither the reward model nor the KL penalty measures.

Results and limits

The main metric is labeler preference on held-out API prompts. The 175B InstructGPT is preferred to the 175B GPT-3 85 ± 3% of the time, and 71 ± 4% of the time to a 175B GPT-3 given a few-shot prefix that prompts it into instruction-following mode. The 1.3B InstructGPT is preferred to the 175B GPT-3. The paper's Figure 1 plots each model's win rate against the 175B SFT model rather than the two head to head, so the text reports no exact rate for that matchup. Switch matchups:

The FLAN and T0 comparisons test a different kind of instruction data. FLAN and T0 are collections of public NLP tasks rewritten with natural-language instructions, and the paper fine-tuned 175B GPT-3 on about 1 million examples from each. Those models beat default GPT-3, perform about as well as GPT-3 with a well-chosen prompt, and fall below the SFT baseline. The paper attributes this to the task mix: classification and QA are about 18% of API prompts while generation and brainstorming are about 57%, and public datasets cover little of the latter.

Held-out labelers, who produced no training data, also prefer InstructGPT outputs to GPT-3's by a wide margin, so the gains are not specific to the labelers who produced the training data. On labeler-collected metadata, InstructGPT more often attempts the correct instruction, follows explicit constraints such as "answer in two paragraphs", and makes up facts less often on closed-domain tasks such as summarization, where the output should contain only information from the input (a 21% hallucination rate against GPT-3's 41%). It also follows instructions in other languages and answers questions about code, although these are a tiny minority of the fine-tuning data.

On honesty and harmlessness the gains are smaller and uneven:

• TruthfulQA: Section 1 reports InstructGPT giving truthful and informative answers about twice as often as GPT-3; Section 4.2 describes the gain for PPO models as small but significant. The exception is the 1.3B PPO-ptx model, slightly worse than a GPT-3 model of the same size.
• Toxicity (RealToxicityPrompts, scored by the Perspective API): asked for a respectful answer, InstructGPT produces about 25% fewer toxic outputs than GPT-3. Without that instruction the advantage disappears. Explicitly asked for toxic output, InstructGPT is much more toxic than GPT-3, consistent with it following the instruction.
• Bias: no significant improvement over GPT-3 on Winogender or CrowS-Pairs.

Figure 9 of the paper shows two failure modes that survive training. Asked "Why is it important to eat socks after meditating?", InstructGPT accepts the false premise and lists theories. Asked what happens when a cannonball is fired at a pumpkin at high speed, it hedges that there is "no way to accurately predict" the outcome instead of answering.

The compute cost is small next to pretraining. Training the 175B SFT model took 4.9 petaflops/s-days and the 175B PPO-ptx model 60, against 3,640 for GPT-3 itself (from Brown et al., 2020). The RL stage is about 1.6% of GPT-3's pretraining compute, 1.8% with SFT included. Section 5.1 concludes that on the API's task distribution, RLHF made the model more helpful to users than a 100× increase in model size did.

"Aligned" here has a narrow meaning. InstructGPT is aligned to the preferences of about 40 contractors, following instructions the researchers wrote, on prompts from OpenAI's API customers, and those labelers agree with each other on about 73% of pairs. Section 5.2 states that the authors are not claiming researchers, labelers, or API customers are the right source of preferences; the claim is that the technique can align a model to a specific human reference group for a specific application.

A later method, Direct Preference Optimization (2023), trains the policy on the same kind of preference pairs directly, with no separate reward model and no PPO.