Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model

Tune a language model on preference pairs with one classification loss, no reward model and no RL.

RLHF fits a reward model to human comparisons and then runs reinforcement learning against it. DPO writes the reward as a log-ratio between the model being tuned and a frozen copy of its starting point, which turns the reward model's training loss into a training loss for that model itself.

A Stanford paper from May 2023. Four lines of algebra take the standard RLHF objective to a loss you minimize with plain gradient descent, and on models up to 6B parameters the result matches or beats PPO-based RLHF.

A pretrained language model continues text. It has no built-in preference for the helpful answer over the rude one, and fine-tuning on good examples (supervised fine-tuning, SFT) only gets part of the way, because for most prompts people find it much easier to say which of two answers is better than to write the best answer themselves. Preference tuning uses those comparisons. The method behind InstructGPT and the first ChatGPT is reinforcement learning from human feedback (RLHF): fit a reward model to the comparisons, then optimize the language model against it with PPO, a reinforcement-learning algorithm.

Direct Preference Optimization (DPO), by Rafailov, Sharma, Mitchell, Ermon, Manning and Finn, starts from the same objective RLHF optimizes and removes both the reward model and the reinforcement learning. It trains on the same preference pairs with a loss that has the form of logistic regression, and it never samples from the model during training. The page follows the derivation in dependency order:

1. the Bradley-Terry model, which turns "a person preferred A to B" into a probability of a reward gap;
2. the KL-constrained objective that RLHF maximizes, and its optimal policy in closed form;
3. that closed form solved for the reward, which expresses any reward through a policy;
4. the reward-model loss rewritten with that expression, which is the DPO loss;
5. what its gradient does, where training converges, and how the official code implements it.

The RLHF pipeline DPO replaces

The data is a set of comparisons. A labeler sees a prompt $x$ and two responses, picks one, and the dataset records the preferred response $y_w$ (the winner) and the other one $y_l$ (the loser). No response gets a score. In the Anthropic Helpful and Harmless dataset the paper uses, each of 170k dialogues ends in a pair of model responses with a human label saying which one was preferred.

In reinforcement-learning vocabulary the language model being tuned is the policy: given a prompt, it defines a probability distribution $\pi(y\mid x)$ over responses. RLHF, as the paper reviews it (the pipeline of Ziegler et al. 2019, then Stiennon et al. 2020, Bai et al. 2022 and InstructGPT), has three phases:

1. SFT. Fine-tune a pretrained model on demonstrations to get $\pi^{\text{SFT}}$, which later serves as the reference policy $\pi_{\text{ref}}$.
2. Reward modeling. Train a separate network $r_\phi(x, y)$, usually the SFT model with a linear layer that outputs one scalar, to score winners above losers.
3. RL fine-tuning. Sample responses from the policy, score each with the reward minus a per-response KL penalty,
   $$r_\phi(x,y) - \beta\big(\log\pi_\theta(y\mid x) - \log\pi_{\text{ref}}(y\mid x)\big)$$
   and update the policy with PPO. Repeat.

The third phase keeps the policy, the reference, the reward model and (for PPO) a learned value function in memory at once, and it generates fresh text from the policy at every step. The paper calls the procedure "complex and often unstable". PPO also depends on reward normalization and several hyperparameters to train well; in the paper's sentiment experiment the authors report tuning their own PPO beyond the off-the-shelf TRL settings. The idea of learning a reward from human comparisons predates this pipeline: Christiano et al. (2017) did it for Atari and simulated robots, with A2C and TRPO as the RL algorithms; PPO entered with Ziegler et al.

DPO keeps phase 1 and replaces phases 2 and 3 with a single supervised loss on the fixed pairs. Figure 1 shows the two pipelines.

From comparisons to a reward: the Bradley-Terry model

To learn from comparisons you need a model of how a comparison comes out. The Bradley-Terry model (1952), the same model behind chess Elo ratings, assumes every response has a hidden score, a reward $r^*(x,y)$, and that a person prefers $y_1$ to $y_2$ with probability

where $\sigma(z) = 1/(1+e^{-z})$ is the logistic sigmoid. The second form comes from dividing the numerator and denominator by $\exp(r^*(x,y_1))$. Equal rewards give $\sigma(0) = 0.5$, a coin flip. A reward gap of 1.2 gives $\sigma(1.2) = 0.77$: the higher-reward response wins 77% of comparisons and loses the other 23%, which is how the model accounts for labelers disagreeing and making mistakes. Adding the same constant to both rewards leaves the gap, and therefore the probability, unchanged, and the derivation of DPO relies on exactly this.

Given a dataset $\mathcal{D}$ of triples $(x, y_w, y_l)$, fit a reward model by maximum likelihood. Under (1), the probability of each observed outcome is $\sigma(r_\phi(x,y_w) - r_\phi(x,y_l))$, and minimizing the negative log-likelihood gives the reward-model loss:

This is binary cross-entropy where the classifier's logit is the reward difference and the label is always "the first one won". For a pair where the model currently scores the winner 0.5 above the loser, the loss is $-\log\sigma(0.5) = 0.47$; at a gap of 3 it falls to 0.05. DPO's loss will be (2) with a different expression substituted for $r_\phi$.

The KL-constrained objective

With a reward model in hand, RLHF looks for a policy $\pi_\theta$ that scores well while staying close to the reference policy:

The KL divergence is the expected log-ratio over responses sampled from the policy,

It is zero when the two distributions agree and grows as the policy moves probability onto responses the reference considered unlikely. The coefficient $\beta$ sets the exchange rate: with $\beta = 0.1$, a response whose log-probability is 10 nats higher (a nat is one unit of natural logarithm, so 10 nats is a factor of $e^{10} \approx 22{,}000$ in probability) under the policy than under the reference costs 1 unit of reward.

Why constrain at all? The reward model is only accurate on text resembling what it was trained on, and that text came from the SFT model. The paper gives three reasons for the constraint: it keeps the policy on "the distribution on which the reward model is accurate", it maintains generation diversity, and it prevents "mode-collapse to single high-reward answers". Without it, maximizing $r_\phi$ finds whatever text the reward model over-scores (what later work calls reward hacking).

Objective (3) takes an expectation over $y \sim \pi_\theta$, text sampled from the model being trained. Sampling discrete tokens is not differentiable, so the gradient of (3) cannot be taken by backpropagating through the sampler, and RLHF estimates it with a policy-gradient method, PPO.

The optimal policy in closed form

Ignore for a moment that $\pi_\theta$ is a neural network, and ask which distribution over responses maximizes (3) for one prompt. Write $\pi$ for any candidate distribution. Pulling $\beta$ out and combining the logarithms (the paper's Appendix A.1) turns the objective into

where

The term $\beta\log Z(x)$ does not depend on $\pi$, and a KL divergence is zero exactly when its two arguments are equal and positive otherwise. So the maximizer is $\pi = \pi_r$. The paper credits this form to earlier work on KL-regularized RL (Peters and Schaal 2007, Peng et al. 2019, Korbak et al., Go et al.). It has the same exponential form as the target policy of maximum-entropy RL methods such as Soft Actor-Critic, with the reference distribution in place of a uniform one.

Read (4) as a recipe. Take the reference distribution, multiply each response's probability by $\exp(r/\beta)$, and divide by $Z(x)$, the sum of those products, so the result sums to 1. $Z(x)$ is called the partition function. A large $\beta$ makes $r/\beta$ small, so every factor is close to 1 and $\pi_r$ stays close to $\pi_{\text{ref}}$. A small $\beta$ makes the factors differ by orders of magnitude, and $\pi_r$ concentrates on the highest-reward response.

In the figure's example, y4 has reference probability 0.13 and reward 2.4, the highest of the eight. At $\beta = 0.7$ the optimum gives it probability 0.58; at $\beta = 2.5$, 0.24.

If the optimum has a formula, why does RLHF run PPO? Because $Z(x)$ sums over every possible response. With a 50,257-token vocabulary (GPT-2's) and responses of 20 tokens there are about $10^{94}$ sequences. The paper notes that estimating $Z(x)$ is expensive enough to make (4) "hard to utilize in practice", even with a trained reward model, and PPO approaches the same optimum by sampling instead.

How DPO works: writing the reward in terms of the policy

Equation (4) gives the optimal policy for a reward. Solve it the other way. Take the logarithm of both sides,

and rearrange for $r$:

This holds for any reward function: the reward equals $\beta$ times the log-ratio of its own optimal policy to the reference, plus $\beta\log Z(x)$. The second term still contains the intractable sum, but it depends only on the prompt $x$, because $Z(x)$ has already summed over all responses.

Now use the property of (1) noted above: Bradley-Terry preferences depend only on reward differences between two responses to the same prompt. Substitute (5) for both rewards and subtract. The two $\beta\log Z(x)$ terms are the same number, so they cancel:

$\pi^*$ denotes the optimal policy for the true reward $r^*$. Put the difference into (1) and the human preference probability is written with policies alone:

Every quantity in (6) is a probability of a specific response, which a language model computes with one forward pass. The same cancellation works for the Plackett-Luce model, which extends Bradley-Terry to rankings of more than two responses (the paper's Appendix A.3).

Does writing rewards as $\beta\log(\pi/\pi_{\text{ref}})$ lose anything? Section 5.1 of the paper answers with a definition, two lemmas and a theorem. Definition 1: two reward functions are equivalent if they differ by a function of the prompt alone, $r'(x,y) = r(x,y) + f(x)$. Lemma 1: equivalent rewards induce the same Bradley-Terry (and Plackett-Luce) preferences. Lemma 2: equivalent rewards have the same optimal policy under (3), because $\exp(f(x)/\beta)$ multiplies every response's weight in (4) by the same factor, which the normalization removes. Theorem 1: under mild assumptions, every equivalence class of rewards consistent with these preference models contains a member of the form $\beta\log(\pi/\pi_{\text{ref}})$ for some model $\pi$. The proof picks that member with a projection:

which subtracts exactly the $\beta\log Z(x)$ offset from (5). Equivalently, the parameterization selects the member of each class whose rewards satisfy

so that $Z(x) = 1$ and $\pi_r$ is a normalized distribution by construction. In Figure 4 the five responses have rewards 0.2, 1.0, −0.5, 1.8 and 0.6, with $\beta = 0.5$, so $\beta\log Z(x) = 0.98$ and the normalized rewards $r - 0.98$ are −0.78, 0.02, −1.48, 0.82 and −0.38. Slide every reward up or down by the same $c$ and watch which numbers move.

Preference data cannot identify that shift either: by Lemma 1, Bradley-Terry comparisons determine a reward only up to a prompt-only function. The policy form therefore gives up nothing the data could have taught a reward model, and it fixes the one free choice by the normalization (9).

The DPO loss

The reward-model loss (2) fits $r_\phi$ to the preference data by maximum likelihood. The previous section showed that every reward that matters can be written as $\beta\log(\pi/\pi_{\text{ref}})$ for some policy $\pi$. So instead of a separate reward network, parameterize the reward by the policy being trained and substitute it into (2):

The leading minus sign makes this a loss to minimize, the same sign as (2). The reward being fitted is

which the paper calls the implicit reward, "the reward implicitly defined by the language model $\pi_\theta$ and reference model $\pi_{\text{ref}}$". Minimizing (7) fits $\hat{r}_\theta$ to the preferences exactly as (2) fits $r_\phi$. The optimal policy for $\hat r_\theta$ is $\pi_\theta$ itself: put $\hat r_\theta$ into (4) and the reference cancels, $\pi_{\text{ref}}\cdot\pi_\theta/\pi_{\text{ref}}$, leaving $\pi_\theta$, whose sum over responses is already 1. Fitting the reward and solving for its optimal policy happen in the same step, which is what the paper's subtitle refers to.

For one pair, write $a_w$ for how far the policy has raised the winner's log-probability relative to the reference, and $a_l$ for the same on the loser:

The pair's loss is $-\log\sigma(\beta(a_w - a_l))$. It depends on the difference $a_w - a_l$ only, and $\beta$ converts that difference in log-probability into a reward margin. At the reference point $a_w = a_l = 0$ the loss is $\log 2 = 0.69$ for every pair.

The default point is the worked example from the implementation section below: $a_w = +4$, $a_l = -4$, $\beta = 0.1$. The margin is 8 nats, the reward margin $\beta \cdot 8 = 0.8$, and the loss $-\log\sigma(0.8) = 0.37$. Raise $\beta$ to 1 at the same point and the loss drops to 0.0003, because the same 8-nat shift now counts as a reward margin of 8. With small $\beta$, the policy must move its log-probabilities further from the reference to reach a given loss.

What one DPO gradient step does

Let $z$ be the implicit reward margin of a pair, $\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)$. The pair's loss is $-\log\sigma(z)$, and the chain rule needs two pieces (the reference terms in $z$ are constants):

Multiplying them gives the gradient the paper writes in Section 4:

A gradient-descent step moves against this gradient, so for each pair it increases $\log\pi(y_w\mid x)$ and decreases $\log\pi(y_l\mid x)$. The scalar in front, $\sigma(\hat r_\theta(x,y_l) - \hat r_\theta(x,y_w))$, sets how hard. It is the probability that the current implicit reward, read as a Bradley-Terry model, prefers the loser: near 1 when the model ranks the pair backwards, 0.5 at the reference, near 0 when the winner is already well ahead. The paper's annotation under the term reads "higher weight when reward estimate is wrong". The factor $\beta$ scales every step.

The paper's Unlikelihood baseline uses the same direction with a constant weight: it raises $\log\pi(y_w)$ and lowers $\log\pi(y_l)$ on every pair forever, keeps pushing on pairs it already ranks correctly, and on TL;DR it degenerated into output such as "when when when when" repeated to the end of the summary (Appendix Table 3).

Where DPO training converges

Suppose every pair of responses appears in the data many times, and the fraction of labels preferring $y_i$ to $y_j$ is $\sigma(r^*(x,y_i) - r^*(x,y_j))$, as Bradley-Terry assumes. The expected DPO loss is then a cross-entropy between those label frequencies and the implicit model's predictions, and it is minimized when the two agree for every pair:

The implicit reward then equals $r^*$ up to a prompt-only shift, and by Lemma 2 the policy is the optimum (4) of the true reward, the same policy an exact RLHF optimizer would reach.

Figure 7 runs this on the eight responses of Figure 3, with every one of the 28 pairs in the data, using the DPO gradient on the policy's logits. With Bradley-Terry labels, the teal bars start at the reference and settle onto the dashed outlines, which are (4) computed directly. At $\beta = 0.7$, y4 goes from 0.13 to 0.581, matching the closed form to three decimals, and the loss stops at 0.498, not 0: with noisy labels, even the true reward cannot predict every comparison.

Switch to unanimous labels, where the higher-reward response wins every comparison. This is what a dataset with one label per pair looks like: each observed preference frequency is 0 or 1. Bradley-Terry can match a frequency of 1 only with an infinite reward gap, so the loss keeps decreasing as the margins grow, and the policy sharpens past the dashed optimum toward y4. At $\beta = 0.7$ it reaches 0.94 after 400 steps and is still rising; at $\beta = 2.5$, where the optimum gives y4 only 0.24, it reaches 0.53. Azar et al. (2023, the IPO paper) made this argument: with deterministic preferences, the DPO minimizer assigns the loser probability zero for any $\beta$, so the KL term stops restraining the policy.

The official code carries two remedies aimed at this. Setting label_smoothing to $\varepsilon \in (0, 0.5)$ gives conservative DPO, which assumes each label was flipped with probability $\varepsilon$, so the loss is minimized at a finite margin. The ipo loss (Azar et al.) replaces the log-sigmoid with a squared error that pulls the log-ratio margin toward $1/(2\beta)$. Both default to off, and the paper's experiments use plain DPO.

Implementing DPO

A response $y$ is a token sequence, and $\log\pi(y\mid x)$ is the sum of the per-token log-probabilities of the response tokens, conditioned on the prompt and on the earlier response tokens. The prompt tokens are masked out. The official code computes it like this:

# trainers.py, _get_batch_logps: log pi(y|x) for each sequence in the batch
# logits: (B, T, V); labels: (B, T) with prompt and padding tokens set to -100
labels = labels[:, 1:].clone()        # token t is predicted from tokens < t
logits = logits[:, :-1, :]
loss_mask = labels != -100            # True on response tokens only
labels[labels == -100] = 0            # dummy index; masked out below
per_token_logps = torch.gather(logits.log_softmax(-1), dim=2,
                               index=labels.unsqueeze(2)).squeeze(2)
return (per_token_logps * loss_mask).sum(-1)   # (B,): SUM, not mean

With the default average_log_prob=False the function returns a sum, so a 20-token response has a log-probability around −30 to −40 in the example below, not a per-token average near −1.7. Each pair needs four such numbers: $y_w$ and $y_l$ under the policy and under the reference. The code concatenates winners and losers into one batch, so that is one forward call per model. The reference call runs under torch.no_grad(), and only the policy receives gradients. The loss itself:

# trainers.py, preference_loss (label_smoothing = 0: plain DPO, Eq. 7)
pi_logratios  = policy_chosen_logps    - policy_rejected_logps
ref_logratios = reference_chosen_logps - reference_rejected_logps
logits = pi_logratios - ref_logratios          # a_w - a_l
losses = -F.logsigmoid(beta * logits)          # one loss per pair
# implicit rewards beta * log(pi / pi_ref), detached: logged, never trained on
chosen_rewards   = beta * (policy_chosen_logps
                           - reference_chosen_logps).detach()
rejected_rewards = beta * (policy_rejected_logps
                           - reference_rejected_logps).detach()

The code factors $\beta$ out of the two log-ratios once, which is (7) with the terms regrouped: the policy's winner-minus-loser log-ratio minus the reference's. Put together, one training step is

# one DPO step on a batch of B preference pairs (x, y_w, y_l)
pc, pr = logps(policy, x, y_w), logps(policy, x, y_l)      # (B,), with grad
with torch.no_grad():
    rc, rr = logps(ref, x, y_w), logps(ref, x, y_l)        # (B,), frozen
loss = -F.logsigmoid(beta * ((pc - pr) - (rc - rr))).mean()
loss.backward()                  # gradient flows into the policy only
optimizer.step(); optimizer.zero_grad()

A worked pair at $\beta = 0.1$. The reference gives summed log-probabilities $\log\pi_{\text{ref}}(y_w) = -34$ and $\log\pi_{\text{ref}}(y_l) = -31$, so it slightly prefers the loser. After some training the policy gives $\log\pi_\theta(y_w) = -30$ and $\log\pi_\theta(y_l) = -35$.

• pi_logratios = −30 − (−35) = 5 and ref_logratios = −34 − (−31) = −3, so logits = 5 − (−3) = 8.
• $\beta \cdot 8 = 0.8$, and the loss is $-\log\sigma(0.8) = 0.37$, down from $\log 2 = 0.69$ at the start of training.
• Implicit rewards: $\hat r_w = 0.1 \cdot (-30 + 34) = 0.4$ and $\hat r_l = 0.1 \cdot (-35 + 31) = -0.4$, a reward margin of 0.8.
• Gradient weight: $\sigma(\hat r_l - \hat r_w) = \sigma(-0.8) = 0.31$, down from 0.5, so this pair contributes less to the next update than a pair the model still ranks wrong.

The reference model is $\pi^{\text{SFT}}$ when one exists, since the preference pairs were usually sampled from it. For Anthropic-HH there was no SFT model, so the authors fine-tuned Pythia-2.8B on the preferred responses alone,

to reduce the distribution shift between the data and the reference. The policy is initialized from the same model.

Hyperparameters from Appendix B: $\beta = 0.1$ unless noted, $\beta = 0.5$ for TL;DR summarization, batch size 64, learning rate $10^{-6}$ with a linear warmup over 150 steps, and the RMSprop optimizer (the first arXiv version, v1, says Adam). The released config leaves beta: ??? unset with the comment "lower values mean we care less about the reference model", and the README suggests 0.1 to 0.5 as a starting range; its DPO example command, on Anthropic-HH, uses 0.1. The paper says it "did not meaningfully tune DPO's β".

Compared with PPO-based RLHF, the training loop keeps two models in memory (policy and frozen reference) instead of four, generates no text, and runs a forward and backward pass over fixed data like any supervised job. The code also has a reference_free option, which sets the reference log-ratio to zero, the same as a reference that gives every response equal probability. It is off by default.

What the experiments show

The paper tests three tasks with models up to 6B parameters. The abstract's summary: DPO exceeds PPO-based RLHF at controlling sentiment and matches or improves on it in summarization and single-turn dialogue.

Controlled sentiment (IMDb). The prompt is the first 2 to 8 tokens of a movie review, and the policy (GPT-2-large) must continue it with positive sentiment. A sentiment classifier (siebert/sentiment-roberta-large-english) serves as the true reward and labels the preference pairs, so the true reward of any output can be computed. The authors trained 22 runs across algorithms and settings (DPO with $\beta \in \{0.05, 0.1, 1, 5\}$, PPO with target KL in {3, 6, 9, 12}) and plotted average reward against KL from the reference. DPO's frontier gives the highest reward at every KL value, strictly dominating PPO, and it beats PPO-GT, a PPO run given the true classifier reward instead of a learned one. Both methods target objective (3) here and the true reward is known, so the frontier compares how well each one optimizes it.

TL;DR summarization. Reddit posts, the human preference data of Stiennon et al., and a GPT-J SFT model. GPT-4 judges each method's summaries against the human-written reference summaries. DPO wins about 61% of comparisons at sampling temperature 0.0; PPO's best is 57%, also at 0.0. DPO's win rate holds up as the temperature rises, while PPO's falls toward the base GPT-J model's at high temperatures. In a direct human evaluation, raters preferred DPO samples (temperature 0.25) over PPO samples (temperature 0) 58% of the time.

Anthropic-HH single-turn dialogue. Pythia-2.8B, with the Preferred-FT reference described above. DPO is the only computationally efficient method that improves over the preferred responses in the test set, and it matches or beats Best of 128, which samples 128 responses per prompt and returns the one a learned reward model scores highest. PPO is absent from this comparison: the authors could not find a prompt or temperature at which a public PPO-trained HH model beat base Pythia-2.8B, and they treat Best of 128 as a rough proxy for PPO-level performance.

Distribution shift. The TL;DR policies were also run on CNN/DailyMail news articles, which neither saw in training. GPT-4 win rates against the reference summaries are 0.36 and 0.31 for DPO (temperatures 0 and 0.25) against 0.26 and 0.23 for PPO, which the authors call initial evidence that DPO generalizes about as well as PPO.

The GPT-4 judge. Section 6.4 checks it on TL;DR with human raters. With the concise prompt used for the main results, GPT-4 agreed with humans on 67% to 85% of judgments depending on the comparison, while humans agreed with each other on 65% and 87% in the two comparisons where that was measured. Humans agree with GPT-4 about as often as with each other.

Section 5.2 gives the paper's explanation for PPO's difficulties. Rewriting the RLHF objective in DPO's terms shows that the reward PPO effectively optimizes includes the $\beta\log Z(x)$ normalization, which the authors read as the soft value function of the reference policy. It does not change the optimal policy, but without it the policy-gradient estimate has high variance. Prior RLHF work handles it with a learned value function, which is itself hard to optimize, or with a human-completion baseline, a one-sample Monte Carlo estimate of the normalizer. DPO's reward is already normalized in the sense of (9), so no baseline is needed.

Limitations and open questions

The paper's own Limitations paragraph lists these open questions:

• how DPO policies generalize out of distribution compared with policies trained against an explicit reward (the CNN/DailyMail result favors DPO but is one experiment);
• whether self-labeling with the DPO policy can make use of unlabeled prompts, which PPO uses and DPO does not;
• how reward over-optimization shows up in DPO, and whether the slight late decline in the dialogue win rate during training (the paper's Figure 3, right) is an instance of it;
• scaling beyond the 6B parameters tested;
• the sensitivity of GPT-4 win rates to the judging prompt.

Two structural points follow from the method itself. DPO learns only from the pairs in its dataset: it never samples from the policy it is training, so if the policy drifts toward text unlike anything in the pairs, no comparison covers it. Later work refreshes the pairs from the current policy each round, for example the iterative scheme in Proxy-KD, where the reference is the previous round's model rather than one frozen SFT model. And DPO inherits the Bradley-Terry assumption that one scalar reward per response explains the comparisons, plus the one-label-per-pair effect shown in Figure 7, which conservative DPO and IPO, both in the official repository, are designed to soften.