PPO: Proximal Policy Optimization Algorithms

Clipping the probability ratio lets a policy train on each batch of experience for several epochs.

A plain policy gradient takes one step per batch, because further steps on the same data can push the policy somewhere much worse. PPO limits how much reward the objective gives for changing any single action's probability, so many minibatch steps on one batch stay close to the policy that collected it.

TRPO kept each update small with a KL constraint and a second-order solver. PPO gets similar behavior from a min and a clip, a change of a few lines to a vanilla policy gradient, and OpenAI later used it to fine-tune InstructGPT from human feedback.

Data the agent pays for

A supervised model trains on a dataset that already exists. A reinforcement-learning agent has to produce its own data by acting: it runs its current behavior in an environment (a physics simulator, an Atari game), records what happened, and learns from that record. Every batch costs simulator time or real-world interaction, and only a reasonably good policy produces useful batches. Algorithms are therefore compared by sample efficiency, how much the policy improves per timestep of experience.

Standard policy-gradient methods collect a batch, take one gradient step, and throw the batch away. PPO, published by Schulman, Wolski, Dhariwal, Radford and Klimov at OpenAI in July 2017, changes the objective so that the same batch supports several epochs of minibatch updates. It comes out of a line of work on one question, how far a policy may move in one update: conservative policy iteration (2002) posed it, and TRPO (2015) answered it with a KL-divergence constraint solved by second-order optimization. PPO keeps the goal and replaces the machinery with a clipped ratio that ordinary gradient ascent can optimize.

The page builds it in order: the policy gradient and the advantage that weights it; why a batch is normally used once; the trust region TRPO uses to prevent that; the clipped objective; the KL-penalty alternative the paper tested; and the full algorithm with its hyperparameters and results.

The policy gradient

The trained object is a policy $\pi_\theta$, a neural network with parameters $\theta$ that takes the current state $s$ (joint angles and velocities, or a stack of game frames) and outputs a probability distribution over actions $a$. In the paper's MuJoCo experiments it is a two-hidden-layer MLP with 64 tanh units per layer whose output is the mean of a Gaussian over torques, with learned standard deviations. The agent samples its action from that distribution, receives a reward, and moves to a new state. The return is the discounted sum of rewards from a point onward, and training adjusts $\theta$ to raise the expected return.

The return depends on $\theta$ only through which actions get sampled, and the environment in between is not differentiable, so there is no computation graph from $\theta$ to the return to backpropagate through. The score-function identity gets around this. Since $\nabla_\theta \pi = \pi\,\nabla_\theta \log \pi$, the gradient of an expectation over sampled actions becomes an expectation of $\nabla_\theta \log \pi$ times the outcome, which a batch of samples can estimate. The paper writes the estimator as

For each action $a_t$ taken in the batch, $\nabla_\theta \log \pi_\theta(a_t \mid s_t)$ is the direction in parameter space that makes that action more likely in that state. It is scaled by $\hat{A}_t$, an estimate of how much better the action was than usual, and $\hat{\mathbb{E}}_t$ is the empirical average over the batch. This is REINFORCE (Williams, 1992) with a baseline. The update is gradient ascent: $\theta \leftarrow \theta + \alpha \hat{g}$, because the aim is to maximize return. Autodiff frameworks get $\hat{g}$ by differentiating an objective whose gradient is (1):

The paper calls $L^{PG}$ a "loss", following autodiff habit, but it is maximized; code that expects a loss to minimize is handed $-L^{PG}$.

The weight $\hat{A}_t$ estimates the advantage:

$V^\pi(s)$ is the expected return from state $s$ when following the policy, and $Q^\pi(s,a)$ is the expected return when taking action $a$ first and following the policy after. The difference measures how much better than average the action was in that state. Weighting by the raw return would also give a correct gradient on average, but a noisier one. If every action in some state earns a return near 100, raw returns push up every action taken there, including mediocre ones, and only the small differences between them carry information about which action to prefer. Subtracting a baseline $b(s)$ that depends only on the state leaves the expected gradient unchanged, because the baseline term sums to zero over the policy's actions:

The expectation is unchanged and the variance drops, since each sample's weight is now centered near zero. Figure 1 runs this on a one-dimensional action. The amber curve is the return for each action, which the agent cannot see; the teal bell is the policy. Each step samples a few actions, compares each return to the baseline, and pushes the policy toward the actions above it. Press Play and watch the bell move toward the peak of the amber curve:

Estimating the advantage

The agent never observes $A^\pi$; it has one trajectory of rewards and a learned value network $V$. PPO uses generalized advantage estimation (GAE), built from the one-step residual

$\delta_t$ compares what the value network predicted at step $t$ with the reward received plus the discounted prediction one step later. A positive $\delta_t$ means the step went better than predicted. The discount $\gamma$ (0.99 in every experiment in the paper) makes rewards further ahead count for less. GAE sums the residuals from $t$ to the end of the collected segment of $T$ steps, with weights that shrink by $\gamma\lambda$ per step:

$\lambda$ sets how far the estimate looks ahead. At $\lambda = 0$ only $\delta_t$ counts: low variance, but the estimate inherits every error in $V$. At $\lambda = 1$ the sum telescopes to the discounted return actually collected (bootstrapped with $V(s_T)$ at the end of the segment) minus $V(s_t)$. That estimate depends on $V$ only through the final bootstrap and has high variance. The paper uses $\lambda = 0.95$, so a residual 11 steps later gets about half the weight of $\delta_t$.

Figure 2 works an 8-step segment. The value network predicts 0.55 at step 0 and 0.60 at step 1, and the step-0 reward is 0, so $\delta_0 = 0 + 0.99 \times 0.60 - 0.55 = 0.044$. A reward of +1 arrives at step 5 that the network underpredicted. With $\lambda = 0$, $\hat{A}_0 = 0.044$ and step 0 barely registers as good. With $\lambda = 1$, $\hat{A}_0 = 0.482$, the Monte Carlo value. The paper's $\lambda = 0.95$ gives 0.384. Drag $\lambda$ and watch the bright bars, the weighted residuals that sum to $\hat{A}_t$, grow toward the step-5 reward:

GAE with $\lambda < 1$ is biased on purpose. The baseline argument above shows that subtracting the exact $V^\pi(s)$ costs nothing in bias; bootstrapping from a learned $V$ inside the residuals introduces some wherever $V$ is wrong, and GAE accepts it for lower variance. None of what follows depends on how $\hat{A}_t$ was computed.

Why a batch is used once

Equation (1) averages over actions sampled from the current policy. After one gradient step, $\theta$ has changed, and the batch now comes from a policy that no longer exists. A second step on the same batch computes the gradient of the new policy from actions the new policy would sample with different frequencies, so the estimate is biased, and each further step makes the mismatch larger. The paper puts it directly: multiple steps on $L^{PG}$ with the same trajectory are "not well-justified" and "often [lead] to destructively large policy updates". In its experiments that setting scored similar to or worse than the no-clipping baseline of Section 6.1, the worst row in Table 1.

The standard correction for samples from the wrong distribution is importance sampling: reweight each sample by how much more or less likely the current policy is to take that action than the policy that collected it. The weight is the probability ratio

$\theta_\text{old}$ is the parameter vector frozen at collection time. Suppose the old policy chose some action with probability 0.30 and, after a few gradient steps, the new policy gives it 0.39. Then $r_t = 1.30$: the new policy takes that action 30% more often. Every ratio starts at 1 at the beginning of an update. Weighting the advantages by the ratio gives the surrogate objective

The superscript stands for conservative policy iteration (Kakade and Langford, 2002), where this objective was proposed. At $\theta_\text{old}$ its gradient equals (1), since $\nabla_\theta r_t = r_t \nabla_\theta \log \pi_\theta$ and every $r_t = 1$ there. The TRPO paper showed a related property: this surrogate matches the true expected return to first order around $\theta_\text{old}$, so for small steps, improving it improves the policy.

For large steps nothing in (6) limits the optimizer. Take the action above with $\hat{A}_t = +2$. Its term $r_t \hat{A}_t$ grows linearly in $r_t$: 2.0 at the start, 2.6 at $r_t = 1.3$, 6.0 at $r_t = 3$, with the same slope throughout. Gradient ascent keeps raising the probability of the few actions with large positive advantages far past the range where the batch's estimate of their advantage is reliable. The paper: "Without a constraint, maximization of $L^{CPI}$ would lead to an excessively large policy update."

TRPO's trust region

TRPO makes "do not move far" an explicit constraint. It measures how different the new policy is from the old one with the KL divergence, a standard score of how different two probability distributions are (zero when they are identical, larger as they separate), averaged over the states in the batch:

The set of policies within the KL budget $\delta$ is the trust region: the neighborhood of $\theta_\text{old}$ where the surrogate is trusted as a stand-in for the return. The KL is written with the old policy first, $\mathrm{KL}[\pi_{\theta_\text{old}}, \pi_\theta]$, and the order matters because KL is not symmetric. For two Gaussians over one action with the same standard deviation $\sigma$, it reduces to

so TRPO's $\delta = 0.01$ allows the mean to move by $\sqrt{0.02}\,\sigma \approx 0.14\,\sigma$. In Figure 3 the amber curve is the old policy; drag the new policy's mean and watch the KL bar cross $\delta$ at the edge of the shaded band:

TRPO's theory calls for a penalty rather than a constraint, that is, maximizing

because a version of this surrogate that uses the maximum KL over states, rather than the mean, is a lower bound on the policy's true performance. Raising a lower bound guarantees that the true return does not fall, which gives monotonic improvement. The practical algorithm gives up that guarantee twice: it replaces the maximum KL with the mean in (4), and it swaps the penalty for a hard constraint. It uses a constraint because, per the PPO paper, no single fixed $\beta$ works across different problems, "or even within a single problem, where the characteristics change over the course of learning".

TRPO solves (3)–(4) approximately with the conjugate gradient algorithm, using a linear approximation of the objective and a quadratic approximation of the constraint, followed by a line search. Conjugate gradient with a line search is a second-order method, and the PPO paper lists its costs: TRPO is "relatively complicated" and does not fit architectures with noise such as dropout, or with parameters shared between the policy and value function. PPO's goal is TRPO's data efficiency and reliability "while using only first-order optimization".

The clipped objective

The failure in (6) was that $r_t \hat{A}_t$ keeps growing as the ratio moves further from 1. PPO removes that growth outside a band around 1. It clips the ratio to $[1-\epsilon,\ 1+\epsilon]$ and takes the minimum of the clipped and unclipped terms:

$\epsilon$ is a hyperparameter; the paper's example value is 0.2, which is also the best value in its sweep. The first term inside the min is $L^{CPI}$. The clipped term, in the paper's words, "removes the incentive for moving $r_t$ outside of the interval". Which clip matters depends on the sign of the advantage.

Positive advantage. The gradient raises the action's probability, so $r_t$ grows. The term follows $r_t \hat{A}_t$ up to $r_t = 1+\epsilon$ and then stays flat at $(1+\epsilon)\hat{A}_t$. Past that point this sample's gradient is exactly zero. The lower clip $1-\epsilon$ never binds for a positive advantage.

Negative advantage. The gradient lowers the action's probability, so $r_t$ shrinks. Now the lower clip binds: the term is flat at $(1-\epsilon)\hat{A}_t$ for $r_t < 1-\epsilon$ and follows $r_t \hat{A}_t$ everywhere above that. Written per sign, the term is

The min becomes a max in the second case because multiplying by a negative $\hat{A}_t$ reverses the order.

With the numbers from before and $\epsilon = 0.2$: the action with $\hat{A}_t = +2$ at $r_t = 1.30$ has an unclipped term of 2.60 and a clipped term of $1.2 \times 2 = 2.40$. The min is 2.40, the flat branch, so the objective no longer rewards raising that action's probability. Figure 4 plots one sample's term against $r$. Switch the sign of the advantage, move $r$ past each edge of the band, and read $dL/dr$ in the line under the plot:

Keeping each update near the old policy is what proximal refers to. There is no KL computation in (7), no Fisher matrix, no conjugate gradient and no line search: the objective is a min, a clip and one hyperparameter, and any autodiff framework differentiates it. Because it only changes the objective, it works unchanged with a network that shares layers between the policy and the value function, which TRPO's constraint made difficult.

The clip approximates the trust region and does not reproduce it. TRPO bounds the average KL of the whole update. The clip acts on each sample separately: once a sample's ratio leaves the band on the improving side, that sample stops contributing gradient, so the samples still pulling the parameters are those still inside the band or pushed out on the worsening side (next section). Nothing bounds how far the shared parameters move the other samples in the meantime, and the paper states no KL guarantee for $L^{CLIP}$.

Why the objective takes a minimum

Clipping alone would flatten a good action's term past $1+\epsilon$, but it would also flatten it below $1-\epsilon$, where the action has become less likely than before. The min restores the unclipped term there. The paper: "we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse."

Take the bad-action case with numbers. The old policy chose an action with probability 0.30 and it had $\hat{A}_t = -1.5$, but updates driven by other samples have raised it to 0.40, so $r_t = 1.333$. The unclipped term is $1.333 \times (-1.5) = -2.00$; the clipped term is $1.2 \times (-1.5) = -1.80$. The min picks $-2.00$, the unclipped branch, so this sample's gradient is still $\hat{A}_t = -1.5$ per unit of ratio and it pulls $r_t$ back down. A clip without the min would have used $-1.80$, a flat branch with zero gradient, and left the mistake uncorrected. This is the violet zone in Figure 4.

Since the min never exceeds its unclipped argument, $L^{CLIP} \le L^{CPI}$ for every $\theta$. The two are equal while every ratio is inside its band, and in particular, as the paper notes, to first order around $r = 1$. The paper calls $L^{CLIP}$ a pessimistic lower bound on the unclipped objective. It bounds the surrogate, not the true return; the bound on the return belongs to TRPO's max-KL theory.

Figure 5 reconstructs the paper's Figure 2, which interpolates the parameters from $\theta_\text{old}$ to the result of one PPO update and evaluates the objectives along the way. Here the batch is synthetic. The unclipped $L^{CPI}$ keeps rising the further the policy moves. $L^{CLIP}$ rises, peaks, and then falls, because past the peak the samples on the worse side of their band are still counted at full cost while the samples on the improving side no longer add anything. In the paper the peak sits at the updated policy, whose KL from the old one is about 0.02:

A small calculation connects $\epsilon$ to a KL. Take the one-action Gaussian from Figure 3 with $\sigma = 1$ and move its mean by 0.2. The KL is $0.2^2 / 2 = 0.02$. The log-ratio for an action $a$ is $0.2\,(a - \mu_\text{old}) - 0.02$, so an action one standard deviation above the old mean has $r = e^{0.18} = 1.197$, right at the $1+\epsilon$ edge for $\epsilon = 0.2$, and one two standard deviations above it has $r = e^{0.38} = 1.46$. In this toy, a clip of 0.2 starts to bind on typical good actions at about the KL the paper reports after one update.

The adaptive KL penalty

The paper also tests the direct alternative: keep TRPO's penalty form (5) and optimize it with SGD, adjusting $\beta$ so that each update lands near a target KL $d_\text{targ}$. Each policy update first runs several epochs of minibatch SGD on

and then measures $d = \hat{\mathbb{E}}_t[\mathrm{KL}[\pi_{\theta_\text{old}}, \pi_\theta]]$ and updates $\beta$ for the next policy update:

With $d_\text{targ} = 0.01$, an update that reaches a KL below 0.0067 halves the penalty and one above 0.015 doubles it. The factors 1.5 and 2 were "chosen heuristically", and the paper reports the algorithm is not very sensitive to them or to the initial $\beta$. Figure 6 runs the rule on a toy model in which the KL an update reaches is proportional to $1/\beta$ times a pull that drifts 25-fold over training. Compare adaptive $\beta$ against a fixed $\beta = 1$, and count the updates that land in the band:

At $d_\text{targ} = 0.01$ the adaptive rule keeps 37 of the toy's 60 updates in the band against 8 for the fixed penalty, which misses because the pull drifts during training, the change "within a single problem" that the paper gives as the reason a fixed $\beta$ is not enough. On the real benchmark the penalty variants work but trail the clip: the best adaptive setting ($d_\text{targ} = 0.01$) scores 0.74 and the best fixed one ($\beta = 3$) 0.72, against 0.82 for clipping at $\epsilon = 0.2$. The paper keeps the penalty as "an important baseline" and recommends the clip.

The full algorithm

A practical PPO objective adds two terms to the clip. When the policy and value function share a network, the shared layers also need the value network's error signal, so a value loss enters the objective. An entropy bonus $S$ rewards the policy for keeping its action distribution spread out, which keeps it exploring:

Expression (9) is maximized, "approximately", once per iteration. The value error is subtracted because it should be small; the entropy is added because it should be large. Code that minimizes a loss negates all of (9), which flips all three signs. The paper does not define $V_t^{\text{targ}}$; the official code uses $\hat{A}_t + V(s_t)$, the GAE advantage plus the old value prediction, which at $\lambda = 0.95$ is a mix of observed rewards and bootstrapped values. On MuJoCo the paper uses separate policy and value networks, so $c_1$ has no effect, and no entropy bonus. On Atari the networks share parameters, with $c_1 = 1$ and $c_2 = 0.01$.

The loop (Algorithm 1): each iteration, $N$ actors run the frozen policy $\pi_{\theta_\text{old}}$ in parallel for $T$ timesteps each and compute advantages. The objective is then optimized on those $NT$ samples for $K$ epochs (an epoch is one pass over all of them) with minibatches of size $M \le NT$, using minibatch SGD or, "usually for better performance", Adam. Finally $\theta_\text{old} \leftarrow \theta$ and the next batch is collected. The paper's three settings give different amounts of reuse:

• MuJoCo (Table 3): $T = 2048$, $K = 10$, minibatch 64, Adam step size $3 \times 10^{-4}$, $\gamma = 0.99$, $\lambda = 0.95$. With one environment, as in the official script, a batch of 2048 samples gives 32 minibatches per epoch and 320 Adam steps before it is discarded.
• Atari (Table 5): 8 actors, $T = 128$, $K = 3$, minibatch $32 \times 8$. A batch of 1024 samples gives 4 minibatches of 256 per epoch and 12 Adam steps. The step size $2.5 \times 10^{-4}$ and $\epsilon = 0.1$ are both multiplied by a factor $\alpha$ that is annealed linearly from 1 to 0.
• Roboschool humanoids (Table 4): 32 actors (128 for the flagrun tasks), $T = 512$, $K = 15$, minibatch 4096. The Adam step size is adjusted from the measured KL.

In Figure 7, drag $K$ to change how many epochs each collected batch supports:

The same iteration in code. The ratio is computed as the exponential of a difference of log-probabilities, which avoids dividing small probabilities, and the last lines show the ascent-as-negated-loss convention. The official implementation also standardizes the advantages in each batch to zero mean and unit variance before the epochs start, a detail the paper does not mention.

# one PPO iteration (Algorithm 1, actor-critic style)
data = []
for actor in range(N):                  # N actors, in parallel
    traj = run(pi_old, T)               # T timesteps with the frozen policy
    adv  = gae(traj, V, gamma, lam)     # A_hat_t, eqs (11)-(12)
    ret  = adv + V(traj.s)              # value target (official code)
    data += zip(traj, adv, ret, logp(pi_old, traj))   # keep old log-probs

for epoch in range(K):                  # reuse the SAME data K times
    for mb in minibatches(data, M):     # M <= N*T
        r      = exp(logp(pi, mb) - mb.logp_old)       # ratio r_t, eq (6)
        unclip = r * mb.adv
        clip   = clamp(r, 1 - eps, 1 + eps) * mb.adv
        L_clip = mean(min(unclip, clip))               # eq (7)
        L_vf   = mean((V(mb.s) - mb.ret) ** 2)         # squared error
        L      = L_clip - c1 * L_vf + c2 * mean(ent(pi, mb))  # eq (9)
        loss   = -L                     # frameworks minimize, so negate
        loss.backward(); adam.step()

pi_old = copy(pi)                       # then collect a fresh batch

Figure 8 shows why the repeated epochs stay controlled. It optimizes one fixed minibatch of ten samples twice, once on $L^{CPI}$ and once on $L^{CLIP}$, and plots each sample's ratio in the strip below the curves. Scrub to epoch 10 and compare where the two rows of dots end up:

In the toy each sample has its own ratio, so a sample that stops is never moved again. In a real network the samples share parameters, and a stopped sample's ratio can still drift because of the others. That residual drift is one reason the clip gives no hard bound on the KL.

Results

The first experiment compares the surrogate objectives on 7 MuJoCo tasks (HalfCheetah, Hopper, InvertedDoublePendulum, InvertedPendulum, Reacher, Swimmer, Walker2d), one million timesteps each, 3 seeds per task. Each run is scored by its average total reward over the last 100 episodes, rescaled per environment so that a random policy scores 0 and the best result scores 1, and the 21 runs are averaged into one number per setting. Select a bar to read its setting:

$\epsilon = 0.2$ beats 0.1 (0.76) and 0.3 (0.70), and the paper reuses it for the later MuJoCo comparison. The value comes from this sweep in Section 6.1; it is not a row of Table 3. The $-0.39$ for "no clipping or penalty" is the unbounded ratio from (6) in practice: repeated epochs on $L^{CPI}$ drive HalfCheetah to a score far worse than the random initial policy, enough to pull the 7-task average below zero.

Against other algorithms on the same tasks (TRPO, the cross-entropy method, vanilla policy gradient with an adaptive step size, A2C, and A2C with a trust region), PPO with $\epsilon = 0.2$ "outperforms the previous methods on almost all the continuous control environments". On Roboschool it trains a 3D humanoid to run, to steer toward a target that moves every 200 timesteps or when reached, and, in the hardest task, to get up off the ground while being pelted by cubes.

Atari is closer. Over 49 games, with the winner of each game decided by the average of three runs, PPO wins 30 games on average reward over all of training, against 18 for ACER (actor-critic with experience replay, which also learns from stored off-policy samples and adds its own trust-region step) and 1 for A2C (the synchronous version of A3C). That metric rewards learning fast. On average reward over the last 100 episodes, which rewards final performance, ACER wins 28 games, PPO 19, A2C 1, with 1 tie. The paper's own summary is that PPO does significantly better than A2C in sample complexity and similarly to ACER "though it is much simpler".

That simplicity drove adoption: the change to a policy-gradient implementation is a few lines, it runs on Adam with a shared policy-value network, and it has one main hyperparameter, $\epsilon$. The human-preference paper of Christiano et al. appeared five weeks before PPO and used A2C and TRPO. Later RLHF work used PPO, including InstructGPT (2022), which fine-tunes GPT-3 against a learned reward model with PPO plus a per-token KL penalty from the supervised model. That penalty does a job similar to the clip's, holding the fine-tuned model near a reference policy, here the supervised model rather than $\theta_\text{old}$. Methods such as DPO later removed the RL loop from preference tuning altogether.