GAE: High-Dimensional Continuous Control Using Generalized Advantage Estimation

Score each action by a decaying sum of a learned value function's one-step prediction errors, and policy gradients get far less noisy for a small, tunable bias.

A policy gradient makes actions that turned out better than expected more likely. Generalized advantage estimation measures "better than expected" as a weighted sum of the value function's errors after each action, with weights that shrink by a factor $\gamma\lambda$ per step. The parameter $\lambda$ moves the estimate between a one-step version, low in noise but biased by every error in the value function, and the full observed return, unbiased but noisy.

A 2015 Berkeley paper that trained 3D simulated robots to run and to stand up from raw joint states. Its estimator became the advantage computation in PPO and in most on-policy RL code written since.

The paper has two parts. The first is an estimator: given one sampled trajectory and an imperfect value function, how to score each action so that a policy gradient built from those scores has low variance and little bias. The second is an algorithm: TRPO for the policy, a similar trust region for the value network, and the estimator between them, which together learned bipedal and quadrupedal running gaits and a getting-up motion in the MuJoCo simulator. The networks map the robot's 33 (or 29) state numbers straight to 10 (or 8) joint torques, with no hand-designed gait structure.

The page follows the paper's order. It starts with the policy gradient and why its raw form is noisy, defines the advantage function and the discount, builds the estimator from one-step prediction errors, and then spends two sections on its bias: where it comes from, and how the two parameters $\gamma$ and $\lambda$ trade it against variance. The value-function fit, the full algorithm, the experiments and the released code follow.

Why policy gradients are noisy

A stochastic policy $\pi_\theta(a \mid s)$ is a network that outputs a probability distribution over actions; for the robots, as in TRPO, the network outputs the mean of a Gaussian over the 10 joint torques. A trajectory is the sequence $(s_0, a_0, s_1, a_1, \dots)$ produced by sampling an action, stepping the simulator, and repeating, with a reward $r_t$ after every step. The goal is to maximize the expected total reward. Policy gradient methods estimate the gradient of that expectation with respect to $\theta$ from sampled trajectories, and every estimator the paper considers has the form

$\nabla_\theta \log \pi_\theta(a_t \mid s_t)$ is the direction in parameter space that makes the action actually taken at step $t$ more probable. $\Psi_t$ is a scalar that says how much, and with what sign, to move along it. The simplest choice sets $\Psi_t$ to the trajectory's total reward (REINFORCE, Williams 1992): every action in a good episode becomes more likely and every action in a bad episode less likely. The estimate is unbiased and, on long episodes, very noisy.

Take one of the paper's biped episodes: up to 2,000 steps of 0.01 seconds each. If the robot stumbles at step 1,500, the total reward drops, and that drop multiplies the gradient term of the action at step 10 as much as the action at step 1,499. Over many episodes, the actions that had nothing to do with the stumble average out, but only slowly, because each action's effect is mixed with the effects of thousands of other random actions. The paper puts it as the variance of the estimator scaling unfavorably with the time horizon. The usual fixes are to drop rewards that came before $a_t$ (they cannot depend on it) and to subtract a baseline. The paper lists six choices of $\Psi_t$:

1. the total reward of the trajectory, $\textstyle\sum_{t=0}^{\infty} r_t$;
2. the reward following the action, $\textstyle\sum_{t'=t}^{\infty} r_{t'}$;
3. the same minus a baseline $b(s_t)$;
4. the state-action value $Q^\pi(s_t, a_t)$;
5. the advantage $A^\pi(s_t, a_t)$;
6. the TD residual $r_t + V^\pi(s_{t+1}) - V^\pi(s_t)$.

The last three use value functions, defined in the next section. The paper argues for the advantage: a gradient step should raise the probability of better-than-average actions and lower that of worse-than-average ones, and $\Psi_t = A^\pi(s_t, a_t)$ points the term toward a more probable $a_t$ exactly when the action beat the policy's average. The paper says this choice gives "almost the lowest possible variance" and refers to Greensmith et al. (2004) for the formal analysis. The advantage is unknown and has to be estimated from the same samples, and the rest of the paper is about how.

The advantage and the discount

The value function $V^\pi(s_t)$ is the expected sum of future rewards from state $s_t$ if the policy chooses every action. The state-action value $Q^\pi(s_t, a_t)$ is the same expectation with the first action fixed to $a_t$. The advantage is their difference:

The subscripts on $\mathbb{E}$ list what is averaged over: future states drawn from the simulator and future actions drawn from the policy. $V^\pi$ averages over $a_t$ as well; $Q^\pi$ does not. A positive $A^\pi(s_t, a_t)$ means that taking $a_t$ and then following the policy is better, in expectation, than following the policy from $s_t$ directly.

The problem the paper solves is undiscounted: maximize the plain sum of rewards. It then introduces a discount $\gamma \in [0, 1]$ anyway, as a knob on the estimator rather than part of the goal. Rewards $l$ steps after an action are multiplied by $\gamma^l$, which shrinks the contribution of delayed, mostly unrelated rewards to each action's score. The discounted value functions $V^{\pi,\gamma}$, $Q^{\pi,\gamma}$, $A^{\pi,\gamma}$ (equations 4 and 5) are (2) and (3) with $\gamma^l r_{t+l}$ in place of $r_{t+l}$, and the gradient they define is

$g^\gamma$ is already a biased version of the true gradient $g$ whenever $\gamma < 1$, because it ignores the tail of each action's effect. With the weights $\gamma^l$, rewards more than about $1/(1-\gamma)$ steps out barely count: 100 steps at $\gamma = 0.99$, 200 steps at $\gamma = 0.995$. At the robots' 0.01-second timestep that is 1 or 2 seconds of simulated time, and any consequence of an action that arrives later is mostly invisible to the gradient.

From here on, the target is $g^\gamma$, and the question is which estimates of $A^{\pi,\gamma}$ leave it unbiased. The paper calls an estimator $\hat A_t$, which may depend on the whole trajectory, γ-just if substituting it for $A^{\pi,\gamma}$ in (6) gives the same expectation:

Proposition 1 gives a sufficient condition. If $\hat A_t = Q_t - b_t$, where $Q_t$ may depend on anything from step $t$ on but averages to $Q^{\pi,\gamma}(s_t, a_t)$, and $b_t$ depends only on states and actions before $a_t$, then $\hat A_t$ is γ-just. The proof rests on one identity. For any fixed state,

so a baseline $b_t$ that does not change with $a_t$ multiplies a quantity that is zero on average and adds nothing to the expected gradient. It can still change the variance a great deal. Four examples are γ-just: the discounted return $\sum_l \gamma^l r_{t+l}$, $Q^{\pi,\gamma}$, $A^{\pi,\gamma}$ itself, and the one-step residual built from the exact value function:

One-step errors and k-step estimates

Let $V$ be an approximate value function, in practice a neural network. Its TD residual at step $t$ is

It compares two predictions of the same discounted return: $V(s_t)$, made before the step, and $r_t + \gamma V(s_{t+1})$, made after seeing the reward and the next state. If the step went better than the network expected, $\delta^V_t > 0$. With the exact $V = V^{\pi,\gamma}$, the expected residual is exactly the advantage, because averaging $r_t + \gamma V^{\pi,\gamma}(s_{t+1})$ over the next state gives $Q^{\pi,\gamma}(s_t, a_t)$:

With any other $V$, the residual is biased, and the bias is the error of $V$ at $s_{t+1}$ (scaled by $\gamma$) that does not cancel. The paper's next step is to sum $k$ consecutive residuals. The intermediate values cancel in pairs, the $+\gamma V(s_{t+1})$ of one residual against the $-V(s_{t+1})$ of the next once it is multiplied by $\gamma$, leaving a $k$-step return with the value function at both ends:

A three-step example with $\gamma = 0.99$: rewards 0, 0, 1, value predictions $V(s_0) = 0.5$, $V(s_1) = 0.6$, $V(s_2) = 0.8$, and a terminal $s_3$ with value 0. The residuals and the three estimates are

The first is the one-step estimate. The second equals $-0.5 + 0.99^2 \times 0.8$, and the third equals $-0.5 + 0.99^2 \times 1$, the observed discounted return minus the baseline.

In (14) the term $-V(s_t)$ is a baseline: it does not depend on $a_t$, so it adds no bias. All of the bias sits in the bootstrap term $\gamma^k V(s_{t+k})$, and its weight $\gamma^k$ shrinks as $k$ grows. Variance moves the other way, because each extra step adds one more random reward to the sum. As $k \to \infty$ the bootstrap vanishes and

the empirical return minus a value-function baseline, choice 3 in the list above. A3C, published the next year, picks one $k$ per state (up to 5 steps on Atari), where GAE averages over all of them.

How GAE works

The generalized advantage estimator takes an exponentially weighted average of the $k$-step estimators, with weight $(1-\lambda)\lambda^{k-1}$ on $\hat A^{(k)}_t$. The weights sum to 1 for any $\lambda \in [0, 1)$. Written in terms of the residuals, residual $\delta^V_{t+l}$ appears in every estimator with $k > l$, multiplied by $\gamma^l$, so its total weight is $\gamma^l$ times the sum of those estimators' weights, a geometric tail that sums to $\lambda^l$:

Figure 1 lays the regrouping out as a grid. Each row is one $k$-step estimator and the residuals it contains, shaded by the row's weight; the bars underneath are the column sums times $\gamma^l$. Pick a column and compare the readout with $\lambda^l$.

At $\lambda = 0.7$, the default in the figure, column 2 collects 0.7² = 0.49 of the row weight, and the weight on $\delta_{t+2}$ is $0.99^2 \times 0.49 = 0.480$. The two ends of the slider are the estimators from the previous section:

$\mathrm{GAE}(\gamma, 1)$ is γ-just whatever the accuracy of $V$, since $V$ appears only as a baseline, and it has high variance from the long sum of rewards. $\mathrm{GAE}(\gamma, 0)$ is γ-just only when $V = V^{\pi,\gamma}$ and otherwise biased, and it typically has much lower variance. Values in between trade one for the other. The construction is the one behind TD(λ), with a different target: TD(λ) averages $k$-step returns to train a value function, while (16) averages $k$-step advantages. Since each $k$-step advantage is the $k$-step return minus $V(s_t)$, GAE is the λ-return minus $V(s_t)$.

Equation (16) also gives a cheap way to compute it. Splitting off the first term gives the recursion

so one backward pass over a trajectory produces every advantage in $O(T)$ time. Figure 2 runs that computation on an 8-step segment that ends without terminating, so the last residual uses the value of the state after the cut (the hollow dot). A reward of +1 arrives at step 5, which the value network underpredicted.

At step 0, $\lambda = 0$ gives $\hat A_0 = 0.044$: the first step looks barely better than expected, because the value network's prediction rose only from 0.55 to 0.60. $\lambda = 1$ gives 0.482, which includes the full surprise at step 5. The figure starts at $\lambda = 0.95$, which gives 0.384; at the paper's robot setting $\gamma = 0.995, \lambda = 0.96$, the product $\gamma\lambda = 0.955$ halves the weight every 15 steps.

Where the bias comes from

The previous section said that $\lambda < 1$ adds bias only when $V$ is wrong. Expanding (16) shows which values of $V$ enter the estimate. Collecting the terms of each residual,

which follows because $V(s_{t+l})$ appears as $+\gamma V$ in residual $l-1$ (weight $(\gamma\lambda)^{l-1}$) and as $-V$ in residual $l$ (weight $(\gamma\lambda)^l$). The first sum is a return discounted by $\gamma\lambda$ instead of $\gamma$. $-V(s_t)$ is a baseline. The last sum puts the value network's predictions at later states into the score, and those states depend on $a_t$, so an error in $V$ there becomes an error in the gradient. At $\lambda = 1$ the last sum vanishes and the first becomes the ordinary discounted return. When $V = V^{\pi,\gamma}$, the later-state values supply exactly the expected return that the steeper discount cut away.

Section 4 of the paper reads the same algebra as reward shaping. Ng, Harada and Russell (1999) showed that adding $\gamma\Phi(s') - \Phi(s)$ to every reward, for any function $\Phi$ of the state, leaves the optimal policy of a discounted problem unchanged:

Summed along a trajectory with discount $\gamma$, the added terms cancel in pairs and leave only $-\Phi(s_t)$, so the shaped problem has $\tilde Q = Q - \Phi$, $\tilde V = V - \Phi$ and the same advantage $\tilde A^{\pi,\gamma} = A^{\pi,\gamma}$ (equations 22 to 24). Choose $\Phi = V$ and the shaped reward is the TD residual, $\tilde r_t = \delta^V_t$. GAE is then the sum of shaped rewards under the steeper discount $\gamma\lambda$:

To say how much a steeper discount loses, the paper defines the response function: how much the expected reward $l$ steps later changes when the action is fixed to $a_t$, compared with the policy's average,

The advantage is the discounted sum of the responses,

so the response function spreads the advantage across timesteps. A delayed consequence is a response at a large lag $l$. Any discount weights the response by its envelope, and the part at lags well beyond $1/(1-\gamma\lambda)$ is lost. Shaping with the exact $V^{\pi,\gamma}$ moves the entire response to lag 0: once $s_{t+1}$ is known, the expected future shaped reward is zero, so only $\delta_t$ carries information about $a_t$, and no discount can lose anything. A good but imperfect $V$ moves part of it. In the paper's words, GAE reshapes the rewards with $V$ to shrink the temporal extent of the response, then uses the steeper discount to cut off the noise from long delays.

Consider a small example. At step 0 a policy chooses between two actions with probability 0.5 each. The good action pays +1 exactly 20 steps later and the other pays nothing; no other reward depends on the choice. The value network is a blend $V = (1-\kappa)\,V^{\pi,\gamma}$, where $\kappa = 0$ is the exact value function and $\kappa = 1$ assigns no value to the pending reward. Working through the residuals, the difference between the two actions shows up as $(1-\kappa)\gamma^{20}$ in $\delta_0$ and as $\kappa$ in $\delta_{20}$, and zero at every other lag. GAE weights the second by $(\gamma\lambda)^{20}$, so the credit the estimator gives the good action is

against a true, undiscounted value of 1. The factor $\gamma^{20}$ is the bias from $\gamma$ (0.818 at $\gamma = 0.99$). The bracket is the bias from $\lambda$, and it equals 1 when $\kappa = 0$ or $\lambda = 1$. Figure 3 draws the two responses; move κ to 0 and the teal mass jumps to lag 0, where no discount touches it.

At the default $\kappa = 0.5$, $\lambda = 0.96$: 0.409 lands at lag 0 and survives intact, and 0.5 lands at lag 20, of which $0.9504^{20} = 0.362$ survives, for a total of 0.590. With $\kappa = 1$ (no useful value function) the credit is $(\gamma\lambda)^{20} = 0.362$, and at $\lambda = 0$ it is zero: the one-step estimate uses only $\delta_0$, which contains none of a reward that the value function did not predict.

Choosing γ and λ

Both parameters lower variance by shortening the sum, and both can add bias, but the paper separates their roles. $\gamma$ sets the scale of the value function the network has to learn, $V^{\pi,\gamma}$, which does not depend on $\lambda$. $\gamma < 1$ biases the gradient regardless of how accurate the value function is. $\lambda < 1$ biases it only when the value function is inaccurate. The paper reports that the best $\lambda$ is much lower than the best $\gamma$, likely because $\lambda$ adds far less bias for a reasonably accurate value function.

Figure 4, a toy of this page's rather than the paper's, adds noise to the example above and measures the policy gradient itself. The episode runs 200 steps and every reward carries independent Gaussian noise with standard deviation 0.3. With a two-action policy at $p = 0.5$, write $a = 1$ for the good action and $a = 0$ for the other; the gradient with respect to the policy's logit from one episode is $\hat A_0 \times (a - 0.5)$, and its true undiscounted value is 0.25. The curves show, as fractions of 0.25, the bias, the noise of an average over $N = 50$ episodes, and the root-mean-square error that combines them. All three are exact formulas, checked against a 4,000-episode simulation.

At $\kappa = 0.5$ and $\gamma = 0.99$, $\lambda = 0$ has a total error of 0.597, almost all bias; $\lambda = 1$ has 0.626, mostly noise (its bias, 0.182, is the $\gamma$ part alone); the minimum is 0.474 at $\lambda = 0.98$. Three settings show the paper's claims. Set $\kappa = 0$ and the best $\lambda$ is 0, because an exact value function makes the one-step estimate unbiased. Set $\gamma = 1$ and $\lambda = 1$ becomes the worst choice (1.202) while the minimum moves to 0.413 at $\lambda = 0.96$, since nothing then limits the 200 noisy rewards in the sum. Lower $\gamma$ to 0.9 and every curve is dominated by the $\gamma^{20} = 0.12$ factor, a bias no value function can remove.

The noise in Figure 4 has one source, the reward noise, and it grows with the effective length of the sum: the standard deviation of $\hat A_0$ at $\lambda = 1$ is 2.18 times its value at $\lambda = 0.96$. In the robot tasks much of the noise comes from the randomness of later actions and their effect on the dynamics, and it grows with the length of the sum in the same way.

Fitting the value function

Every estimate above uses a value network $V_\phi$. The simplest way to fit it is regression onto the discounted returns observed in the batch:

where $n$ runs over all $N$ timesteps of the batch (50,000 for the biped). The targets are Monte Carlo returns, sometimes called TD(1) targets. The paper also tried TD(λ) targets and saw no difference.

The returns from one batch are noisy, and a large network fitted to convergence on them would chase that noise. The paper constrains each fit with a trust region instead. It first measures how well the old network fits the new targets,

and then solves

In words: the mean squared change in predictions over the batch may be at most $2\sigma^2\epsilon$, so the root-mean-square change is capped at $\sigma\sqrt{2\epsilon}$. If $V_\phi(s)$ is read as the mean of a Gaussian with variance $\sigma^2$, the left side is the average KL divergence between the old and new Gaussians, since the KL divergence between two Gaussians with equal variance is $(\mu_1 - \mu_2)^2 / 2\sigma^2$, the same form as the KL constraint on the policy step.

To solve it, the paper linearizes the objective and approximates the constraint with the Gauss-Newton matrix $H = \tfrac{1}{N}\sum_n j_n j_n^{\top}$, where $j_n = \nabla_\phi V_\phi(s_n)$. The conjugate gradient method, using only products $v \mapsto Hv$, approximately solves the resulting quadratic program (equation 30): it finds the direction $s \approx -H^{-1} g$, with $g$ the gradient of the objective, and rescales the step to the boundary of the constraint. $H$ is, up to a factor of $\sigma^2$, the Fisher information matrix of that Gaussian, so this is a natural-gradient step for the value network, the same procedure TRPO uses for the policy.

Figure 5 runs one such step on a one-dimensional problem small enough to solve exactly: 30 noisy return targets, a value function that is linear in 8 parameters, and an old fit that is close to the truth but not equal to it. Watch the two error curves at the bottom as ε grows.

At $\epsilon = 0$ the fit does not move. As $\epsilon$ grows, the step moves the parameters along the natural-gradient direction toward the batch's least-squares fit; for a linear model with squared error, that straight line is the exact solution path of (29). The error on the batch falls all the way. The error against the true value function falls, bottoms out, and rises again once the step starts fitting this batch's noise, the overfitting to the most recent batch that the paper names as the reason for the trust region. The paper does not report the $\epsilon$ it used.

The full algorithm

The policy update is TRPO. Each iteration it approximately solves

where $\overline{D}_{\mathrm{KL}}$ is the KL divergence between the old and new action distributions, averaged over the batch's states, and $\hat A_n$ is the GAE estimate at timestep $n$ of the batch. Linearizing the objective and taking a quadratic approximation of the constraint gives a step along $F^{-1} g$, where $F$ is the average Fisher information matrix of the policy and $g$ the policy gradient estimate: the natural policy gradient direction, with TRPO's own rule for the step length.

One iteration of the algorithm simulates the current policy for $N$ timesteps, computes $\delta^V_t$ at every step with the current value network $V_{\phi_i}$, computes $\hat A_t = \sum_l (\gamma\lambda)^l \delta^V_{t+l}$, takes the TRPO step to $\theta_{i+1}$, and then takes the value step (30) to $\phi_{i+1}$. In code, with the shapes of a biped batch:

for i in range(n_iters):
    paths = simulate(policy_theta, n_steps=50_000)  # biped batch: 50,000 steps
    for p in paths:                                 # obs (T, 33), act (T, 10)
        p.adv, p.vtarg = gae(p.rew, V_phi(p.obs), v_last(p))
    # Policy step, eq. (31). adv is a constant: no gradient reaches phi here.
    theta = trpo_step(theta, paths)                 # grad flows via log pi only
    # Value step, eq. (30), AFTER the policy step, on the same batch.
    phi = vf_trust_region_step(phi, paths)          # regress V_phi on vtarg

The advantages are computed once from the network outputs and then treated as constants. The policy loss differentiates through $\log \pi_\theta$ only, and the value network's parameters receive gradient only from the regression loss. If the value network were updated first, on the same batch, it would partly absorb that batch's returns, and the residuals that score the policy would shrink toward zero. In the extreme case of a value network that fits every return exactly, every $\delta^V_t$ is zero and so is the policy gradient.

The per-trajectory advantage and value-target computation, written out:

import numpy as np

def gae(rewards, values, v_last, gamma=0.99, lam=0.96):
    """One trajectory of T steps. values[t] = V(s_t) from the value net
    BEFORE this iteration's update; v_last = V(s_T), or 0.0 if s_T is
    terminal. Returns advantages and value targets, both of shape (T,)."""
    T = len(rewards)
    v_next = np.append(values[1:], v_last)
    deltas = rewards + gamma * v_next - values      # TD residuals
    adv = np.zeros(T)
    run = 0.0
    for t in reversed(range(T)):                    # A_t = d_t + g*l*A_{t+1}
        run = deltas[t] + gamma * lam * run
        adv[t] = run
    ret = np.zeros(T)                               # targets for eq. (28)
    run = 0.0
    for t in reversed(range(T)):
        run = rewards[t] + gamma * run
        ret[t] = run
    return adv, ret

What the experiments show

The experiments ask two questions: how $\gamma$ and $\lambda$ affect learning when the optimizer is held fixed, and whether GAE with the two trust regions can train large neural network policies on hard control problems. All results are reported as cost, the negative reward, so lower is better. "No VF" in the plots is a baseline that depends only on the timestep: the average return at that timestep across the batch.

Cart-pole. A linear policy and a value network with one 20-unit hidden layer, 20 trajectories per batch of at most 1,000 steps, averaged over 21 random seeds. At $\gamma = 0.99$, the paper compares $\lambda$ = 0, 0.36, 0.68, 0.84, 0.92, 0.96, 0.98, 0.99 and 1, plus no value function; the fastest improvement comes from $\lambda$ in [0.92, 0.98]. A heat map of cost after 20 iterations over a grid of $\gamma$ and $\lambda$ puts the best region at $\gamma \in [0.96, 0.99]$ and $\lambda \in [0.92, 0.99]$.

3D robots. The policy and value networks share an architecture: three tanh hidden layers of 100, 50 and 25 units with a linear output layer. The simulated humanoid has 33 state dimensions and 10 actuated joints, the quadruped 29 and 8. Episodes end after 2,000 steps of 0.01 s, and the walking episodes end earlier if the center of mass falls below 0.8 m (biped) or 0.2 m (quadruped). The rewards for biped walking, quadruped walking and biped getting up are

with $v_{\mathrm{fwd}}$ the forward velocity, $u$ the joint torques, $f_{\mathrm{impact}}$ the impact forces and $h_{\mathrm{head}}$ the head height. The constant offset pays the walkers for staying up; without it, the quadratic penalties could make ending the episode early the best policy.

Biped locomotion. 50,000 timesteps per batch, nine seeds, about 2 hours per run on a 16-core machine. The plotted conditions vary $\gamma$ from 0.96 to 1 at $\lambda = 0.96$, and $\lambda$ from 0.92 to 1 at $\gamma = 0.995$, plus $\gamma = 1$ without a value function. The best curves come from $\gamma \in [0.99, 0.995]$ and $\lambda \in [0.96, 0.99]$. After 1,000 iterations the gait is fast, smooth and stable. The paper converts this to robot time: $0.01 \times 50{,}000 \times 1{,}000$ seconds is 5.8 days of simulated experience, short enough that the authors call training on a real robot plausible if it could be reset safely.

Quadruped and standing up. 200,000 timesteps per batch, five seeds, about 4 hours per run on a 32-core machine, and a smaller comparison: $\lambda = 0.96$, $\lambda = 1$, and no value function (the Provenance panel below covers a mismatch between the text and the plots here). For the quadruped, $\lambda = 0.96$ is best. For standing up, a value function always helps, and $\lambda = 0.96$ and $\lambda = 1$ end up about the same. The standing-up curves run 500 iterations; at 200,000 steps each that is 11.6 days of simulated time. With the 5.8 days of the walking run, that roughly matches the abstract's "1-2 weeks" for the biped tasks.

The discussion summarizes the sweep as: an intermediate $\lambda$ in [0.9, 0.99] usually works best. It also reports that $\lambda = 0$, the one-step estimate, was too biased to perform well on these tasks, and contrasts this with DDPG and related concurrent work, which differentiate a learned Q-function with respect to the action and made one-step methods work on lower-dimensional problems with careful tuning. The paper compares no other algorithms; it relies on the TRPO paper for those comparisons.

The paper leaves several questions open. Its analysis is, in its own words, "intuitive but informal": there is no bound relating the value function's error to the gradient's error, so the choice of regression loss for $V$ is not matched to what the policy gradient needs. $\gamma$ and $\lambda$ are fixed by hand. And the policy and value networks are separate; sharing features between them is listed as future work.

Implementation notes

John Schulman's modular_rl repository has the reference implementation, released after the paper. Its compute_advantage function is equation (16) with three practical additions. A trajectory that ended in a terminal state bootstraps the last residual with 0; one cut off by the time limit bootstraps it with the last predicted value of the path, an approximation to $V(s_T)$. The discounted sum is computed with a backward linear filter (scipy.signal.lfilter), which evaluates the backward recursion from the GAE section in vectorized form. And the advantages are standardized to mean 0 and standard deviation 1 across the whole batch before the policy step.

The released value fit differs from Section 5. It runs 25 iterations of L-BFGS with a small L2 penalty, and instead of a trust region it regresses onto targets mixed 10% new return and 90% old prediction, which also limits how far one batch can move the network. The value network gets the normalized timestep t / timestep_limit as an extra input, which lets it represent the shrinking return near a time limit. The command-line defaults are $\gamma = 0.99$ and $\lambda = 1.0$.

PPO (2017) kept GAE and replaced the TRPO step with a clipped objective. Its MuJoCo runs use $\gamma = 0.99$ and $\lambda = 0.95$ on segments of 2,048 steps, with the sum in (16) truncated at the segment end and bootstrapped there. Its value target is $\hat A_t + V(s_t)$, which by the identity in the GAE section is the λ-return, the TD(λ) target the paper's footnote describes.