VerifiedarXiv:1802.0947732 min
Reinforcement learning · Continuous control

TD3: Addressing Function Approximation Error in Actor-Critic Methods

An actor trained on a learned critic drifts toward the critic's overestimates. TD3 builds its targets from the lower of two critics, so the error stops compounding.

TD3 changes DDPG in three places: it trains two critics and uses the smaller of their estimates in the learning target, updates the actor half as often as the critics, and adds clipped noise to the action inside the target.

Explaining the paperAddressing Function Approximation Error in Actor-Critic MethodsFujimoto, van Hoof, Meger · ICML 2018 · arXiv:1802.09477 ↗

A 2018 paper from McGill and the University of Amsterdam that measured how far DDPG's value estimates sit above the returns its policy actually gets, traced the gap to the actor maximizing a noisy critic, and fixed it with changes that take a few lines of code each.

Scott Fujimoto, Herke van Hoof and David Meger posted the paper in February 2018 and presented it at ICML that summer. It starts from DDPG, the deep deterministic policy gradient algorithm, in which an actor network outputs an action and is trained to make a critic network's value estimate large. The paper shows that this arrangement overestimates values for the same reason DQN does, that the fix used for DQN (Double DQN) barely helps here, and that taking the minimum of two critics does. On seven MuJoCo tasks from OpenAI Gym, the resulting algorithm, TD3, reaches a maximum average return of 9,637 on HalfCheetah, against 3,306 for the OpenAI baselines DDPG and 8,577 for the authors' own re-tuned DDPG.

The page covers, in order (equation numbers follow the paper):

  1. why the maximum of noisy estimates is biased upward, and why an actor trained by gradient ascent inherits that bias, equations (1) to (7);
  2. why Double DQN and Double Q-learning do not remove it in actor-critic form, equations (8) and (9);
  3. clipped double Q-learning, equation (10), and what its minimum does to the bias;
  4. how residual errors accumulate through bootstrapping, equations (11) and (12), and how target networks and delayed actor updates keep them small;
  5. target policy smoothing, equations (13) and (14);
  6. the full algorithm with tensor shapes, the results in Tables 1 and 2, and the limitations.

Why a maximum over noisy estimates is too high

An agent in state ss takes action aa, receives reward rr and lands in state s′s'. Its goal is to maximize the return, the sum of future rewards with each one discounted by a factor γ\gamma per step of delay:

Rt=∑i=tTγ i−t r(si,ai)R_t = \sum_{i=t}^{T} \gamma^{\,i-t}\, r(s_i, a_i)

The paper uses γ=0.99\gamma = 0.99. The action-value function Qπ(s,a)Q^\pi(s, a) is the expected return after taking aa in ss and following the policy π\pi afterward. It satisfies the Bellman equation, which splits off the first reward:

Qπ(s,a)=r+γ Es′,a′[Qπ(s′,a′)],a′∼π(s′)Q^{\pi}(s,a)=r+\gamma\,\mathbb{E}_{s',a'}\big[Q^{\pi}(s',a')\big],\quad a'\sim\pi(s')
(2)

Q-learning turns (2) into a regression. A network QθQ_\theta is trained to match a target built from its own estimate at the next state; in DQN the target is r+γmax⁡a′Q(s′,a′)r + \gamma \max_{a'} Q(s', a'), with the maximum over the game's 4 to 18 actions. Every estimate carries some error ϵ\epsilon, because the network is fit from finite, noisy data. The maximum picks whichever action's estimate is highest, and the action whose error happens to be the largest positive one has the best chance of being picked. So even when every error has mean zero, the maximum of the estimates is on average above the maximum of the true values (Thrun and Schwartz, 1993):

Eϵ[max⁡a′(Q(s′,a′)+ϵ)]≥max⁡a′Q(s′,a′)\mathbb{E}_{\epsilon}\big[\max_{a'}(Q(s',a')+\epsilon)\big]\geq\max_{a'}Q(s',a')

A concrete case: two actions with the same true value, and independent errors with standard deviation σ\sigma. The expected maximum of two independent standard normals is 1/π≈0.5641/\sqrt{\pi} \approx 0.564, so the target is too high by 0.564σ0.564\sigma on average, and with more actions the excess grows. Thrun and Schwartz bounded it for errors spread uniformly on [−ϵ,ϵ][-\epsilon, \epsilon]: each target is overestimated by up to γϵ(m−1)/(m+1)\gamma\epsilon(m-1)/(m+1) with mm actions. The excess then enters the next target through the bootstrap, so it can build up; Section 5 of the paper works out how far.

Double Q-learning (van Hasselt, 2010) removes this bias in the tabular case by keeping two tables, QAQ^A and QBQ^B, trained on different experiences. To update QAQ^A it picks the best next action by QAQ^A and reads that action's value from QBQ^B. An error in QAQ^A that made an action look best is not repeated in QBQ^B, so the value read from QBQ^B is not inflated by the selection. Double DQN (van Hasselt, Guez and Silver, 2016) uses the online network to select and DQN's target network, a copy refreshed every 10,000 updates, to evaluate.

Overestimation in actor-critic methods

With continuous actions such as the six joint torques of HalfCheetah, there is no list of actions to take a maximum over. DDPG trains an actor πϕ(s)\pi_\phi(s) with parameters ϕ\phi to output the action, and updates it with the deterministic policy gradient (Silver et al., 2014):

∇ϕJ(ϕ)=Es∼pπ[∇aQπ(s,a)∣a=π(s)∇ϕπϕ(s)]\nabla_{\phi}J(\phi)=\mathbb{E}_{s\sim p_{\pi}}\big[\nabla_{a}Q^{\pi}(s,a)|_{a=\pi(s)}\nabla_{\phi}\pi_{\phi}(s)\big]∇ϕJ(ϕ)=Es∼pπ[∇aQπ(s,a)∣a=π(s)∇ϕπϕ(s)]\begin{aligned}\nabla_{\phi}J(\phi)=\mathbb{E}_{s\sim p_{\pi}}\big[&\nabla_{a}Q^{\pi}(s,a)|_{a=\pi(s)}\\&\nabla_{\phi}\pi_{\phi}(s)\big]\end{aligned}
(1)

The gradient of the critic with respect to the action points the way that raises the value, and the chain rule carries that direction back into the actor's weights. In practice the true QπQ^\pi is replaced by the learned critic QθQ_\theta, which is trained toward DQN's target with the maximum replaced by the target actor's action. Target networks, marked with a prime, are lagged copies of the online networks:

y=r+γQθ′(s′,a′),a′∼πϕ′(s′)y=r+\gamma Q_{\theta^{\prime}}(s^{\prime},a^{\prime}),\quad a^{\prime}\sim\pi_{\phi^{\prime}}(s^{\prime})
(3)

There is no max⁡\max in (3), so it is not obvious that the bias of the previous section applies. Section 4.1 of the paper argues that it does, because the actor does the maximizing. Take one actor update from parameters ϕ\phi. Let ϕapprox\phi_{\text{approx}} be the parameters after a step along the gradient of the learned critic, and ϕtrue\phi_{\text{true}} the parameters after a step along the gradient of the true QπQ^\pi, which is not available during training:

ϕapprox=ϕ+αZ1Es∼pπ[∇ϕπϕ(s)∇aQθ(s,a)∣a=πϕ(s)]ϕtrue=ϕ+αZ2Es∼pπ[∇ϕπϕ(s)∇aQπ(s,a)∣a=πϕ(s)]\begin{aligned}\phi_{\text{approx}}&=\phi+\frac{\alpha}{Z_{1}}\mathbb{E}_{s\sim p_{\pi}}\big[\nabla_{\phi}\pi_{\phi}(s)\nabla_{a}Q_{\theta}(s,a)|_{a=\pi_{\phi}(s)}\big]\\\phi_{\text{true}}&=\phi+\frac{\alpha}{Z_{2}}\mathbb{E}_{s\sim p_{\pi}}\big[\nabla_{\phi}\pi_{\phi}(s)\nabla_{a}Q^{\pi}(s,a)|_{a=\pi_{\phi}(s)}\big]\end{aligned}ϕapprox=ϕ+αZ1Es∼pπ[∇ϕπϕ(s)∇aQθ(s,a)∣a=πϕ(s)]ϕtrue=ϕ+αZ2Es∼pπ[∇ϕπϕ(s)∇aQπ(s,a)∣a=πϕ(s)]\begin{aligned}\phi_{\text{approx}}&=\phi+\frac{\alpha}{Z_{1}}\mathbb{E}_{s\sim p_{\pi}}\big[\nabla_{\phi}\pi_{\phi}(s)\\&\qquad\nabla_{a}Q_{\theta}(s,a)|_{a=\pi_{\phi}(s)}\big]\\\phi_{\text{true}}&=\phi+\frac{\alpha}{Z_{2}}\mathbb{E}_{s\sim p_{\pi}}\big[\nabla_{\phi}\pi_{\phi}(s)\\&\qquad\nabla_{a}Q^{\pi}(s,a)|_{a=\pi_{\phi}(s)}\big]\end{aligned}
(4)

Z1Z_1 and Z2Z_2 normalize the two gradients to length 1, so both steps have the same size α\alpha and differ only in direction (Appendix B handles unnormalized gradients under a slightly stronger assumption). A step along the gradient of a function raises that function at least as much as any other step of the same small size. Applied to the learned critic, its value of the policy it was climbed on is at least its value of the other policy:

E[Qθ(s,πapprox(s))]≥E[Qθ(s,πtrue(s))]\mathbb{E}\big[Q_{\theta}(s,\pi_{\text{approx}}(s))\big]\geq\mathbb{E}\big[Q_{\theta}(s,\pi_{\text{true}}(s))\big]
(5)

Applied to the true value, the policy that followed the true gradient is at least as good in reality:

E[Qπ(s,πtrue(s))]≥E[Qπ(s,πapprox(s))]\mathbb{E}\big[Q^{\pi}(s,\pi_{\text{true}}(s))\big]\geq\mathbb{E}\big[Q^{\pi}(s,\pi_{\text{approx}}(s))\big]
(6)

Each holds for step sizes below some ϵ1\epsilon_1 and ϵ2\epsilon_2 respectively. Now assume the critic is not already pessimistic about πtrue\pi_{\text{true}}:

E[Qθ(s,πtrue(s))]≥E[Qπ(s,πtrue(s))]\mathbb{E}\big[Q_\theta(s, \pi_{\text{true}}(s))\big] \geq \mathbb{E}\big[Q^\pi(s, \pi_{\text{true}}(s))\big]

Then for α<min⁡(ϵ1,ϵ2)\alpha < \min(\epsilon_1, \epsilon_2) the three inequalities chain together: the critic's value of πapprox\pi_{\text{approx}} is at least its value of πtrue\pi_{\text{true}} (5), which is at least the true value of πtrue\pi_{\text{true}} (the assumption), which is at least the true value of πapprox\pi_{\text{approx}} (6). Dropping the middle terms:

E[Qθ(s,πapprox(s))]≥E[Qπ(s,πapprox(s))]\mathbb{E}\big[Q_{\theta}(s,\pi_{\text{approx}}(s))\big]\geq\mathbb{E}\big[Q^{\pi}(s,\pi_{\text{approx}}(s))\big]
(7)

The critic overestimates the value of the policy the actor ends up with. Each update adds only a little, but the paper names two consequences: the excess can build up over many updates through the bootstrap, and the actor is steered toward actions the critic rates too highly, which the next update then reinforces.

Figure 1 is a one-state toy of this argument. The amber dashed curve is a true value over a single action dimension, peaking at 1.0. The teal curve is a critic equal to the truth plus a smooth random error with standard deviation σ\sigma at every action, zero on average at any fixed action. The actor starts at a=−0.7a = -0.7 and climbs the critic by gradient ascent until it stops at a local peak.

Figure 1 · An actor that climbs a noisy critic
σ = 0.15
average over 400 critics: error at the actor's action +0.137 · at a fixed action -0.006
Slide σ to change the critic's error and press New critic to draw another error function. The header compares the critic's value at the actor's final action with the true value there; the line under the controls averages both errors over 400 random critics.

At the default σ=0.15\sigma = 0.15 the average error at a fixed action is about 0 (−0.006 over the 400 critics), while the average error at the action the actor chose is +0.137, and it grows roughly in proportion to σ\sigma: +0.077 at 0.10, +0.30 at 0.30. Some draws give a negative error, when the actor stops on a local peak of the critic that sits below the truth, and they are outnumbered by draws where the peak was created by a positive error. The true value of the chosen action also falls as σ\sigma grows, from 1.0 to 0.86 at σ=0.15\sigma = 0.15, because the actor was pulled off the true peak. At σ=0\sigma = 0 both errors are exactly 0.

The paper checked that this happens in a real agent. Its Figure 1 plots DDPG's average value estimate over 10,000 states during 1 million steps on Hopper-v1 and Walker2d-v1, against an estimate of the true value: the average discounted return of 1,000 episodes that follow the current policy from states sampled from the replay buffer. DDPG's estimate sits above the true value for most of training on both tasks; read off the plot at 1 million steps, about 350 against 270 on Hopper and 430 against 300 on Walker2d. The same figure shows the clipped double Q-learning critic of the next sections staying close to the true value of its own policy.

Why Double DQN does not carry over

The actor-critic analogue of Double DQN selects the next action with the current actor and evaluates it with the target critic:

y=r+γQθ′(s′,πϕ(s′))y=r+\gamma Q_{\theta^{\prime}}(s^{\prime},\pi_{\phi}(s^{\prime}))
(8)

In DQN, selection by the online network and evaluation by the target network works because the target network is a frozen copy for 10,000 updates at a time, and the two networks differ enough that their errors are partly independent. In DDPG the target critic is a soft copy that moves a fraction τ\tau of the way toward the online critic after every update, and the actor itself changes slowly. The paper found the two critics too similar to give an independent estimate; its Figure 2 shows this variant, DDQN-AC, overestimating about as much as DDPG.

The original Double Q-learning form trains two independent actor-critic pairs, (πϕ1,Qθ1)(\pi_{\phi_1}, Q_{\theta_1}) and (πϕ2,Qθ2)(\pi_{\phi_2}, Q_{\theta_2}), where each actor climbs its own critic and each critic's target uses the other critic's value of its own actor's action:

y1=r+γQθ2′(s′,πϕ1(s′))y2=r+γQθ1′(s′,πϕ2(s′))y_{1}=r+\gamma Q_{\theta^{\prime}_{2}}(s^{\prime},\pi_{\phi_{1}}(s^{\prime})) \qquad y_{2}=r+\gamma Q_{\theta^{\prime}_{1}}(s^{\prime},\pi_{\phi_{2}}(s^{\prime}))y1=r+γQθ2′(s′,πϕ1(s′))y2=r+γQθ1′(s′,πϕ2(s′))\begin{gathered}y_{1}=r+\gamma Q_{\theta^{\prime}_{2}}(s^{\prime},\pi_{\phi_{1}}(s^{\prime})) \\ y_{2}=r+\gamma Q_{\theta^{\prime}_{1}}(s^{\prime},\pi_{\phi_{2}}(s^{\prime}))\end{gathered}
(9)

This variant, DQ-AC, overestimates less, but its Figure 2 curves still sit above the true value. The paper explains why: the two critics are not independent. Each is trained toward targets that contain the other, and both learn from the same replay buffer, so their errors are correlated. Where they are, Qθ2Q_{\theta_2} also tends to be high at the action πϕ1\pi_{\phi_1} chose for being high under Qθ1Q_{\theta_1}, and for some states the second critic is even higher than the first at that action:

Qθ2(s,πϕ1(s))>Qθ1(s,πϕ1(s))Q_{\theta_2}(s, \pi_{\phi_1}(s)) > Q_{\theta_1}(s, \pi_{\phi_1}(s))

Qθ1(s,πϕ1(s))Q_{\theta_1}(s, \pi_{\phi_1}(s)) already overestimates on average, so in those states the Double Q target is higher than the single-critic target it was meant to correct.

How clipped double Q-learning works

The paper caps the less biased Qθ2Q_{\theta_2} by the biased Qθ1Q_{\theta_1}, using the estimate that (7) says is too high as an upper bound. Capping one by the other is the same as taking the minimum of the two, which gives the target of clipped double Q-learning:

y1=r+γmin⁡i=1,2Qθi′(s′,πϕ1(s′))y_{1}=r+\gamma\min_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime},\pi_{\phi_{1}}(s^{\prime}))
(10)

The minimum is never larger than Qθ1′Q_{\theta'_1}, so this target cannot overestimate more than the single-critic target does. To save computation, TD3 keeps only one actor, trained on Qθ1Q_{\theta_1}, and gives both critics the same target, y2=y1y_2 = y_1. When Qθ2>Qθ1Q_{\theta_2} > Q_{\theta_1} at the target action, the target is the standard one. When Qθ2<Qθ1Q_{\theta_2} < Q_{\theta_1}, the paper takes it as a sign that the first critic overestimated, and the target is lowered as Double Q-learning would lower it.

Figure 2 measures the three targets on a toy where every action's true value is 0. Critic 1 has a unit-variance error at each of NN candidate actions, critic 2 has an error correlated with it by ρ\rho, and the actor picks the action where critic 1 is highest (a maximum over NN stands in for gradient ascent). Each dot is one draw of the two critics' errors at the chosen action. A single-critic target like DDPG's uses the horizontal coordinate, the Double Q target uses the vertical one, and the clipped target uses whichever is lower.

Figure 2 · Three targets read off the same two critics
N = 10
ρ = 0.50
Slide N, the number of actions the actor compares, and ρ, the correlation between the critics' errors. Dots below the diagonal are teal (critic 2 is lower, so the minimum takes it); dots above are amber. Bars give each target's average error in units of the critic error's standard deviation.

At the default N=10N = 10, ρ=0.5\rho = 0.5, the single-critic target is too high by 1.55 standard deviations, the Double Q target by 0.77 and the clipped target by 0.68. The Double Q bias is ρ\rho times the single-critic bias, because critic 2's error at the chosen action is ρ\rho times critic 1's plus independent noise. Raise ρ\rho to 0.95, a stand-in for DDQN-AC with its slow-moving target critic, and Double Q is back to 1.47 against DDPG's 1.55. At ρ=1\rho = 1 the critics are identical and all three bars are equal. At N=1N = 1 the actor does no selection, the first two targets are unbiased, and the clipped one is below the truth by (1−ρ)/π\sqrt{(1-\rho)/\pi} (0.56 at ρ=0\rho = 0), a downward bias the minimum adds when no selection inflated the estimates.

The paper argues that underestimation does less harm than overestimation. The actor moves toward actions with high estimated value. An overestimated action is therefore likely to be chosen, its inflated value enters the next target through πϕ(s′)\pi_\phi(s'), and the error is copied backward. An underestimated action is avoided by the actor, so its low value rarely appears in a target.

The minimum also favors states whose estimates agree. For two estimates with the same mean μ\mu and standard deviation σ\sigma, the expected minimum is μ−σ(1−ρ)/π\mu - \sigma\sqrt{(1-\rho)/\pi}, so the noisier the estimates of a state, the lower its clipped value. The actor is steered toward states and actions whose value the critics agree on, which the paper calls safer policy updates with stable learning targets.

Appendix A proves convergence for a tabular version: two lookup tables in a finite MDP, the next action chosen greedily by QAQ^A, and both tables updated toward the same target,

a∗=argmax⁡aQA(s′,a),y=r+γmin⁡(QA(s′,a∗),QB(s′,a∗))a^{*}=\operatorname{argmax}_{a}Q^{A}(s^{\prime},a), \qquad y=r+\gamma\min(Q^{A}(s^{\prime},a^{*}),Q^{B}(s^{\prime},a^{*}))a∗=argmax⁡aQA(s′,a),y=r+γmin⁡(QA(s′,a∗),QB(s′,a∗))\begin{gathered}a^{*}=\operatorname{argmax}_{a}Q^{A}(s^{\prime},a), \\ y=r+\gamma\min(Q^{A}(s^{\prime},a^{*}),Q^{B}(s^{\prime},a^{*}))\end{gathered}
(16)

under the usual conditions (every state-action pair visited infinitely often, step sizes summing to infinity with finite sum of squares, γ<1\gamma < 1, bounded reward variance). Since both tables move toward the same target with the same step, their difference shrinks by a factor 1−αt1 - \alpha_t at each update and goes to 0. The gap between the clipped target and the Q-learning target goes to 0 with it, and the proof reduces to the standard one for Q-learning.

How errors accumulate through bootstrapping

Section 5 turns from bias to variance. A critic trained by temporal-difference learning never satisfies the Bellman equation exactly. Write δ(s,a)\delta(s, a) for the residual TD error each update leaves behind:

Qθ(s,a)=r+γ E[Qθ(s′,a′)]−δ(s,a)Q_{\theta}(s,a)=r+\gamma\,\mathbb{E}[Q_{\theta}(s^{\prime},a^{\prime})]-\delta(s,a)
(11)

Substitute (11) for the QθQ_\theta on the right, again and again, along the trajectory. Each substitution adds one more discounted reward and one more discounted residual:

Qθ(st,at)=rt+γE[Qθ(st+1,at+1)]−δt=rt+γE[rt+1+γE[Qθ(st+2,at+2)−δt+1]]−δt=Esi∼pπ,ai∼π[∑i=tTγi−t(ri−δi)]\begin{aligned}Q_{\theta}(s_{t},a_{t})&=r_{t}+\gamma\mathbb{E}[Q_{\theta}(s_{t+1},a_{t+1})]-\delta_{t}\\&=r_{t}+\gamma\mathbb{E}\big[r_{t+1}+\gamma\mathbb{E}\big[Q_{\theta}(s_{t+2},a_{t+2})-\delta_{t+1}\big]\big]-\delta_{t}\\&=\mathbb{E}_{s_{i}\sim p_{\pi},a_{i}\sim\pi}\Big[\sum_{i=t}^{T}\gamma^{i-t}(r_{i}-\delta_{i})\Big]\end{aligned}Qθ(st,at)=rt+γE[Qθ(st+1,at+1)]−δt=rt+γE[rt+1+γE[Qθ(st+2,at+2)−δt+1]]−δt=Esi∼pπ,ai∼π[∑i=tTγi−t(ri−δi)]\begin{aligned}Q_{\theta}(s_{t},a_{t})&=r_{t}+\gamma\mathbb{E}[Q_{\theta}(s_{t+1},a_{t+1})]-\delta_{t}\\&=r_{t}+\gamma\mathbb{E}\big[r_{t+1}\\&\quad+\gamma\mathbb{E}\big[Q_{\theta}(s_{t+2},a_{t+2})-\delta_{t+1}\big]\big]-\delta_{t}\\&=\mathbb{E}_{s_{i}\sim p_{\pi},a_{i}\sim\pi}\Big[\sum_{i=t}^{T}\gamma^{i-t}(r_{i}-\delta_{i})\Big]\end{aligned}
(12)

So the critic estimates the expected return minus the expected discounted sum of all future residuals. The sign convention in (11) makes a negative δ\delta mean the estimate sits above its target; Figure 3 plots ei=−δie_i = -\delta_i summed with discounts, which is exactly Qθ−QπQ_\theta - Q^\pi. With independent per-backup errors of mean μ\mu and standard deviation ss, the sum of a geometric series gives the limits:

E[∑i≥0γiei]=μ1−γ,Std⁡[∑i≥0γiei]=s1−γ2\mathbb{E}\Big[\sum_{i\ge 0}\gamma^{i}e_{i}\Big]=\frac{\mu}{1-\gamma}, \qquad \operatorname{Std}\Big[\sum_{i\ge 0}\gamma^{i}e_{i}\Big]=\frac{s}{\sqrt{1-\gamma^{2}}}E[∑i≥0γiei]=μ1−γ,Std⁡[∑i≥0γiei]=s1−γ2\begin{gathered}\mathbb{E}\Big[\sum_{i\ge 0}\gamma^{i}e_{i}\Big]=\frac{\mu}{1-\gamma}, \\ \operatorname{Std}\Big[\sum_{i\ge 0}\gamma^{i}e_{i}\Big]=\frac{s}{\sqrt{1-\gamma^{2}}}\end{gathered}
Figure 3 · Residual errors summed along a trajectory
800 steps
γ = 0.99
μ = 0.01
std = 0.10
Press Play or drag the step slider to reveal 40 random paths of the running sum. Change the discount γ, the bias left by each backup, and its noise; the dashed line and band mark the limits above.

At γ=0.99\gamma = 0.99 the mean is multiplied by 100 and the standard deviation by 7.09. A backup that leaves the estimate 0.01 too high with noise of 0.1 produces a value estimate off by 1.0 on average with a spread of 0.71. At γ=0\gamma = 0 there is no bootstrap and each error stays its own size. The paper adds that each gradient step reduces the error only on its minibatch, with no guarantee for states outside it, so the per-backup residual is not something training drives to zero.

Target networks and delayed policy updates

A target network makes the target yy in (3) depend on a lagged copy of the critic, so many gradient steps fit nearly the same objective instead of chasing a target that shifts at every step. Its weights follow the online critic by Polyak averaging, θ′←τθ+(1−τ)θ′\theta' \leftarrow \tau\theta + (1-\tau)\theta'; τ=1\tau = 1 means no target network at all. TD3 uses τ=0.005\tau = 0.005, so after ln⁡2/−ln⁡(1−τ)≈138\ln 2 / -\ln(1 - \tau) \approx 138 target updates an old weight has half its influence left (the DDPG explainer has a figure of this averaging).

The paper's Figure 3 plots the value estimate of one randomly chosen Hopper state over 100,000 steps for τ\tau of 1, 0.1 and 0.01. With the policy held fixed, all three settle near the true value; τ=1\tau = 1 is only noisier. With the policy trained on the current critic, the estimates for τ=1\tau = 1 and 0.1 climb into the thousands on a log-scale axis while the true value stays in the low hundreds, and τ=0.01\tau = 0.01 stays close to it. The authors conclude that divergence comes from the interaction: the policy is updated on a high-variance value estimate, the value estimates grow through overestimation when the policy is poor, and the policy gets worse when the value estimate is wrong.

That suggests updating the actor less often than the critic, so that the critic's error has been reduced before the actor uses it. TD3 updates the actor, and all target networks, only once every dd critic updates, with d=2d = 2. Over 1 million environment steps with one critic update per step, the actor receives 500,000 updates and the targets move 500,000 times, which with τ=0.005\tau = 0.005 puts the target half-life at about 277 critic updates. The paper notes that a larger dd would reduce accumulated error further, but with the critic trained only once per step, training the actor too rarely would cripple learning. It also points out that this makes TD3 a two-timescale algorithm, the structure convergence proofs for linear actor-critic methods require (Konda and Tsitsiklis, 2003).

Target policy smoothing

A deterministic actor trained on a critic with a narrow, spurious peak will put its action on the peak, and the target yy then evaluates the critic exactly there, where the error is largest. Section 5.3 regularizes the target with the idea that similar actions should have similar values, and fits the value of a small region around the target action instead of a single point. The paper writes the target as an expectation over noise ϵ\epsilon added to the target action, with the discount γ\gamma restored (the paper's equation (13) drops it; (14) and (15) have it):

y=r+γ Eϵ[Qθ′(s′,πϕ′(s′)+ϵ)]y=r+\gamma\,\mathbb{E}_{\epsilon}\big[Q_{\theta^{\prime}}(s^{\prime},\pi_{\phi^{\prime}}(s^{\prime})+\epsilon)\big]
(13)

In practice the expectation is replaced by a single noise draw per transition, and the averaging happens across the minibatch and across updates:

y=r+γQθ′(s′,πϕ′(s′)+ϵ),ϵ∼clip⁡(N(0,σ),−c,c)y=r+\gamma Q_{\theta^{\prime}}(s^{\prime},\pi_{\phi^{\prime}}(s^{\prime})+\epsilon),\quad\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma),-c,c)y=r+γQθ′(s′,πϕ′(s′)+ϵ),ϵ∼clip⁡(N(0,σ),−c,c)\begin{gathered}y=r+\gamma Q_{\theta^{\prime}}(s^{\prime},\pi_{\phi^{\prime}}(s^{\prime})+\epsilon),\\\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma),-c,c)\end{gathered}
(14)

The paper uses σ=0.2\sigma = 0.2 (a standard deviation) and c=0.5c = 0.5, for MuJoCo actions in [−1,1][-1, 1]. The clip keeps the noisy action close to the original: 0.5 is 2.5 standard deviations, so 1.24% of draws are clipped. Appendix D adds a second clip, of the noisy action to the valid action range, so the critic is never queried at an action the environment would reject. This noise is separate from exploration: the actions sent to the environment get Gaussian noise with standard deviation 0.1, and the smoothing noise exists only inside the target.

In Figure 4 the teal curve is a target critic made of a broad hump (amber dashed, the value it should have learned) plus a narrow spike of height 0.8 at a=0.3a = 0.3, and the target action sits on the spike. The shaded window is the distribution of the noisy target action; the bars at its edges are the probability mass the clip piles up at ±c\pm c.

Figure 4 · Averaging the target over a clipped noise window
0.30
σ = 0.20
c = 0.50
0.80
Drag on the plot or use the first slider to move the target action. Set σ to 0 for DDPG's deterministic target. The header gives the deterministic target, the smoothed target (computed exactly over the window, marked by the short bright bar) and the value without the spike.

At the paper's settings the deterministic target is 1.26, the smoothed target 0.54, and the hump alone 0.46. Without the spike the smoothed target would be 0.42, so smoothing cuts the spike's contribution from 0.80 to 0.12. The noise window is about seven times wider than the spike, so the spike covers a small share of it. With σ=0.05\sigma = 0.05 the smoothed target is still 0.87. The no-spike smoothed value of 0.42 sits below the hump's 0.46 at the same action, because averaging a curved value function over noisy actions lowers it: the critic learns the value of a policy that adds noise of size σ\sigma to its actions, as the paper notes. The paper relates the update to SARSA and expected SARSA, which learn the value of the behavior including its randomness, and observes that values learned this way favor actions that still do well when perturbed.

The full TD3 algorithm

TD3 is DDPG with all three changes. It keeps one actor and two critics, each with a target copy. At each step both critics regress onto the same target, built from the smaller target critic at a smoothed target action:

y=r+γmin⁡i=1,2Qθi′(s′,πϕ′(s′)+ϵ),ϵ∼clip⁡(N(0,σ),−c,c)y=r+\gamma\min_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime},\pi_{\phi^{\prime}}(s^{\prime})+\epsilon),\quad\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma),-c,c)y=r+γmin⁡i=1,2Qθi′(s′,πϕ′(s′)+ϵ),ϵ∼clip⁡(N(0,σ),−c,c)\begin{gathered}y=r+\gamma\min_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime},\pi_{\phi^{\prime}}(s^{\prime})+\epsilon),\\\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma),-c,c)\end{gathered}
(15)

Every dd iterations the actor takes one step of the deterministic policy gradient on Qθ1Q_{\theta_1}, and all three target networks take one soft update. For a transition that ended in a failure state the bootstrap term is dropped and y=ry = r; when an episode ends because it reached the time limit (1,000 steps on the locomotion tasks), the transition is treated as not terminal, since the state itself still has future value. Algorithm 1, written as PyTorch for HalfCheetah (17-dimensional state, 6-dimensional action):

# One TD3 iteration on HalfCheetah: state 17, action 6, minibatch N = 100.
s, a, r, s2, not_done = buffer.sample(100)     # 100x17, 100x6, 100x1, 100x17, 100x1

with torch.no_grad():                          # the target y is a constant
    eps = (0.2 * torch.randn(100, 6)).clamp(-0.5, 0.5)    # smoothing noise
    a2 = (actor_targ(s2) + eps).clamp(-1.0, 1.0)          # 100x6, eq. (14)
    q1_t, q2_t = critic_targ(s2, a2)                      # two target critics
    y = r + not_done * 0.99 * torch.min(q1_t, q2_t)       # 100x1, eq. (15)

q1, q2 = critic(s, a)                          # both critics, the stored action a
loss_q = ((q1 - y) ** 2).mean() + ((q2 - y) ** 2).mean()  # same y for both
critic_opt.zero_grad(); loss_q.backward(); critic_opt.step()

it += 1
if it % 2 == 0:                                # delayed: every d = 2 iterations
    loss_pi = -critic.Q1(s, actor(s)).mean()   # actor ascends critic 1 only
    actor_opt.zero_grad(); loss_pi.backward(); actor_opt.step()
    for p_t, p in zip(target_params, online_params):      # both critics + actor
        p_t.mul_(1 - 0.005).add_(0.005 * p)    # tau = 0.005

Only the critics' parameters receive gradients from loss_q, and the target y is computed without gradient. loss_pi backpropagates through critic 1 into the actor, and only the actor's optimizer steps on it. The paper's networks are two hidden layers of 400 and 300 ReLU units with a tanh output on the actor, and the critic takes state and action concatenated at its first layer. For HalfCheetah that is 129,306 actor parameters and 130,201 per critic. Everything is trained with Adam at learning rate 10−310^{-3} on minibatches of 100 drawn uniformly from a buffer holding the agent's whole history.

SettingTD3DDPG (baselines)
Critic / actor learning rate1e-3 / 1e-31e-3 / 1e-4
Critic weight decaynone1e-2
Target update rate τ0.0050.001
Batch size10064
Exploration noiseN(0, 0.1)OU, θ 0.15, σ 0.2
Normalized observationsnoyes
Policy delay d21
Target noise σ, clip c0.2, 0.5none

The first six rows are the paper's Table 3, which lists them for the re-tuned DDPG that TD3 is built on. Both use a discount of 0.99 and one training iteration per environment step. TD3 also starts with purely random actions to fill the buffer: 10,000 steps on HalfCheetah and Ant, which the paper calls stable-length environments, and 1,000 steps on the other tasks. The released code has drifted from these numbers since publication; the Provenance panel lists the differences.

Results on seven MuJoCo tasks

Each method ran 1 million steps on each of seven Gym tasks, with 10 random seeds, evaluated every 5,000 steps by the average return of 10 episodes without exploration noise. PPO, TRPO, ACKTR and DDPG come from OpenAI's baselines repository, SAC from its authors' code, and "Our DDPG" is the re-tuned DDPG with none of the three changes. Table 1 reports each method's maximum average return:

TaskTD3DDPGOur DDPGPPOTRPOACKTRSAC
HalfCheetah9636.95 ± 859.0653305.608577.291795.43−15.571450.462347.19
Hopper3564.07 ± 114.742020.461860.022164.702471.302428.392996.66
Walker2d4682.82 ± 539.641843.853098.113317.692321.471216.701283.67
Ant4372.44 ± 1000.331005.30888.771083.20−75.851821.94655.35
Reacher−3.60 ± 0.56−6.51−4.01−6.18−111.43−4.26−4.44
InvPendulum1000.00 ± 0.001000.001000.001000.00985.401000.001000.00
InvDoublePendulum9337.47 ± 14.969355.528369.958977.94205.859081.928487.15

TD3 has the highest value on five tasks and ties on InvertedPendulum, where six of the seven methods reach the episode maximum of 1,000. On InvertedDoublePendulum the baselines DDPG is 18 points higher, about 1.2 of TD3's across-seed standard deviations (14.96); Table 1 gives no spread for the baselines. The largest gaps are on the harder locomotion tasks: on Ant, TD3's 4,372 is 2.4 times the next best (ACKTR's 1,822) and almost 5 times the re-tuned DDPG's 889, and on Walker2d it is 1.4 times PPO's 3,318. The comparison is one update per environment step for every off-policy method; Appendix E says the SAC numbers are lower than SAC's own paper partly because SAC was run with 1 update per step instead of 4, and over fewer steps. A maximum over evaluations takes the best point on each learning curve, which flatters methods whose curves are noisy; Table 2 averages the last 10 evaluations instead, and its TD3 HalfCheetah value is 9,533 against Table 1's 9,637.

Which of the three changes matters

Table 2 removes each change from TD3 and adds each to the re-tuned DDPG (labelled AHE, for architecture, hyperparameters and exploration), on four tasks. It also swaps clipped double Q-learning for the actor-critic forms of Double Q-learning (DQ-AC) and Double DQN (DDQN-AC), with delayed updates and smoothing kept in both.

Figure 5 · Table 2 ablation, per task
Choose a task. Bars are average returns over the last 10 evaluations of 10 seeds, scaled to the task's best value (★); the dashed line marks full TD3. Amber rows add one change to the re-tuned DDPG, violet rows remove or replace one in TD3.

On Hopper, adding any one change to AHE gives 908 to 1,465, against TD3's 3,305, and removing any one from TD3 gives 1,837 to 2,407; the changes help mostly in combination. On Ant, clipped double Q-learning does most of the work: AHE + CDQ reaches 3,819 and TD3 − CDQ falls to 850, and DQ-AC and DDQN-AC, which reduce overestimation less, land at 2,446 and 1,092. The paper interprets Ant as a task that benefits from any reduction in overestimation. On HalfCheetah the picture reverses: AHE already scores 8,401, the best result is DDQN-AC's 10,307, and removing CDQ or the delay from TD3 raises its score. TD3 has the highest value in two of the four tasks, Hopper and Ant, and is within 3% of the best on Walker2d. Removing the delay costs 897 points on Hopper and 431 on Ant while gaining 58 on HalfCheetah and 130 on Walker2d, and it doubles the number of actor updates; the paper counts the shorter training time among the delay's benefits.

Limitations and what came next

The bias argument of Section 4.1 is local: it holds for small enough actor steps and assumes the critic is not already below the truth at the true-gradient policy. The convergence proof is for lookup tables. The rest of the case is empirical, on seven Gym tasks with 10 seeds each, and the ablation shows the contribution of each change varies from task to task. Clipped double Q-learning trades overestimation for a bias downward, and target policy smoothing makes the critic learn the value of a slightly noisy policy rather than the deterministic one it serves. The delay d=2d = 2 and the noise scale were set for these tasks; the paper does not report a sweep of either.

None of the three changes depends on the rest of DDPG, and the paper's conclusion says they can be added to any actor-critic algorithm. Appendix E notes that the later version of Soft Actor-Critic includes clipped double Q-learning in its value update and is competitive with TD3 on most tasks. That version keeps two critics and the minimum, and replaces the deterministic actor with a stochastic one rewarded for entropy.

Provenance Verified against primary literatureHow we verify
Fujimoto, van Hoof & Meger (ICML 2018; arXiv 1802.09477v3)Equations (1) to (15) and (16) to (21) in the appendix, Algorithm 1, Table 1 (all 49 cells) and Table 2 (all 44 cells, copied into Figure 5), Table 3 hyperparameters, Section 6.1 settings (400-300 ReLU networks, Adam 1e-3, batch 100, tau 0.005, d = 2, smoothing N(0, 0.2) clipped to (-0.5, 0.5), exploration N(0, 0.1), 10,000 or 1,000 random start steps, 10 seeds, evaluation every 5,000 steps over 10 episodes), Appendix D terminal handling and action clipping, Appendix E on SAC.
sfujim/TD3 at commit 25dfc0a (September 2018), TD3.py and main.pyTD3.py lines 18-20 and 37-44: actor 400-300-action_dim, two critics with state and action concatenated at the first layer. Lines 102-110: noise normal(0, 0.2) clamped to 0.5, next action clamped to the action range, target r + done * discount * min(Q1, Q2). Line 116: one loss summing both critics' squared errors against the same target. Lines 124-139: actor loss -Q1(s, actor(s)) and soft updates of all target networks when it % policy_freq == 0. main.py lines 89-96: training runs episode_timesteps iterations at the end of each episode. Line 123: done is 0 at the time limit.
sfujim/TD3 master (TD3.py, main.py, README)Current defaults differ from the paper: 256-256 networks, Adam 3e-4, batch 256, 25,000 random start steps, smoothing noise and clip scaled by max_action, one training iteration after every environment step (main.py line 137). The README states the code "is no longer exactly representative of the code used in the paper"; the released learning curves are the paper's.
van Hasselt, Double Q-learning (NeurIPS 2010)Algorithm 1: two tables, each update picks one at random, a* = argmax_a Q^A(s', a), target r + gamma Q^B(s', a*). Section 3: the double estimator can underestimate.
van Hasselt, Guez & Silver, Deep Reinforcement Learning with Double Q-learning (AAAI 2016)Equation (4) Double Q target and the Double DQN target R + gamma Q(S', argmax_a Q(S', a; theta_t), theta_t^-): the online network selects, the target network evaluates. Thrun and Schwartz (1993) bound: errors uniform on [-epsilon, epsilon] overestimate each target by up to gamma epsilon (m - 1)/(m + 1) for m actions.
This page, arithmetic and toysE[max] of two independent N(0, 1) draws = 1/sqrt(pi) = 0.564; E[min(X1, X2)] = -sqrt((1 - rho)/pi) for correlation rho, checked by Monte Carlo (400,000 draws); the geometric sums 1/(1 - gamma) = 100 and 1/sqrt(1 - gamma^2) = 7.09 at gamma = 0.99; the half-life ln 2 / -ln(1 - tau) = 138 target updates; the clip rate 2(1 - Phi(2.5)) = 1.24%; parameter counts 129,306 (actor) and 130,201 per critic for HalfCheetah. Figures 1 to 4 are toys built for this page, not the paper's experiments.
correctionFive places where the paper's text disagrees with its own tables, equations or code. (1) Equation (13) is missing the discount: it reads y = r + E[Q(s', pi(s') + eps)], and the gamma that equations (14) and (15) carry belongs there too. (2) Algorithm 1 writes the delayed update as "if t mod d then". Read literally that runs when t mod d is nonzero; with d = 2 the rate happens to be the same, but d = 3 would update on two steps of every three. The code tests total_it % policy_freq == 0, which is what this page teaches. (3) Appendix C's layer tables end the actor with (300, 1) and give the re-tuned critic a second layer of (action dim + 400, 300). The code ends the actor with (300, action_dim) and, since the action enters the critic at its first layer, gives the critic (400, 300). (4) The abstract claims TD3 outperforms the state of the art "in every environment tested". In Table 1 six methods tie at 1000.00 on InvertedPendulum, and on InvertedDoublePendulum the baselines DDPG scores 9355.52 against TD3's 9337.47 (the table bolds both). Section 6.1's "matches or outperforms" is the accurate wording. Section 6.2 says the full algorithm beats every other combination "in most tasks"; in Table 2 it has the highest value in two of four, Hopper and Ant. (5) Section 6.1 says the networks are trained after each time step. The paper-era code runs the same number of updates in a burst at the end of each episode, so the actor that collects an episode does not change during it. The current repository trains after every step but has also changed the network width, learning rate and batch size. A convention note: N(0, sigma) in the paper is parameterized by the standard deviation (the code multiplies a unit normal by 0.2).

Questions you might still have

?

What does TD3 stand for, and what does it change in DDPG?
Twin Delayed Deep Deterministic policy gradient. It keeps DDPG's deterministic actor, replay buffer and soft target networks, and makes three changes: two critics whose smaller estimate forms the learning target (twin), actor and target-network updates only every second critic update (delayed), and clipped Gaussian noise added to the target action. The paper's DDPG baseline was also re-tuned: no batch normalization, no weight decay, Gaussian instead of Ornstein-Uhlenbeck exploration noise, the action fed to the critic's first layer.

?

Why take the minimum of the two critics instead of their average?
At the action the actor chose, critic 1's error is selected to be high, and critic 2's error is high too in proportion to how correlated the critics are. The average of the two keeps that upward bias, shrunk by a factor (1 + rho)/2 in the toy of Figure 2 on this page. The minimum can cancel it, and the paper argues that the underestimate the minimum produces in some states does less damage, because the actor moves away from actions whose value is underestimated instead of feeding them back into later targets.

?

Does clipped double Q-learning underestimate?
It can. When the two critics' errors are independent and the actor's choice has not inflated them, the minimum of two unbiased estimates sits about 0.56 standard deviations below the truth. The paper accepts this and gives the reason above. In its Figure 1 the clipped critic's estimate stays close to the true value of its policy on Hopper and Walker2d, while DDPG's ends well above.

?

Why is only the first critic used to train the actor?
For cost. The full double Q-learning form would train two actors, each against its own critic. The paper keeps one actor, trains it on Q1, and gives both critics the same target. When Q2 is above Q1 at the target action the target equals the ordinary one; when Q2 is lower, the target is pulled down.

?

Is TD3 on-policy or off-policy?
Off-policy. Like DDPG it learns from a replay buffer holding the agent's entire history, and the target uses the target actor's action at the next state, not the stored next action. The smoothing noise in the target is chosen separately from the exploration noise used to collect data.

?

How much of the gain comes from re-tuning DDPG rather than from the three changes?
It depends on the task. In Table 2 the re-tuned DDPG without any of the changes (AHE) already lifts HalfCheetah from 3,163 to 8,401, most of the way to TD3's 9,533. On the other three tasks re-tuning alone helps less or hurts: Hopper falls from 1,732 to 1,062, Walker2d rises from 1,521 to 2,362, and Ant falls from 816 to 564. Adding the three changes then takes them to 3,305, 4,565 and 4,185.

?

Does the convergence proof cover neural networks?
No. Appendix A proves that clipped double Q-learning converges to the optimal action values in a finite MDP with lookup tables, under the usual step-size and infinite-visit conditions, with both tables updated toward the same target. Nothing is proved for the deep, deterministic-actor version, whose case rests on the experiments.

?

Why does delaying the actor help if the critic is still trained once per step?
The critic and the target networks change between actor updates, so each actor update sees a critic that has taken d more gradient steps toward the current targets. With d = 2 the actor gets 500,000 updates in a 1-million-step run instead of a million, and Table 2 shows removing the delay costs on Hopper and Ant while slightly helping on HalfCheetah and Walker2d.

?

How is TD3 related to Soft Actor-Critic?
The two were published months apart in 2018, and TD3's Table 1 compares against the first SAC. Appendix E notes that a later version of SAC uses clipped double Q-learning in its value update and is competitive with TD3 on most tasks. SAC keeps the minimum over two critics and replaces the deterministic actor with a stochastic one trained with an entropy bonus.

Footnotes & further reading

  1. The paper: Fujimoto, van Hoof & Meger, Addressing Function Approximation Error in Actor-Critic Methods (ICML 2018). Code and the paper's learning curves: github.com/sfujim/TD3; the paper-era version is commit 25dfc0a.
  2. Overestimation and double estimators: Thrun & Schwartz, Issues in using function approximation for reinforcement learning (1993 Connectionist Models Summer School); van Hasselt, Double Q-learning (NeurIPS 2010); van Hasselt, Guez & Silver, Deep Reinforcement Learning with Double Q-learning (AAAI 2016), one of the six parts of Rainbow.
  3. The base algorithm: Silver et al., Deterministic Policy Gradient Algorithms (ICML 2014); Lillicrap et al., Continuous control with deep reinforcement learning (DDPG, ICLR 2016; explained in DDPG).
  4. Two-timescale actor-critic: Konda & Tsitsiklis, On actor-critic algorithms (SIAM Journal on Control and Optimization, 2003).
  5. The SAC comparison: Haarnoja et al., Soft Actor-Critic (ICML 2018; explained in SAC), and the later version with two critics, Haarnoja et al., Soft Actor-Critic Algorithms and Applications (2018).