TD3: Addressing Function Approximation Error in Actor-Critic Methods
An actor trained on a learned critic drifts toward the critic's overestimates. TD3 builds its targets from the lower of two critics, so the error stops compounding.
TD3 changes DDPG in three places: it trains two critics and uses the smaller of their estimates in the learning target, updates the actor half as often as the critics, and adds clipped noise to the action inside the target.
Explaining the paperAddressing Function Approximation Error in Actor-Critic MethodsA 2018 paper from McGill and the University of Amsterdam that measured how far DDPG's value estimates sit above the returns its policy actually gets, traced the gap to the actor maximizing a noisy critic, and fixed it with changes that take a few lines of code each.
Scott Fujimoto, Herke van Hoof and David Meger posted the paper in February 2018 and presented it at ICML that summer. It starts from DDPG, the deep deterministic policy gradient algorithm, in which an actor network outputs an action and is trained to make a critic network's value estimate large. The paper shows that this arrangement overestimates values for the same reason DQN does, that the fix used for DQN (Double DQN) barely helps here, and that taking the minimum of two critics does. On seven MuJoCo tasks from OpenAI Gym, the resulting algorithm, TD3, reaches a maximum average return of 9,637 on HalfCheetah, against 3,306 for the OpenAI baselines DDPG and 8,577 for the authors' own re-tuned DDPG.
The page covers, in order (equation numbers follow the paper):
- why the maximum of noisy estimates is biased upward, and why an actor trained by gradient ascent inherits that bias, equations (1) to (7);
- why Double DQN and Double Q-learning do not remove it in actor-critic form, equations (8) and (9);
- clipped double Q-learning, equation (10), and what its minimum does to the bias;
- how residual errors accumulate through bootstrapping, equations (11) and (12), and how target networks and delayed actor updates keep them small;
- target policy smoothing, equations (13) and (14);
- the full algorithm with tensor shapes, the results in Tables 1 and 2, and the limitations.
Why a maximum over noisy estimates is too high
An agent in state takes action , receives reward and lands in state . Its goal is to maximize the return, the sum of future rewards with each one discounted by a factor per step of delay:
The paper uses . The action-value function is the expected return after taking in and following the policy afterward. It satisfies the Bellman equation, which splits off the first reward:
Q-learning turns (2) into a regression. A network is trained to match a target built from its own estimate at the next state; in DQN the target is , with the maximum over the game's 4 to 18 actions. Every estimate carries some error , because the network is fit from finite, noisy data. The maximum picks whichever action's estimate is highest, and the action whose error happens to be the largest positive one has the best chance of being picked. So even when every error has mean zero, the maximum of the estimates is on average above the maximum of the true values (Thrun and Schwartz, 1993):
A concrete case: two actions with the same true value, and independent errors with standard deviation . The expected maximum of two independent standard normals is , so the target is too high by on average, and with more actions the excess grows. Thrun and Schwartz bounded it for errors spread uniformly on : each target is overestimated by up to with actions. The excess then enters the next target through the bootstrap, so it can build up; Section 5 of the paper works out how far.
Double Q-learning (van Hasselt, 2010) removes this bias in the tabular case by keeping two tables, and , trained on different experiences. To update it picks the best next action by and reads that action's value from . An error in that made an action look best is not repeated in , so the value read from is not inflated by the selection. Double DQN (van Hasselt, Guez and Silver, 2016) uses the online network to select and DQN's target network, a copy refreshed every 10,000 updates, to evaluate.
Overestimation in actor-critic methods
With continuous actions such as the six joint torques of HalfCheetah, there is no list of actions to take a maximum over. DDPG trains an actor with parameters to output the action, and updates it with the deterministic policy gradient (Silver et al., 2014):
The gradient of the critic with respect to the action points the way that raises the value, and the chain rule carries that direction back into the actor's weights. In practice the true is replaced by the learned critic , which is trained toward DQN's target with the maximum replaced by the target actor's action. Target networks, marked with a prime, are lagged copies of the online networks:
There is no in (3), so it is not obvious that the bias of the previous section applies. Section 4.1 of the paper argues that it does, because the actor does the maximizing. Take one actor update from parameters . Let be the parameters after a step along the gradient of the learned critic, and the parameters after a step along the gradient of the true , which is not available during training:
and normalize the two gradients to length 1, so both steps have the same size and differ only in direction (Appendix B handles unnormalized gradients under a slightly stronger assumption). A step along the gradient of a function raises that function at least as much as any other step of the same small size. Applied to the learned critic, its value of the policy it was climbed on is at least its value of the other policy:
Applied to the true value, the policy that followed the true gradient is at least as good in reality:
Each holds for step sizes below some and respectively. Now assume the critic is not already pessimistic about :
Then for the three inequalities chain together: the critic's value of is at least its value of (5), which is at least the true value of (the assumption), which is at least the true value of (6). Dropping the middle terms:
The critic overestimates the value of the policy the actor ends up with. Each update adds only a little, but the paper names two consequences: the excess can build up over many updates through the bootstrap, and the actor is steered toward actions the critic rates too highly, which the next update then reinforces.
Figure 1 is a one-state toy of this argument. The amber dashed curve is a true value over a single action dimension, peaking at 1.0. The teal curve is a critic equal to the truth plus a smooth random error with standard deviation at every action, zero on average at any fixed action. The actor starts at and climbs the critic by gradient ascent until it stops at a local peak.
At the default the average error at a fixed action is about 0 (−0.006 over the 400 critics), while the average error at the action the actor chose is +0.137, and it grows roughly in proportion to : +0.077 at 0.10, +0.30 at 0.30. Some draws give a negative error, when the actor stops on a local peak of the critic that sits below the truth, and they are outnumbered by draws where the peak was created by a positive error. The true value of the chosen action also falls as grows, from 1.0 to 0.86 at , because the actor was pulled off the true peak. At both errors are exactly 0.
The paper checked that this happens in a real agent. Its Figure 1 plots DDPG's average value estimate over 10,000 states during 1 million steps on Hopper-v1 and Walker2d-v1, against an estimate of the true value: the average discounted return of 1,000 episodes that follow the current policy from states sampled from the replay buffer. DDPG's estimate sits above the true value for most of training on both tasks; read off the plot at 1 million steps, about 350 against 270 on Hopper and 430 against 300 on Walker2d. The same figure shows the clipped double Q-learning critic of the next sections staying close to the true value of its own policy.
Why Double DQN does not carry over
The actor-critic analogue of Double DQN selects the next action with the current actor and evaluates it with the target critic:
In DQN, selection by the online network and evaluation by the target network works because the target network is a frozen copy for 10,000 updates at a time, and the two networks differ enough that their errors are partly independent. In DDPG the target critic is a soft copy that moves a fraction of the way toward the online critic after every update, and the actor itself changes slowly. The paper found the two critics too similar to give an independent estimate; its Figure 2 shows this variant, DDQN-AC, overestimating about as much as DDPG.
The original Double Q-learning form trains two independent actor-critic pairs, and , where each actor climbs its own critic and each critic's target uses the other critic's value of its own actor's action:
This variant, DQ-AC, overestimates less, but its Figure 2 curves still sit above the true value. The paper explains why: the two critics are not independent. Each is trained toward targets that contain the other, and both learn from the same replay buffer, so their errors are correlated. Where they are, also tends to be high at the action chose for being high under , and for some states the second critic is even higher than the first at that action:
already overestimates on average, so in those states the Double Q target is higher than the single-critic target it was meant to correct.
How clipped double Q-learning works
The paper caps the less biased by the biased , using the estimate that (7) says is too high as an upper bound. Capping one by the other is the same as taking the minimum of the two, which gives the target of clipped double Q-learning:
The minimum is never larger than , so this target cannot overestimate more than the single-critic target does. To save computation, TD3 keeps only one actor, trained on , and gives both critics the same target, . When at the target action, the target is the standard one. When , the paper takes it as a sign that the first critic overestimated, and the target is lowered as Double Q-learning would lower it.
Figure 2 measures the three targets on a toy where every action's true value is 0. Critic 1 has a unit-variance error at each of candidate actions, critic 2 has an error correlated with it by , and the actor picks the action where critic 1 is highest (a maximum over stands in for gradient ascent). Each dot is one draw of the two critics' errors at the chosen action. A single-critic target like DDPG's uses the horizontal coordinate, the Double Q target uses the vertical one, and the clipped target uses whichever is lower.
At the default , , the single-critic target is too high by 1.55 standard deviations, the Double Q target by 0.77 and the clipped target by 0.68. The Double Q bias is times the single-critic bias, because critic 2's error at the chosen action is times critic 1's plus independent noise. Raise to 0.95, a stand-in for DDQN-AC with its slow-moving target critic, and Double Q is back to 1.47 against DDPG's 1.55. At the critics are identical and all three bars are equal. At the actor does no selection, the first two targets are unbiased, and the clipped one is below the truth by (0.56 at ), a downward bias the minimum adds when no selection inflated the estimates.
The paper argues that underestimation does less harm than overestimation. The actor moves toward actions with high estimated value. An overestimated action is therefore likely to be chosen, its inflated value enters the next target through , and the error is copied backward. An underestimated action is avoided by the actor, so its low value rarely appears in a target.
The minimum also favors states whose estimates agree. For two estimates with the same mean and standard deviation , the expected minimum is , so the noisier the estimates of a state, the lower its clipped value. The actor is steered toward states and actions whose value the critics agree on, which the paper calls safer policy updates with stable learning targets.
Appendix A proves convergence for a tabular version: two lookup tables in a finite MDP, the next action chosen greedily by , and both tables updated toward the same target,
under the usual conditions (every state-action pair visited infinitely often, step sizes summing to infinity with finite sum of squares, , bounded reward variance). Since both tables move toward the same target with the same step, their difference shrinks by a factor at each update and goes to 0. The gap between the clipped target and the Q-learning target goes to 0 with it, and the proof reduces to the standard one for Q-learning.
How errors accumulate through bootstrapping
Section 5 turns from bias to variance. A critic trained by temporal-difference learning never satisfies the Bellman equation exactly. Write for the residual TD error each update leaves behind:
Substitute (11) for the on the right, again and again, along the trajectory. Each substitution adds one more discounted reward and one more discounted residual:
So the critic estimates the expected return minus the expected discounted sum of all future residuals. The sign convention in (11) makes a negative mean the estimate sits above its target; Figure 3 plots summed with discounts, which is exactly . With independent per-backup errors of mean and standard deviation , the sum of a geometric series gives the limits:
At the mean is multiplied by 100 and the standard deviation by 7.09. A backup that leaves the estimate 0.01 too high with noise of 0.1 produces a value estimate off by 1.0 on average with a spread of 0.71. At there is no bootstrap and each error stays its own size. The paper adds that each gradient step reduces the error only on its minibatch, with no guarantee for states outside it, so the per-backup residual is not something training drives to zero.
Target networks and delayed policy updates
A target network makes the target in (3) depend on a lagged copy of the critic, so many gradient steps fit nearly the same objective instead of chasing a target that shifts at every step. Its weights follow the online critic by Polyak averaging, ; means no target network at all. TD3 uses , so after target updates an old weight has half its influence left (the DDPG explainer has a figure of this averaging).
The paper's Figure 3 plots the value estimate of one randomly chosen Hopper state over 100,000 steps for of 1, 0.1 and 0.01. With the policy held fixed, all three settle near the true value; is only noisier. With the policy trained on the current critic, the estimates for and 0.1 climb into the thousands on a log-scale axis while the true value stays in the low hundreds, and stays close to it. The authors conclude that divergence comes from the interaction: the policy is updated on a high-variance value estimate, the value estimates grow through overestimation when the policy is poor, and the policy gets worse when the value estimate is wrong.
That suggests updating the actor less often than the critic, so that the critic's error has been reduced before the actor uses it. TD3 updates the actor, and all target networks, only once every critic updates, with . Over 1 million environment steps with one critic update per step, the actor receives 500,000 updates and the targets move 500,000 times, which with puts the target half-life at about 277 critic updates. The paper notes that a larger would reduce accumulated error further, but with the critic trained only once per step, training the actor too rarely would cripple learning. It also points out that this makes TD3 a two-timescale algorithm, the structure convergence proofs for linear actor-critic methods require (Konda and Tsitsiklis, 2003).
Target policy smoothing
A deterministic actor trained on a critic with a narrow, spurious peak will put its action on the peak, and the target then evaluates the critic exactly there, where the error is largest. Section 5.3 regularizes the target with the idea that similar actions should have similar values, and fits the value of a small region around the target action instead of a single point. The paper writes the target as an expectation over noise added to the target action, with the discount restored (the paper's equation (13) drops it; (14) and (15) have it):
In practice the expectation is replaced by a single noise draw per transition, and the averaging happens across the minibatch and across updates:
The paper uses (a standard deviation) and , for MuJoCo actions in . The clip keeps the noisy action close to the original: 0.5 is 2.5 standard deviations, so 1.24% of draws are clipped. Appendix D adds a second clip, of the noisy action to the valid action range, so the critic is never queried at an action the environment would reject. This noise is separate from exploration: the actions sent to the environment get Gaussian noise with standard deviation 0.1, and the smoothing noise exists only inside the target.
In Figure 4 the teal curve is a target critic made of a broad hump (amber dashed, the value it should have learned) plus a narrow spike of height 0.8 at , and the target action sits on the spike. The shaded window is the distribution of the noisy target action; the bars at its edges are the probability mass the clip piles up at .
At the paper's settings the deterministic target is 1.26, the smoothed target 0.54, and the hump alone 0.46. Without the spike the smoothed target would be 0.42, so smoothing cuts the spike's contribution from 0.80 to 0.12. The noise window is about seven times wider than the spike, so the spike covers a small share of it. With the smoothed target is still 0.87. The no-spike smoothed value of 0.42 sits below the hump's 0.46 at the same action, because averaging a curved value function over noisy actions lowers it: the critic learns the value of a policy that adds noise of size to its actions, as the paper notes. The paper relates the update to SARSA and expected SARSA, which learn the value of the behavior including its randomness, and observes that values learned this way favor actions that still do well when perturbed.
The full TD3 algorithm
TD3 is DDPG with all three changes. It keeps one actor and two critics, each with a target copy. At each step both critics regress onto the same target, built from the smaller target critic at a smoothed target action:
Every iterations the actor takes one step of the deterministic policy gradient on , and all three target networks take one soft update. For a transition that ended in a failure state the bootstrap term is dropped and ; when an episode ends because it reached the time limit (1,000 steps on the locomotion tasks), the transition is treated as not terminal, since the state itself still has future value. Algorithm 1, written as PyTorch for HalfCheetah (17-dimensional state, 6-dimensional action):
# One TD3 iteration on HalfCheetah: state 17, action 6, minibatch N = 100.
s, a, r, s2, not_done = buffer.sample(100) # 100x17, 100x6, 100x1, 100x17, 100x1
with torch.no_grad(): # the target y is a constant
eps = (0.2 * torch.randn(100, 6)).clamp(-0.5, 0.5) # smoothing noise
a2 = (actor_targ(s2) + eps).clamp(-1.0, 1.0) # 100x6, eq. (14)
q1_t, q2_t = critic_targ(s2, a2) # two target critics
y = r + not_done * 0.99 * torch.min(q1_t, q2_t) # 100x1, eq. (15)
q1, q2 = critic(s, a) # both critics, the stored action a
loss_q = ((q1 - y) ** 2).mean() + ((q2 - y) ** 2).mean() # same y for both
critic_opt.zero_grad(); loss_q.backward(); critic_opt.step()
it += 1
if it % 2 == 0: # delayed: every d = 2 iterations
loss_pi = -critic.Q1(s, actor(s)).mean() # actor ascends critic 1 only
actor_opt.zero_grad(); loss_pi.backward(); actor_opt.step()
for p_t, p in zip(target_params, online_params): # both critics + actor
p_t.mul_(1 - 0.005).add_(0.005 * p) # tau = 0.005Only the critics' parameters receive gradients from loss_q, and the target y is computed without gradient. loss_pi backpropagates through critic 1 into the actor, and only the actor's optimizer steps on it. The paper's networks are two hidden layers of 400 and 300 ReLU units with a tanh output on the actor, and the critic takes state and action concatenated at its first layer. For HalfCheetah that is 129,306 actor parameters and 130,201 per critic. Everything is trained with Adam at learning rate on minibatches of 100 drawn uniformly from a buffer holding the agent's whole history.
| Setting | TD3 | DDPG (baselines) |
|---|---|---|
| Critic / actor learning rate | 1e-3 / 1e-3 | 1e-3 / 1e-4 |
| Critic weight decay | none | 1e-2 |
| Target update rate τ | 0.005 | 0.001 |
| Batch size | 100 | 64 |
| Exploration noise | N(0, 0.1) | OU, θ 0.15, σ 0.2 |
| Normalized observations | no | yes |
| Policy delay d | 2 | 1 |
| Target noise σ, clip c | 0.2, 0.5 | none |
The first six rows are the paper's Table 3, which lists them for the re-tuned DDPG that TD3 is built on. Both use a discount of 0.99 and one training iteration per environment step. TD3 also starts with purely random actions to fill the buffer: 10,000 steps on HalfCheetah and Ant, which the paper calls stable-length environments, and 1,000 steps on the other tasks. The released code has drifted from these numbers since publication; the Provenance panel lists the differences.
Results on seven MuJoCo tasks
Each method ran 1 million steps on each of seven Gym tasks, with 10 random seeds, evaluated every 5,000 steps by the average return of 10 episodes without exploration noise. PPO, TRPO, ACKTR and DDPG come from OpenAI's baselines repository, SAC from its authors' code, and "Our DDPG" is the re-tuned DDPG with none of the three changes. Table 1 reports each method's maximum average return:
| Task | TD3 | DDPG | Our DDPG | PPO | TRPO | ACKTR | SAC |
|---|---|---|---|---|---|---|---|
| HalfCheetah | 9636.95 ± 859.065 | 3305.60 | 8577.29 | 1795.43 | −15.57 | 1450.46 | 2347.19 |
| Hopper | 3564.07 ± 114.74 | 2020.46 | 1860.02 | 2164.70 | 2471.30 | 2428.39 | 2996.66 |
| Walker2d | 4682.82 ± 539.64 | 1843.85 | 3098.11 | 3317.69 | 2321.47 | 1216.70 | 1283.67 |
| Ant | 4372.44 ± 1000.33 | 1005.30 | 888.77 | 1083.20 | −75.85 | 1821.94 | 655.35 |
| Reacher | −3.60 ± 0.56 | −6.51 | −4.01 | −6.18 | −111.43 | −4.26 | −4.44 |
| InvPendulum | 1000.00 ± 0.00 | 1000.00 | 1000.00 | 1000.00 | 985.40 | 1000.00 | 1000.00 |
| InvDoublePendulum | 9337.47 ± 14.96 | 9355.52 | 8369.95 | 8977.94 | 205.85 | 9081.92 | 8487.15 |
TD3 has the highest value on five tasks and ties on InvertedPendulum, where six of the seven methods reach the episode maximum of 1,000. On InvertedDoublePendulum the baselines DDPG is 18 points higher, about 1.2 of TD3's across-seed standard deviations (14.96); Table 1 gives no spread for the baselines. The largest gaps are on the harder locomotion tasks: on Ant, TD3's 4,372 is 2.4 times the next best (ACKTR's 1,822) and almost 5 times the re-tuned DDPG's 889, and on Walker2d it is 1.4 times PPO's 3,318. The comparison is one update per environment step for every off-policy method; Appendix E says the SAC numbers are lower than SAC's own paper partly because SAC was run with 1 update per step instead of 4, and over fewer steps. A maximum over evaluations takes the best point on each learning curve, which flatters methods whose curves are noisy; Table 2 averages the last 10 evaluations instead, and its TD3 HalfCheetah value is 9,533 against Table 1's 9,637.
Which of the three changes matters
Table 2 removes each change from TD3 and adds each to the re-tuned DDPG (labelled AHE, for architecture, hyperparameters and exploration), on four tasks. It also swaps clipped double Q-learning for the actor-critic forms of Double Q-learning (DQ-AC) and Double DQN (DDQN-AC), with delayed updates and smoothing kept in both.
On Hopper, adding any one change to AHE gives 908 to 1,465, against TD3's 3,305, and removing any one from TD3 gives 1,837 to 2,407; the changes help mostly in combination. On Ant, clipped double Q-learning does most of the work: AHE + CDQ reaches 3,819 and TD3 − CDQ falls to 850, and DQ-AC and DDQN-AC, which reduce overestimation less, land at 2,446 and 1,092. The paper interprets Ant as a task that benefits from any reduction in overestimation. On HalfCheetah the picture reverses: AHE already scores 8,401, the best result is DDQN-AC's 10,307, and removing CDQ or the delay from TD3 raises its score. TD3 has the highest value in two of the four tasks, Hopper and Ant, and is within 3% of the best on Walker2d. Removing the delay costs 897 points on Hopper and 431 on Ant while gaining 58 on HalfCheetah and 130 on Walker2d, and it doubles the number of actor updates; the paper counts the shorter training time among the delay's benefits.
Limitations and what came next
The bias argument of Section 4.1 is local: it holds for small enough actor steps and assumes the critic is not already below the truth at the true-gradient policy. The convergence proof is for lookup tables. The rest of the case is empirical, on seven Gym tasks with 10 seeds each, and the ablation shows the contribution of each change varies from task to task. Clipped double Q-learning trades overestimation for a bias downward, and target policy smoothing makes the critic learn the value of a slightly noisy policy rather than the deterministic one it serves. The delay and the noise scale were set for these tasks; the paper does not report a sweep of either.
None of the three changes depends on the rest of DDPG, and the paper's conclusion says they can be added to any actor-critic algorithm. Appendix E notes that the later version of Soft Actor-Critic includes clipped double Q-learning in its value update and is competitive with TD3 on most tasks. That version keeps two critics and the minimum, and replaces the deterministic actor with a stochastic one rewarded for entropy.
Questions you might still have
What does TD3 stand for, and what does it change in DDPG?
Twin Delayed Deep Deterministic policy gradient. It keeps DDPG's deterministic actor, replay buffer and soft target networks, and makes three changes: two critics whose smaller estimate forms the learning target (twin), actor and target-network updates only every second critic update (delayed), and clipped Gaussian noise added to the target action. The paper's DDPG baseline was also re-tuned: no batch normalization, no weight decay, Gaussian instead of Ornstein-Uhlenbeck exploration noise, the action fed to the critic's first layer.
Why take the minimum of the two critics instead of their average?
At the action the actor chose, critic 1's error is selected to be high, and critic 2's error is high too in proportion to how correlated the critics are. The average of the two keeps that upward bias, shrunk by a factor (1 + rho)/2 in the toy of Figure 2 on this page. The minimum can cancel it, and the paper argues that the underestimate the minimum produces in some states does less damage, because the actor moves away from actions whose value is underestimated instead of feeding them back into later targets.
Does clipped double Q-learning underestimate?
It can. When the two critics' errors are independent and the actor's choice has not inflated them, the minimum of two unbiased estimates sits about 0.56 standard deviations below the truth. The paper accepts this and gives the reason above. In its Figure 1 the clipped critic's estimate stays close to the true value of its policy on Hopper and Walker2d, while DDPG's ends well above.
Why is only the first critic used to train the actor?
For cost. The full double Q-learning form would train two actors, each against its own critic. The paper keeps one actor, trains it on Q1, and gives both critics the same target. When Q2 is above Q1 at the target action the target equals the ordinary one; when Q2 is lower, the target is pulled down.
Is TD3 on-policy or off-policy?
Off-policy. Like DDPG it learns from a replay buffer holding the agent's entire history, and the target uses the target actor's action at the next state, not the stored next action. The smoothing noise in the target is chosen separately from the exploration noise used to collect data.
How much of the gain comes from re-tuning DDPG rather than from the three changes?
It depends on the task. In Table 2 the re-tuned DDPG without any of the changes (AHE) already lifts HalfCheetah from 3,163 to 8,401, most of the way to TD3's 9,533. On the other three tasks re-tuning alone helps less or hurts: Hopper falls from 1,732 to 1,062, Walker2d rises from 1,521 to 2,362, and Ant falls from 816 to 564. Adding the three changes then takes them to 3,305, 4,565 and 4,185.
Does the convergence proof cover neural networks?
No. Appendix A proves that clipped double Q-learning converges to the optimal action values in a finite MDP with lookup tables, under the usual step-size and infinite-visit conditions, with both tables updated toward the same target. Nothing is proved for the deep, deterministic-actor version, whose case rests on the experiments.
Why does delaying the actor help if the critic is still trained once per step?
The critic and the target networks change between actor updates, so each actor update sees a critic that has taken d more gradient steps toward the current targets. With d = 2 the actor gets 500,000 updates in a 1-million-step run instead of a million, and Table 2 shows removing the delay costs on Hopper and Ant while slightly helping on HalfCheetah and Walker2d.
How is TD3 related to Soft Actor-Critic?
The two were published months apart in 2018, and TD3's Table 1 compares against the first SAC. Appendix E notes that a later version of SAC uses clipped double Q-learning in its value update and is competitive with TD3 on most tasks. SAC keeps the minimum over two critics and replaces the deterministic actor with a stochastic one trained with an entropy bonus.
Footnotes & further reading
- The paper: Fujimoto, van Hoof & Meger, Addressing Function Approximation Error in Actor-Critic Methods (ICML 2018). Code and the paper's learning curves: github.com/sfujim/TD3; the paper-era version is commit 25dfc0a.
- Overestimation and double estimators: Thrun & Schwartz, Issues in using function approximation for reinforcement learning (1993 Connectionist Models Summer School); van Hasselt, Double Q-learning (NeurIPS 2010); van Hasselt, Guez & Silver, Deep Reinforcement Learning with Double Q-learning (AAAI 2016), one of the six parts of Rainbow.
- The base algorithm: Silver et al., Deterministic Policy Gradient Algorithms (ICML 2014); Lillicrap et al., Continuous control with deep reinforcement learning (DDPG, ICLR 2016; explained in DDPG).
- Two-timescale actor-critic: Konda & Tsitsiklis, On actor-critic algorithms (SIAM Journal on Control and Optimization, 2003).
- The SAC comparison: Haarnoja et al., Soft Actor-Critic (ICML 2018; explained in SAC), and the later version with two critics, Haarnoja et al., Soft Actor-Critic Algorithms and Applications (2018).
How could this explainer be improved? Found an error, or something unclear? I read every message.