Soft Actor-Critic (SAC): Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Reward an agent for succeeding and for staying random.

Soft Actor-Critic (SAC) trains a controller that samples each action from a learned probability distribution. It is scored on reward plus the spread of that distribution, learns from a buffer of past experience, and ends up in nearly the same place on every random seed.

A January 2018 paper from Berkeley: four small networks, one hyperparameter tuned per task, and a simulated humanoid with 21 action dimensions that DDPG made no progress on.

In continuous control an agent picks a vector of real numbers every step (joint torques for a simulated robot, for example) and receives a reward. Deep reinforcement learning for this setting had two practical problems in 2017. The stable methods, TRPO and PPO, learn only from experience the current policy collected, so they need millions of environment steps. The method that reused old experience, DDPG, needed far fewer steps, but the SAC paper describes it as notoriously challenging to use because of its extreme brittleness and hyperparameter sensitivity.

SAC keeps DDPG's reuse of old experience and changes the objective: the agent maximizes reward plus the entropy of its own action distribution. The page follows the paper's order: the maximum-entropy objective and its temperature; soft value functions, and why the reward scale is the temperature; the policy-improvement step, a KL projection onto Gaussians; the four networks and their three losses; the reparameterized actor gradient and the tanh squashing of actions; then the experiments, and how this 2018 paper differs from the SAC in today's libraries.

Why off-policy control was hard before SAC

Reinforcement learning methods split by which data they can learn from. An on-policy method such as TRPO or PPO computes each update from experience collected by the current policy. PPO reuses a batch for several epochs of minibatch updates, but once the policy has moved, the batch describes a different policy, so it is discarded and new rollouts are collected. The SAC paper reports that it learns considerably faster than PPO and attributes the gap to the large batches PPO needs to learn stably on the harder tasks.

An off-policy method stores every transition (state, action, reward, next state) in a replay buffer and trains on random minibatches drawn from it, so one real step can feed many gradient updates. DQN did this for discrete actions, and its 2015 Nature version (Mnih et al.) added a slowly updated target network. For continuous actions the standard off-policy method was DDPG (Lillicrap et al., 2015): a deterministic actor that outputs one action per state, trained to climb the gradient of a learned Q-function, with fixed noise added to its actions for exploration.

Off-policy value learning is fragile for a mechanical reason. The Q-network is trained toward a target computed from its own current estimates (bootstrapping); it is a neural network, so an update at one state changes its outputs at nearby states (function approximation); and its data comes from older policies (off-policy). With all three, an error in one estimate can raise the targets of others and grow instead of shrinking. The paper names the combination of off-policy learning and nonlinear function approximation, citing Bhatnagar et al. (2009); Sutton and Barto's textbook calls the three together the deadly triad. A deterministic actor adds to it, because it moves toward whichever action the critic currently rates highest, including actions the critic overrates.

The maximum-entropy objective

Standard reinforcement learning maximizes the expected sum of rewards. The maximum-entropy framework (Ziebart, 2010) adds the entropy of the policy's action distribution at every state the policy visits:

The expectation runs over the states and actions the policy visits, written $\rho_\pi$. The entropy $\mathcal{H}(\pi) = \mathbb{E}[-\log\pi]$ measures how spread out the action distribution is. Over four actions it ranges from 0 nats (always the same action) to $\log 4 \approx 1.39$ nats (uniform). For a continuous action it is the differential entropy, which can be negative: a one-dimensional Gaussian with standard deviation $\sigma$ has entropy $\tfrac12\log(2\pi e\sigma^2)$, which is −0.88 nats at $\sigma = 0.1$ and 1.42 nats at $\sigma = 1$.

The temperature $\alpha$ converts entropy into units of reward. As $\alpha \to 0$ the entropy term vanishes and (1) becomes the conventional objective; a large $\alpha$ makes the agent value randomness over reward. Equation (1) is a finite sum from $t = 0$ to $T$ with no discount factor. The discount $\gamma = 0.99$ enters in the soft Bellman backup (2) below; the paper defers the discounted, infinite-horizon form of the objective to its Appendix A because writing it down is more involved (Thomas, 2014).

The paper gives three reasons to want the entropy term, and all three come from the policy keeping probability on more than one action. A policy that collapses onto one action loses entropy, so the objective pushes it to keep trying others, which is exploration. When two actions are nearly equally good, the maximum-entropy policy keeps probability on both instead of breaking the tie. And the paper cites earlier work showing that such policies hold up better under errors in the model or in the value estimates.

For one state with return $R(a)$ over a one-dimensional action, the policy that maximizes $\mathbb{E}[R] + \alpha\mathcal{H}$ is $\pi^*(a) \propto \exp(R(a)/\alpha)$, derived in the next section. Figure 1 plots it for a return with two peaks, the left one higher. Below about $\alpha = 0.12$ the policy is a spike on the higher peak, the choice a greedy method makes; at the default it has a second, smaller bump on the right peak; above about 1.2 it is close to uniform and the reward barely moves it.

The greedy spike puts all its probability on one action, so if that action leads somewhere bad or the environment changes so that the right peak becomes better, the agent has no experience with the alternative. At a moderate temperature the agent keeps sampling the second option and keeps its value estimate for it up to date.

Soft value functions and the reward scale

Optimizing (1) needs value functions that count future entropy as well as future reward. These are called soft. The paper defines the soft Q-function through a soft Bellman backup operator:

with the soft state value

Equation (2) has the shape of the ordinary Bellman equation: the value of taking action $\mathbf{a}_t$ in state $\mathbf{s}_t$ is the immediate reward plus the discounted value of the next state. The difference is in $V$. Since $\mathcal{H}(\pi) = \mathbb{E}[-\log\pi]$, equation (3) says the soft value is the expected Q under the policy plus the entropy at that state. $Q$ bootstraps from $V$ at the next state, which contains the entropy there and a $Q$ that bootstraps one step further, so a soft Q-value counts the discounted entropy the policy will collect along the rest of the trajectory as well as the reward. Equation (3) is the value of the current policy $\pi$, not of the optimal one.

The temperature is missing from (2) and (3) because the paper folds it into the reward. The objective $r + \alpha\mathcal{H}$ equals $\alpha\,(r/\alpha + \mathcal{H})$, and multiplying an objective by a positive constant does not change which policy maximizes it, so the paper divides every reward by $\alpha$ and writes every equation from (2) on with $\alpha = 1$ and a bare $-\log\pi$.

That makes the temperature and the reward scale the same knob: multiplying every reward by a constant $c$ sets $\alpha = 1/c$. A larger reward scale makes reward outweigh the fixed entropy bonus, so the optimal policy becomes narrower and more deterministic; a smaller one makes it wider. In standard RL, multiplying all rewards by a positive constant leaves the optimal policy unchanged. In maximum-entropy RL it changes the policy. The paper found the reward scale to be the only hyperparameter it had to tune per task: 5 for Hopper, Walker2d, HalfCheetah and Ant, 20 for Humanoid-v1 and 10 for the rllab Humanoid, which correspond to $\alpha$ = 0.2, 0.05 and 0.1.

The best policy for a fixed set of soft Q-values has a closed form. Take one state with four actions whose soft Q-values are 2.0, 1.5, 0.5 and −1.0, at reward scale $c = 1$. The greedy policy always picks the first action: $\mathbb{E}[Q] = 2.00$, $\mathcal{H} = 0$, so $V = 2.00$. The uniform policy has $\mathbb{E}[Q] = 0.75$ and $\mathcal{H} = \log 4 = 1.39$, so $V = 2.14$. The policy proportional to $\exp Q$ puts probabilities 0.53, 0.32, 0.12 and 0.03 on the four actions, with $\mathbb{E}[Q] = 1.58$ and $\mathcal{H} = 1.05$, so $V = 2.63$. No distribution over the four actions does better, because for any policy $\pi$

To see it, expand the KL:

The KL is never negative and is zero only when $\pi = \exp(Q)/Z$, so the maximum soft value is $\log Z = 2.63$, reached by that one policy. This is the Boltzmann, or energy-based, policy. With the temperature written back in it is $\pi \propto \exp(Q/\alpha)$, and as $\alpha \to 0$ it becomes the greedy argmax.

Figure 2 runs this example. The bars are a candidate policy $\pi \propto \exp(Q/T)$, and the curve is its soft value as $T$ sweeps from greedy to uniform. At $c = 1$ the peak sits at $T = 1$. Set the reward scale to 5 and the peak moves to $T = 0.2$, where the best policy is nearly greedy, with probabilities 0.92, 0.08, 0.00 and 0.00.

SAC trains its policy toward this Boltzmann distribution, $\exp(Q(\mathbf{s},\cdot))/Z(\mathbf{s})$, at every state. For a continuous action the distribution can have any shape and $Z$ is an integral over the action space, so the next step needs an approximation.

How SAC improves the policy: a KL projection

Policy iteration alternates two steps: evaluate the current policy (compute its Q-values), then improve it (make it greedy with respect to those Q-values). Soft policy iteration evaluates with the soft backup (2) and replaces $\arg\max_a Q$ with the Boltzmann policy $\exp(Q)/Z$.

For a neural Q-function over a continuous action, the Boltzmann policy can have any number of peaks, and neither sampling from it nor computing $Z(\mathbf{s}) = \int \exp(Q(\mathbf{s},\mathbf{a}))\,d\mathbf{a}$ has a closed form. SAC restricts the policy to a tractable family $\Pi$, in practice Gaussians squashed by $\tanh$, and picks the member of $\Pi$ closest to the Boltzmann target in KL divergence:

The new policy $\pi'$ is in the first slot. The paper calls this the information projection; it is the reverse KL, as opposed to the forward KL with the target first. Expanding it as in the four-action example gives $-(\mathbb{E}_{\pi'}[Q] + \mathcal{H}(\pi')) + \log Z$, so minimizing (4) over $\pi'$ is maximizing the soft value within $\Pi$.

The direction of the KL decides what a single Gaussian does when the target has two peaks. The reverse KL averages $\log\pi' - \log p^*$ over samples from $\pi'$, where $p^*$ is the target. A Gaussian sitting in the valley between the peaks draws its samples where $p^*$ is small, so $\log p^*$ is very negative there and the divergence is large. A Gaussian sitting on one peak draws its samples where $p^*$ is large, and the other peak costs it nothing, because it draws no samples there. So the reverse KL is mode-seeking: the best narrow Gaussian sits on one peak. The forward KL averages over samples from the target instead, charges the Gaussian for every region of target mass it fails to cover, and at a fixed narrow width puts the mean between the peaks.

In Figure 3 the target is the Boltzmann policy, at $\alpha = 0.3$, of a two-peak return like the one in Figure 1. With the default width $\sigma = 0.16$, the Gaussian starts in the valley at a reverse KL of 1.85 nats, and the best mean is −0.44, on the left peak, at 0.52 nats. Switch to the forward KL and the best mean moves to −0.06, in the valley, close to the target's overall mean. Widen $\sigma$ past about 0.6 and both directions put the mean near the middle, because a Gaussian that wide covers both peaks.

Writing the improvement as (4) has two consequences. The normalizer $Z(\mathbf{s})$ depends on the state and the Q-function but not on the policy's parameters, so its gradient with respect to them is exactly zero and SAC never computes it. And the paper proves (Lemma 2) that the projected policy has soft Q-values at least as high as the old policy's at every state-action pair, so alternating evaluation and projection improves the policy monotonically; Theorem 1 says the alternation converges to the best maximum-entropy policy within $\Pi$. Both results assume a finite action set with exact evaluation and exact projection, the tabular setting. The deep algorithm below replaces each exact step with one gradient step on a neural network and has no convergence proof.

The networks and the replay buffer

The deep algorithm approximates both halves of soft policy iteration with networks trained by stochastic gradient descent, interleaved with collecting data. This paper (SAC version one) trains four networks:

• the policy $\pi_\phi$, the actor: state in, mean and log standard deviation of a Gaussian over the $D$-dimensional pre-squash action out;
• two soft Q-networks $Q_{\theta_1}$ and $Q_{\theta_2}$, the critics: state and action in, one number out;
• a soft value network $V_\psi$: state in, one number out.

There is also exactly one target network, $V_{\bar\psi}$, an exponential moving average of the value network's weights that is never trained by gradient. Every network has two hidden layers of 256 ReLU units. The value network is redundant in principle, since (3) gives $V$ from $Q$ and $\pi$; the paper keeps it because, it reports, a separate value approximator stabilizes training and is convenient to train alongside the others.

Figure 4 is the outer loop. The current policy samples an action, the environment returns a reward and a next state, and the transition $(\mathbf{s}, \mathbf{a}, r, \mathbf{s}')$ goes into a buffer holding the last $10^6$ transitions. Then one gradient step updates every network on a minibatch of 256 transitions drawn at random from the buffer, most of them collected by older versions of the policy.

Every loss below can be computed on transitions from an older policy. The Q-loss uses the stored state, action, reward and next state. The value and policy losses use the stored state but draw a fresh action from the current policy, so they never need the action the old policy took. The paper points to this property as the reason the stochastic actor can be trained entirely on replayed data.

How SAC trains the critics

The value network regresses onto the right-hand side of (3):

The state $\mathbf{s}_t$ comes from the buffer $\mathcal{D}$; the action inside the target is sampled from the current policy, because (3) is an expectation under the current policy. The paper estimates the gradient from one such action:

Both Q-networks regress onto one target built from the stored reward and the target value network at the stored next state:

Here the state and the action both come from the buffer, so this loss is fully off-policy. Its one-sample gradient, the paper's (9), is $\nabla_\theta Q_\theta$ times the residual $Q_\theta - r - \gamma V_{\bar\psi}(\mathbf{s}_{t+1})$. (The paper prints the first Q in (9) with its arguments swapped, $Q_\theta(\mathbf{a}_t,\mathbf{s}_t)$; the meaning is the same.)

After every gradient step the target network moves by $\bar\psi \leftarrow \tau\psi + (1-\tau)\bar\psi$ with $\tau = 0.005$. The gap between $\bar\psi$ and $\psi$ shrinks by a factor of 0.995 per step, to 37% after 200 steps (a half-life of about 138 steps), so the Q-target drifts gradually even though $V_\psi$ changes every step. The paper credits target networks to Mnih et al. (2015). Its Figure 3(c) sweeps $\tau$ on Ant-v1 from 0.0001 to 0.1: large values are unstable, small values train slowly, the range of good values is wide, and 0.005 is used on every task. A separate variant copies the weights outright every 1,000 steps ($\tau = 1$, a hard update).

SAC trains two Q-networks with independent parameters $\theta_1$ and $\theta_2$ on the same loss (7), and uses the smaller of their two outputs in the value gradient (6) and in the policy gradient (13), to counter an upward bias. The actor moves toward actions with high Q, and the value target averages Q under that actor, so an action whose Q the critic overrates gets chosen more often and lifts the value target; through the bootstrap in (8), the inflated value then lifts other Q-targets. The paper cites van Hasselt (2010) for this positive bias and takes the minimum of two Q-functions from TD3 (Fujimoto et al., 2018), which was published concurrently.

Figure 5 isolates the statistics. When two estimates of the same true value $q^*$ carry independent Gaussian errors with standard deviation $\sigma$, their maximum averages $q^* + \sigma/\sqrt{\pi}$ and their minimum $q^* - \sigma/\sqrt{\pi}$, about $\pm 0.56\,\sigma$. SAC takes the minimum and accepts a small underestimate in exchange for offsetting the overestimate that a max-seeking actor would otherwise feed into the bootstrap. Averaging the two critics would shrink the noise, but the actor would still select the actions whose average is overrated.

The minimum is not in the Q-target: (8) contains only $V_{\bar\psi}$, and both Q-networks bootstrap from it. And the second critic is a speedup, not a requirement: the paper reports that a single Q-function learns even the 21-dimensional rllab Humanoid, and that two Q-functions significantly speed up training, especially on the harder tasks. The authors' released code applies the minimum in the value target only; its reparameterized policy loss uses $Q_{\theta_1}$ alone.

How SAC trains the actor

The policy minimizes the expected KL (4) over states from the buffer:

Expanding the KL and dropping $\log Z$, which does not depend on $\phi$, leaves an expectation of $\log\pi_\phi(\mathbf{a}\mid\mathbf{s}) - Q_\theta(\mathbf{s},\mathbf{a})$ over actions sampled from $\pi_\phi$. The sampling distribution itself depends on $\phi$, so the gradient cannot be moved inside the expectation.

One standard answer is the likelihood-ratio estimator (Williams, 1992), also called REINFORCE: $\nabla_\phi \mathbb{E}[f(\mathbf{a})]$ equals the expectation of $f(\mathbf{a})\,\nabla_\phi\log\pi_\phi(\mathbf{a}\mid\mathbf{s})$. It needs only the value of $f$ at the sampled action, so it works even when $f$ cannot be differentiated. SAC's $f$ contains the critic, a neural network that can be differentiated with respect to its action input, and the paper uses the reparameterization trick instead, stating that it gives a lower-variance estimator. The action is written as a deterministic function of the state and an independent noise sample:

where $\epsilon_t$ comes from a fixed distribution, such as a spherical Gaussian, that does not depend on $\phi$. For SAC's policy the function is

with $\odot$ the elementwise product. The expectation over actions becomes an expectation over noise, and the gradient can move inside it:

with the one-sample gradient, evaluated at $\mathbf{a}_t = f_\phi(\epsilon_t;\mathbf{s}_t)$,

Take (13) term by term. $\nabla_{\mathbf{a}} Q$ is the critic's slope with respect to the action, and $\nabla_\phi f_\phi$ says how the sampled action moves when the policy parameters move; their product, with the minus sign, moves the action uphill on the critic, the same gradient DDPG uses for its deterministic actor. The two $\log\pi$ terms are the derivative of the entropy part: the first differentiates the density with the action held fixed, the second accounts for the change in $\log\pi$ as the action moves. The paper calls (13) an unbiased estimator that extends DDPG-style policy gradients to any tractable stochastic policy. The gradient passes through only $f_\phi$ and $Q_\theta$, with no model of the environment.

$J_\pi$ is a loss that Algorithm 1 minimizes by gradient descent, while $J(\pi)$ in (1) is an objective to maximize; the paper uses the same letter for both. Minimizing $\mathbb{E}[\log\pi - Q]$ raises $\mathbb{E}[Q]$ and raises the entropy $\mathbb{E}[-\log\pi]$, which is gradient ascent on the soft value.

Figure 6 compares the two estimators on a one-dimensional problem of our own: critic $Q(a) = -a^2$, policy $\mathcal{N}(\mu, \sigma^2)$, and the quantity to estimate is the derivative of $\mathbb{E}[Q(a)]$ with respect to $\mu$, which is $-2\mu$. From one noise sample $\epsilon$, the pathwise estimate is $Q'(\mu + \sigma\epsilon) = -2(\mu + \sigma\epsilon)$, with standard deviation $2\sigma$. The likelihood-ratio estimate is $Q(\mu + \sigma\epsilon)\,\epsilon/\sigma$. At $\mu = 1$ and $\sigma = 0.5$ the true gradient is −2, and the standard deviations are 1.00 for pathwise and 4.66 for likelihood ratio, or 3.24 with the usual baseline $\mathbb{E}[Q]$ subtracted. As $\sigma \to 0$ the pathwise spread goes to zero, leaving DDPG's deterministic gradient, while the likelihood-ratio spread without a baseline grows like $1/\sigma$.

The pathwise estimate is tighter because it uses the critic's slope at the sampled action, which says which direction in action space raises Q. The likelihood-ratio estimate uses only the critic's value at the sample and recovers the direction only through the correlation between those values and the noise, averaged over many samples.

Squashing actions into bounds

Real actuators are bounded, say a torque in $[-1, 1]$, and a Gaussian proposes values anywhere on the real line. SAC samples an unbounded $\mathbf{u}$ from the Gaussian and outputs $\mathbf{a} = \tanh(\mathbf{u})$, applied to each coordinate, which maps every real number into $(-1, 1)$. The density of $\mathbf{a}$ follows from the change-of-variables formula (the paper's Appendix C):

In (20) and (21), $\mu(\mathbf{u}\mid\mathbf{s})$ denotes the Gaussian density, not its mean. Because $\tanh$ acts on each coordinate separately, the Jacobian $d\mathbf{a}/d\mathbf{u}$ is diagonal with entries $1 - \tanh^2(u_i)$, and its log-determinant is a sum over the $D$ action dimensions; this holds whether or not the Gaussian itself is diagonal. Each entry is at most 1, so each log is at most 0, and subtracting it raises the log-probability. At $u_i = 0$ the term is 0. At $u_i = 2$, $\tanh(2) = 0.964$, $1 - 0.964^2 = 0.071$, and the correction adds 2.65 nats in that dimension: tanh compresses a stretch of $u$ into a short stretch of $a$ near the bound, which raises the density there.

In Figure 7 the teal Gaussian along the bottom is the distribution of $u$, and the amber density down the left side is the resulting distribution of the bounded action. Move the mean or widen the Gaussian until it reaches the flat tails of tanh, and the action density piles up against $\pm 1$.

Computing the correction as log(1 - tanh(u)**2) fails near the bounds. In 32-bit floats $\tanh(10)$ rounds to exactly 1.0, so the expression evaluates to $\log 0 = -\infty$, while the true value is −18.61. An exact rewrite that stays finite:

which follows from $1 - \tanh^2 u = 4/(e^{u} + e^{-u})^2$. The authors' released code takes a different route and adds $10^{-6}$ inside the log, which caps each term at $\log 10^{-6} \approx -13.8$.

A full SAC update in code

The listing is one gradient step in PyTorch-style pseudocode, following Algorithm 1: the value network first, then both Q-networks, then the policy, then the target network. The minimum of the two critics appears in the value target and the policy loss and not in the Q-target. The value target uses a plain sample, and the policy loss uses rsample, the reparameterized draw that keeps the gradient path from the action back to the policy's weights. The reward-scale multiply and the (1 - done) mask on terminal states are in the released code but not in the paper's equations.

# one SAC (v1) gradient step on a replayed minibatch; B = 256, D = action dims
def step(opt, loss):
    opt.zero_grad(); loss.backward(); opt.step()

s, a, r, s2, done = buffer.sample(B)   # s (B,obs)  a (B,D)  r (B,)  s2 (B,obs)
r = reward_scale * r                   # c = 1/alpha, e.g. 5 for Hopper

# value network, eq (5): target E_pi[min Q - log pi] with a FRESH action at s
a_new, logp = pi.sample(s)             # (B,D), (B,); no gradient needed here
q_min = torch.min(Q1(s, a_new), Q2(s, a_new))
step(opt_V, 0.5 * ((V(s) - (q_min - logp).detach()) ** 2).mean())

# both Q-networks, eqs (7)-(8): stored (s, a), target from Vbar, no min
q_hat = (r + gamma * (1 - done) * Vbar(s2)).detach()
step(opt_Q, 0.5 * ((Q1(s, a) - q_hat) ** 2).mean()
          + 0.5 * ((Q2(s, a) - q_hat) ** 2).mean())

# policy, eq (12): reparameterized sample, minimize E[log pi - Q]
a_rp, logp_rp = pi.rsample(s)          # a = tanh(mu + sigma * eps), keeps grad
q_rp = torch.min(Q1(s, a_rp), Q2(s, a_rp))   # paper; released code: Q1 only
step(opt_pi, (logp_rp - q_rp).mean())  # opt_pi steps the policy's weights only

# target value network: EMA, tau = 0.005
for p, p_bar in zip(V.parameters(), Vbar.parameters()):
    p_bar.data.mul_(1 - tau).add_(tau * p.data)

For HalfCheetah, with $D = 6$ action dimensions, a minibatch holds 256 states. The policy network outputs a 256 × 6 mean and a 256 × 6 log standard deviation; the noise, the pre-squash $\mathbf{u}$ and the action are 256 × 6; the log-probability sums over the 6 dimensions into 256 numbers; each Q-network takes the state and the 6-dimensional action and returns 256 numbers, as does the value network from the state alone. The sampling step, with the tanh correction:

# tanh-squashed Gaussian policy: sample and log-prob, eqs (11) and (21)
mu, log_sigma = policy_net(s)                  # each (B,D)
u = mu + log_sigma.exp() * torch.randn_like(mu)  # reparameterized, (B,D)
a = torch.tanh(u)                              # every coordinate in (-1, 1)
logp = Normal(mu, log_sigma.exp()).log_prob(u).sum(-1)       # log mu(u|s), (B,)
logp -= (2 * (log(2) - u - F.softplus(-2 * u))).sum(-1)      # tanh correction

Every task in the paper uses the same settings: Adam with learning rate $3 \times 10^{-4}$, discount $\gamma = 0.99$, a buffer of $10^6$ transitions, minibatches of 256, two hidden layers of 256 ReLU units in every network, $\tau = 0.005$, a target update every step, and one gradient step per environment step; the reward scale is set per task as listed above.

Results: sample efficiency and seed stability

The paper trains five seeds of each method on six benchmarks (Hopper-v1, Walker2d-v1, HalfCheetah-v1, Ant-v1, Humanoid-v1 and the rllab Humanoid), with one evaluation rollout every 1,000 environment steps. The baselines are DDPG, PPO, Soft Q-Learning (SQL, the authors' earlier maximum-entropy method) and TD3, with Trust-PCL in an appendix. SAC performs comparably to the baselines on the easier tasks and outperforms them by a large margin on the harder ones, in learning speed and in final return. DDPG fails to make any progress on Ant-v1, Humanoid-v1 and the rllab Humanoid. SQL learns every task, more slowly than SAC and to lower final returns.

The stability claim comes from one experiment, on the rllab Humanoid. Five seeds of SAC are compared with five seeds of a deterministic variant that drops the entropy term. Apart from that, the variant resembles DDPG, except that it keeps two Q-functions, uses hard target updates, has no separate target actor, and explores with fixed rather than learned noise. The SAC runs end close together and the deterministic runs vary widely across seeds. The paper concludes that stochasticity can stabilize training, which matters most on hard tasks where tuning hyperparameters is difficult. Figure 8 redraws the comparison schematically.

The paper gives two mechanisms. In the policy, the entropy term prevents the policy's variance from collapsing too early (the paper points to (10)). In the value function, it raises the value of states from which the agent can keep acting with high entropy, which encourages exploration (the paper points to (5)). The evidence is the one task and five seeds per arm, so it supports the claim on the rllab Humanoid rather than measuring it across the benchmark.

The ablations on Ant-v1 cover three more choices. Sweeping the reward scale over 1, 3, 10, 30 and 100 shows the temperature's two failure modes: at small scales the policy becomes nearly uniform and fails to exploit the reward, and at large scales it learns quickly at first, then becomes nearly deterministic and gets stuck in poor local optima. The $\tau$ sweep is described above. And for evaluation the paper uses the mean action rather than a sample from the policy, which generally gives higher returns. All the curves plot the sum of rewards, not the entropy-augmented objective SAC optimizes.

This paper versus the SAC in libraries

The SAC implementations in Stable-Baselines3, CleanRL and Spinning Up follow a separate, later paper by the same group, Soft Actor-Critic Algorithms and Applications (Haarnoja et al., December 2018, arXiv:1812.05905). It drops the separate value network, bootstraps the Q-targets from target copies of the two Q-networks (taking the minimum of the two there), and learns $\alpha$ during training so that the policy's entropy stays near a target value, which replaces the per-task reward-scale tuning this paper needed. The maximum-entropy objective, the soft Bellman backup, the KL projection, the reparameterized tanh-Gaussian actor and the two critics are all already in this paper.

The paper's own claim of novelty is narrower than "the first off-policy actor-critic", a title DDPG already holds: to the authors' knowledge, SAC is the first off-policy actor-critic method in the maximum-entropy framework. Its predecessor in that framework, Soft Q-Learning, trains a Q-function for the optimal policy and a separate sampler that affects learning only through the data it collects; SAC's actor is the policy whose soft value the critics estimate.