Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Reward an agent for succeeding and for staying random.

Soft Actor-Critic adds a bonus for acting randomly to an off-policy actor-critic. The agent keeps exploring instead of committing early, so it learns hard control tasks from replayed data and lands in nearly the same place on every random seed.

A simulated robot has to learn to run from scratch. Two things usually go wrong: it burns through millions of tries before it improves, and a policy that runs on one random seed collapses on the next.

Those two failures, sample cost and brittleness, are what kept deep reinforcement learning out of the real world for a long time. A robot that needs ten million tries can learn in a simulator but never on real hardware, and a method whose success depends on the random seed is not a method you can ship. Soft Actor-Critic is one clean answer to both, and it became the default off-policy algorithm for continuous control because it works without the babysitting its predecessors needed.

The paper rests on one unusual idea: change what the agent is trying to do. Instead of maximizing reward alone, it maximizes reward plus the randomness of its own behavior. That one addition ripples through every equation, and by the end it buys sample efficiency, exploration, and the seed-to-seed stability the opening scenario was missing. A handful of ideas carry it: the maximum-entropy objective, soft value functions, a policy-improvement step that fits a tractable policy to an ideal one, and two standard tricks (a replay buffer and a reparameterized gradient) that make it all trainable.

Two failures of deep RL

Reinforcement learning splits into two camps by how they use data. On-policy methods like TRPO and PPO improve the policy only from data the current policy collected. The moment the policy changes, the old data no longer describes it, so it has to be dropped and fresh rollouts collected. (This is not one gradient step per sample: PPO reuses each batch for several epochs. The limit is that once the policy has moved enough, the batch is stale and goes in the bin.) That is safe and stable, and it is expensive, because every improvement step starts by throwing away what you just learned from.

Off-policy methods keep every transition the agent has ever seen in a replay buffer and reuse it. DQN made this work for discrete actions: a value network trains on random batches from the buffer, so a single real step of experience feeds many gradient updates. Off-policy methods get their sample efficiency from that reuse. For continuous actions the standard off-policy method was DDPG (Lillicrap et al., 2015), a deterministic actor-critic. It is sample-efficient and, as the paper puts it, notoriously brittle: it works only after careful per-task tuning, and different seeds diverge.

The reason off-policy learning is delicate has a name. When you combine bootstrapping (training a value estimate partly on your own current estimate), function approximation (a neural network), and off-policy data (from an out-of-date policy), the three together can make the value estimates spiral instead of converge. SAC keeps the sample efficiency of the off-policy camp without its instability, so the question it answers is: what makes an off-policy actor-critic stable? SAC answers by changing the objective.

Reward, and also randomness

Standard reinforcement learning maximizes the expected sum of rewards. The maximum-entropy framework (Ziebart, 2010) adds a second term to the objective: the entropy of the policy, a measure of how random its action choices are. The agent is now graded on two things at once, doing well and staying unpredictable:

Read it left to right. Over the trajectory the policy induces, sum up the reward $r$ at each step, plus $\alpha$ times the entropy $\mathcal{H}$ of the action distribution at that state. Entropy is highest when the policy spreads probability evenly over actions and zero when it always picks the same one, so the second term rewards the policy for keeping its options open. The temperature $\alpha$ sets the exchange rate between the two: at $\alpha \to 0$ the entropy term vanishes and you recover ordinary reward maximization, and as $\alpha$ grows the agent cares more about acting randomly than about the reward.

This objective, as written, has a detail worth pausing on: it is a finite sum from $t=0$ to $T$ with no discount factor inside it. The discount $\gamma = 0.99$ shows up later, in the value backup, not here. (The fully rigorous discounted, infinite-horizon version of a maximum-entropy objective is genuinely fiddly to write down, which is why the paper defers it to an appendix.) Keep $\alpha$ in view; it returns as the single knob you actually tune.

Why would you want a deliberately random policy? Three reasons, and they are one mechanism with three jobs. The entropy bonus makes the agent explore, because a policy that collapses onto one action scores badly on the entropy term and gets pulled back open. It makes the agent robust, because a policy that keeps several good options alive degrades gracefully when the world shifts under it. And it lets the agent represent genuinely multimodal behavior: when two different actions are about equally good, standard RL has to break the tie and pick one, while a maximum-entropy policy keeps meaningful probability on both.

That last effect is the one to see rather than take on faith. The figure shows a return over a one-dimensional action with two good choices, the left one slightly better. The teal curve is the optimal maximum-entropy policy, which is proportional to $\exp\!\big(R(a)/\alpha\big)$ (the next section derives why). Drag the temperature and watch the policy change character: near zero it is a spike on the single best action, exactly what a greedy method would pick; raise it and the policy fattens until it covers both modes; raise it further and it washes out toward uniform and stops caring about the reward at all.

The greedy spike is the standard-RL answer, and you can see what it costs. It commits everything to one action and keeps none of its bets on the other, so if that action is a trap, or the environment shifts so the second mode becomes better, the greedy policy has nothing in reserve. The maximum-entropy policy at a sensible temperature hedges, and that hedging is exactly the exploration and robustness the paper is after.

Soft values and the reward scale

To optimize this objective you need value functions that account for the entropy bonus, not just the reward. In ordinary RL the value of a state is the reward-to-go. Here it has to be the reward and entropy to go, and the versions that carry that extra term are called soft. The soft Q-function is defined by a modified Bellman backup:

This is the familiar Bellman equation, the value of taking action $\mathbf{a}_t$ now is the immediate reward plus the discounted value of wherever you land. What makes it soft is the value $V$ it bootstraps from:

The soft state value averages the Q-value under the policy and adds a $-\log\pi$ term. That extra term is the entropy in disguise: the entropy of a distribution is $\mathcal{H}(\pi) = \mathbb{E}[-\log\pi]$, so equation (3) says $V = \mathbb{E}[Q] + \mathcal{H}(\pi)$, expected value plus a bonus for randomness. Because $V$ carries that bonus and $Q$ bootstraps off $V$ at every step, the entropy reward compounds along the entire trajectory. A soft value is not a one-time softened maximum; it counts all the future entropy the agent will collect.

A note on the constant. Where did $\alpha$ go in (3)? The paper folds the temperature into the reward by rescaling $r \to r/\alpha$, so from here on every equation is written with $\alpha = 1$ and the entropy term appears as a bare $-\log\pi$ with coefficient one. This is not a different objective, only a tidier bookkeeping of the same one.

Folding $\alpha$ into the reward has a concrete, useful consequence: the temperature is now the same thing as the reward scale. Multiply all rewards by a constant and you are setting $1/\alpha$. Turn the reward scale up and reward dwarfs the fixed entropy bonus, so the optimal policy goes peaky and deterministic; turn it down and the entropy bonus dominates, so the policy goes flat and nearly uniform. This is the same dial you were dragging in Figure 1 above, read as reward scale rather than temperature. It matters because it flips a rule you may be carrying from standard RL. There, scaling all rewards by a positive constant changes nothing, since it does not move the argmax. In maximum-entropy RL it changes the answer, because it reweights reward against entropy. The paper found the reward scale to be the single hyperparameter that needs per-task tuning, and it has a sweet spot: too large and the policy collapses to near-deterministic early and gets stuck in a poor local optimum, too small and it stays so random it ignores the reward.

The soft view hands you one more thing for the price of the algebra: the best possible maximum-entropy policy has a closed form. If you fix the soft Q-values and ask which action distribution maximizes $\mathbb{E}[Q] + \mathcal{H}(\pi)$, the answer is the Boltzmann (energy-based) policy $\pi^*(\mathbf{a}\mid\mathbf{s}) \propto \exp\!\big(Q(\mathbf{s},\mathbf{a})/\alpha\big)$: probability grows exponentially with value, and $\alpha$ sets how sharply. That exponential is the teal curve from Figure 1, and it is the target the improvement step moves toward.

Fitting the soft-greedy policy

Classical policy iteration alternates two steps: evaluate the current policy's values, then make the policy greedier with respect to them. Standard greedy means jump to $\arg\max_a Q$, a spike on the single best action. The soft version moves toward the Boltzmann policy $\exp(Q)/Z$ instead, which spreads probability by value rather than collapsing to a point. (That spike is exactly the $\alpha\to 0$ limit of the Boltzmann policy, so soft-greedy contains hard-greedy as a special case.)

There is a snag: for a continuous action space you cannot represent $\exp(Q)/Z$ directly, because the normalizer $Z = \int \exp(Q(\mathbf{s},\mathbf{a}))\,d\mathbf{a}$ is an intractable integral and the shape can be anything. So SAC restricts the policy to a tractable family $\Pi$ (Gaussians) and projects the ideal target onto it, by minimizing a KL divergence:

The order of the arguments matters: this is the reverse KL, with the policy$\pi'$ in the first slot, which the paper calls an information projection. Reverse KL is mode-seeking: fitting a single Gaussian to a two-humped target by this KL tucks the Gaussian neatly under one hump rather than draping it across both and spilling probability into the low-value valley between them. The forward KL (target first) would do the draping; SAC uses the reverse one on purpose.

The figure lets you feel the mode-seeking. The amber target is the ideal soft-greedy policy over the same two-mode return from before. The teal Gaussian is the policy SAC can actually represent. Drag its mean along the axis and read the KL: the minimum sits on one mode, never in the valley between, because a Gaussian centered between the modes puts most of its own mass where the target has almost none, which drives the reverse-KL divergence up.

Two payoffs fall out of writing the improvement this way. First, the intractable normalizer $Z$ disappears from the update. Expand the KL and it contributes a $+\log Z$ term; but $Z$ depends only on the state and the Q-values, never on the policy parameters we are optimizing, so its gradient is exactly zero, like the $+C$ in an indefinite integral. SAC never computes it. Second, the paper proves this alternation actually improves the policy: soft policy evaluation converges to the true soft values, and the soft-greedy projection gives a policy with higher soft value than before, so repeating the two steps converges to the best policy in the family. That guarantee is for the exact, tabular case with finitely many actions; the deep version that follows swaps exact steps for single gradient steps and gives up the proof, keeping the recipe.

Four networks and a replay buffer

For a continuous problem you cannot run evaluation and improvement to convergence in a loop, so SAC approximates both with neural networks trained by gradient descent, interleaved with collecting data. Version one of the paper, the one we are reading, uses four learned networks: the policy (the actor) $\pi_\phi$, two soft Q-functions $Q_{\theta_1}, Q_{\theta_2}$ (the critics), and a separate soft value network $V_\psi$.

The value network deserves a word, because in principle it is redundant: equation (3) already writes $V$ in terms of $Q$ and $\pi$, so you could compute it from the other two. The paper keeps a separate approximator anyway because it stabilizes training and is convenient to fit alongside the rest. There is also exactly one target network, a slowly-tracking copy $V_{\bar\psi}$ of the value network, and its only job is to provide a nearly-fixed regression target for the Q-update so the bootstrap does not track a moving target of its own. (The version of SAC most people run today, the 2018 follow-up, drops the value network entirely and keeps target copies of the two Q's instead. This paper does not; more on that at the end.)

All of it runs in the off-policy loop below. The policy acts in the environment, each transition is stored in a replay buffer holding the last million steps, and every gradient update draws a minibatch of pasttransitions, most of them collected by older versions of the policy. That reuse is what off-policy means and where the sample efficiency comes from: one real step of experience feeds many updates. Tap a stage to see what it does.

So there are three losses to write down, one per learnable role: fit the value network, fit the two Q-networks, and push the policy toward the soft-greedy target. The next two sections do the critics and the actor in turn.

Training the critics

The value network is trained to match equation (3). It regresses $V_\psi(\mathbf{s}_t)$ onto the soft-value target, the Q-value of a freshly sampled action minus that action's log-probability:

The state comes from the replay buffer $\mathcal{D}$, but the action inside the target is drawn from the current policy, not the buffer, because the target is an expectation under $\pi_\phi$. The Q-networks are trained to satisfy the soft Bellman equation, minimizing the squared residual against a bootstrapped target:

Here the state and action are both replayed (this part is fully off-policy), and the target $\hat{Q}$ uses the target value network $V_{\bar\psi}$, updated as a slow exponential moving average $\bar\psi \leftarrow \tau\psi + (1-\tau)\bar\psi$ with $\tau = 0.005$. Think of it as a slowly-sliding ruler: if the thing you measure against jerked with every reading you would never settle, so the target inches toward the value network at half a percent per step, keeping the regression target almost fixed while it still tracks the truth over time.

Now the reason there are two Q-networks. The policy improves by chasing high Q-values, which makes it an implicit maximizer over the critic. A max over a noisy estimate is biased upward: if the critic happens to overrate some action, the policy pounces on it, and that inflated value feeds back through the bootstrap and compounds. SAC borrows the fix from TD3 (Fujimoto et al., 2018): train two critics independently and use the minimum of the two wherever a Q-value feeds the value target or the policy update. Two independent estimates rarely overrate the same action, so the smaller one is a deliberately pessimistic guess that cancels the built-in optimism.

The figure makes the bias exact. Two independent noisy estimates of the same true value put their maximum a distance $\sigma/\sqrt{\pi}$ above the truth and their minimum the same distance below. SAC takes the min: it accepts a small underestimate to kill a large, compounding overestimate. Slide the estimation noise and both offsets grow with it.

Two clarifications the paper is careful about. The minimum enters the value target (5) and the policy update, not the Q-target (8); both critics bootstrap from the single target value network. And two critics are a speedup, not a requirement: the paper reports a single Q-function solves even the 21-action Humanoid, with the second critic mainly helping on the harder tasks.

Training the actor: reparameterize, then squash

The actor is trained by the projection of equation (4). Written as a loss over states from the buffer, minimizing the KL to the soft-greedy target is:

How do you take a gradient of this through the sampling? The actor has to draw an action to evaluate the objective, and sampling is not differentiable in the obvious way. The old answer is the likelihood-ratio (REINFORCE) estimator, which only needs the score $\nabla_\phi\log\pi$ times a scalar and treats the action as a black-box draw. It works but is high-variance. SAC uses the lower-variance alternative: the reparameterization trick, the same move that makes the variational autoencoder trainable. Instead of sampling the action directly, sample fixed noise $\epsilon$ and push it through a deterministic network:

Now the randomness lives in $\epsilon$, which does not depend on $\phi$, and the action is a differentiable function of the parameters. The objective becomes an expectation over the fixed noise, and dropping the $\phi$-independent $Z$ as promised:

Read the sign, because the same letter $J$ pointed the other way in the objective (1). This $J_\pi$ is a loss the algorithm minimizes. Minimizing $\mathbb{E}[\log\pi - Q]$ does two things at once: $-\mathbb{E}[\log\pi]$ is the entropy, so pushing it down pushes entropy up, and $-\mathbb{E}[Q]$ pushes the expected soft-Q up. Descending on this loss is ascending on reward-plus-entropy, exactly the maximum-entropy objective. Its gradient backpropagates the critic's slope $\nabla_{\mathbf{a}} Q$ through the sampled action into the policy weights, through only the policy network and the critic, with no model of the environment anywhere. Reparameterizing is why the gradient can do that. REINFORCE's signal is only the scalar "this action scored $X$," while the pathwise gradient traces the path from the noise, through the action, into the critic, so it encodes which direction in action space raises $Q$. In the deterministic limit it reduces to the DDPG policy gradient, generalized here to a stochastic policy.

What is $f_\phi$ concretely? A network reads the state and outputs the mean and standard deviation of a Gaussian over an unbounded variable $\mathbf{u}$; the action is a noise sample squashed through a $\tanh$. Real actuators are bounded, say a torque in $[-1,1]$, and an unbounded Gaussian would keep proposing out-of-range actions. $\mathbf{a} = \tanh(\mathbf{u})$ maps all of the real line into $(-1,1)$. But squashing the samples bends their density, so the log-probability picks up a change-of-variables correction:

The term is subtracted and summed over the $D$ action dimensions. Each $1 - \tanh^2(u_i)$ is below one, so its log is negative, and subtracting a negative number raises the log-probability: squashing crams the density into the finite interval, which concentrates it, which raises it. The per-dimension sum comes from applying $\tanh$ elementwise, so the Jacobian is diagonal, independent of whether the underlying Gaussian is.

The figure below is that transformation. The teal Gaussian marginal sits along the bottom in $\mathbf{u}$-space; the amber density down the left is the resulting distribution over the bounded action $a = \tanh(u)$. Drag the mean and standard deviation. As either pushes the Gaussian into the flat tails of $\tanh$, the action density piles up against the bounds at $\pm 1$, which is the geometric reason the $1-a^2$ correction blows up there and why a bang-bang policy is a natural end state.

If you go build this, mind the numerics. Do not compute the correction as log(1 - tanh(u)**2); near saturation $\tanh^2 u \to 1$ and it underflows to negative infinity. Use the numerically stable rewrite instead:

which agrees with the naive form to machine precision but never divides by a vanishing quantity.

One update, concretely

Stack the three losses and you have the complete learning step. Nothing here is more than a squared error or an expectation you already met; the listing spells out the paper's update for one minibatch drawn from the buffer. Note where the $\min$ appears (in the value target and the policy loss, not the Q-target), that the actor uses rsample (the reparameterized draw) while the value target uses a plain sample, and that the target network moves last, by a slow average:

# one SAC update on a minibatch from the replay buffer D  (v1)
s, a, r, s2 = D.sample(256)            # a PAST batch, off-policy

# --- critics: soft value V and two soft Q's ---
a_pi, logp = pi.sample(s)              # fresh action + log-prob, at s
q_min = min(Q1(s, a_pi), Q2(s, a_pi))  # the min curbs overestimation
loss_V = 0.5 * (V(s) - (q_min - logp).detach())**2       # eq (5)

q_hat = r + gamma * Vbar(s2)           # bootstrap off the TARGET V  (eq 8)
loss_Q = 0.5 * (Q1(s, a) - q_hat.detach())**2 \
       + 0.5 * (Q2(s, a) - q_hat.detach())**2            # eq (7)

# --- actor: reparameterized, minimize E[log pi - Q]  (eq 12) ---
a_pi, logp = pi.rsample(s)             # a = tanh(mu + sigma * eps)
loss_pi = (logp - min(Q1(s, a_pi), Q2(s, a_pi))).mean()

for L, opt in [(loss_V, oV), (loss_Q, oQ), (loss_pi, oPi)]:
    opt.zero_grad(); L.backward(); opt.step()
Vbar = tau * V + (1 - tau) * Vbar      # slow EMA target, tau = 0.005

The policy's sample-and-score step is the tanh-squashed Gaussian, shown here on its own, correction term and all:

# tanh-squashed Gaussian policy: sample and log-prob  (eq 21)
mu, log_sigma = policy_net(s)          # a diagonal Gaussian over u
u = mu + log_sigma.exp() * randn_like(mu)     # reparameterize  (eq 11)
a = tanh(u)                            # bound each action to (-1, 1)
logp = gaussian_logp(u, mu, log_sigma)        # log mu(u|s)
logp -= (2*(log(2) - u - softplus(-2*u))).sum(-1)   # tanh correction

The rest is the outer loop of Figure 3: take one step in the environment with the current policy, store the transition, then run one update like the one above on a random batch. The full setup is deliberately plain and shared across every task the paper tests. Every network (the two Q-critics, the value network, and the policy) is two hidden layers of 256 ReLU units; the optimizer is Adam at $3\times 10^{-4}$; the discount is $\gamma=0.99$; the buffer holds $10^6$ transitions; batches are 256; the target moves at $\tau=0.005$; one gradient step follows each environment step. Only the reward scale changes from task to task. That short, fixed recipe learning a 21-action Humanoid is a large part of why the paper landed.

Stable on every seed

On the standard continuous-control benchmarks SAC matches or beats the strong baselines (DDPG, PPO, and the concurrent TD3), and the gap widens on the hardest tasks, where DDPG fails to make any progress on Ant and both Humanoids. That covers the sample-efficiency and final-performance story. The more distinctive result ties back to the stochastic actor: stability across random seeds.

The figure redraws the paper's own comparison schematically. Run five seeds of full SAC and five seeds of a deterministic variant that drops the entropy term and otherwise resembles DDPG. The stochastic runs climb together in a tight band; the deterministic ones spread wide, some learning and some stalling. Press resample to confirm it is not one lucky draw. The entropy bonus is doing the stabilizing: it keeps the policy exploring instead of collapsing early onto a brittle behavior that happens to depend on the seed.

The convergence proof comes with a scope note. The paper's theorem guarantees that soft policy iteration reaches the best policy in the family, but only for the exact, tabular case with finitely many actions. The deep algorithm swaps those exact steps for single gradient updates on neural networks and comes with no such guarantee; the theory motivates the design rather than certifying the implementation.

A last, practical caveat for anyone arriving here from a code library. The SAC in Stable-Baselines3, CleanRL, or Spinning Up is usually the 2018 follow-up (Haarnoja et al., arXiv:1812.05905), not this paper. The follow-up drops the separate value network, bootstraps from two target Q-networks instead, and replaces the hand-tuned reward scale with automatic temperature tuning: it adjusts $\alpha$ to hit a target entropy, so the one hyperparameter this paper made you tune per task becomes automatic. The maximum-entropy idea, the soft values, the reparameterized actor, and the clipped double-Q are all here in version one. The follow-up is the same skeleton with the value network removed and the temperature learned.

What the paper contributes is a single design decision followed all the way down. Add a bonus for randomness to the objective, and the value functions turn soft, the greedy step becomes a tractable KL projection, and the resulting off-policy actor-critic explores on its own and stops caring which random seed it started from. The stochastic actor that older methods treated as a liability is what makes off-policy learning behave.