DDPG: Continuous Control with Deep Reinforcement Learning

A network outputs one action per state and is trained along the gradient of a learned Q-function, so deep Q-learning works for continuous actions such as joint torques.

DDPG trains two networks from a replay buffer: a critic that estimates the return of a state and action, and an actor that maps a state to an action. The actor is updated by backpropagating the critic's output through the action into the actor's weights.

A 2015 DeepMind paper that took the replay buffer and target network from DQN, combined them with the deterministic policy gradient, and trained one algorithm with one set of hyperparameters on more than 20 simulated physics tasks, from state vectors and from pixels.

Timothy Lillicrap and Jonathan Hunt (equal contribution), with Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver and Daan Wierstra, posted the paper in September 2015 and presented it at ICLR 2016. They named the method Deep DPG, or DDPG. On 26 tasks in the MuJoCo physics simulator, scored so that a random policy gets 0 and a planner with access to the simulator's equations gets 1, the average DDPG run scores 0.72 from joint angles and 0.50 from 64 by 64 pixel images; the best of five runs beats the planner on 10 tasks. With the same buffer and batch normalization but no target networks, the average drops to 0.22.

The page covers, in order (equation numbers follow the paper):

1. why Q-learning needs a finite set of actions, and why discretizing a robot's actions fails;
2. the critic, equations (1) to (5), and why it can learn from old data;
3. the actor update of equation (6), the deterministic policy gradient, and what Silver et al. proved about it;
4. the three stabilizers: replay buffer, soft target networks, batch normalization;
5. exploration with Ornstein-Uhlenbeck noise, equation (7);
6. Algorithm 1 as code with tensor shapes, the networks and hyperparameters;
7. the results in Table 1, the limitations, and what TD3 and SAC changed.

Why DQN cannot output a torque

In reinforcement learning an agent sees a state $s_t$, takes an action $a_t$, receives a reward $r_t$ and a new state, and tries to maximize the discounted sum of future rewards. The Deep Q-Network (DQN) learned Atari games from pixels by training a network to output $Q(s, a)$, the expected discounted return after taking action $a$ in state $s$, for each of the 4 to 18 joystick actions. To act, it picks the action with the largest output. To train, it needs the same maximum at the next state, inside its target $r + \gamma \max_{a'} Q(s', a')$. Both take one forward pass and a max over at most 18 numbers.

A robot's action is a vector of real numbers, usually joint torques. The cheetah task in this paper has 6 action dimensions, so an action is a point in $[-1, 1]^6$, and $\max_a Q(s, a)$ is now a 6-dimensional continuous optimization over a neural network's input. It would have to be solved at every environment step to act, and at every next-state in every minibatch to train. Gradient ascent on the action for a few dozen iterations per query would multiply the cost of training by that number of iterations, and it still finds only a local maximum.

Discretizing the actions runs into the size of the grid. The paper's example is a 7-joint arm with each torque restricted to $\{-k, 0, k\}$: 3⁷ = 2,187 joint actions, each needing its own output unit, and three levels per joint is too coarse for fine control. With 11 levels per joint the cheetah would have $11^6$, about 1.77 million, outputs, and the paper's quadruped (hyq) has 12 action dimensions. A grid also discards the fact that a torque of 0.51 behaves almost like 0.50; each grid point's value is learned separately.

DDPG keeps the action continuous and trains a second network, the actor $\mu(s)$, to output the maximizing action directly. Evaluating $\mu$ is one forward pass, so the max inside the target becomes $Q(s', \mu(s'))$. The rest of the paper is about training $\mu$ and keeping both networks stable.

The critic: learning Q for a deterministic policy

The return from time $t$ is $R_t$, the sum of rewards from $t$ to the end of the episode at $T$, each discounted by $\gamma$ per step of delay:

The paper uses $\gamma = 0.99$, so a reward 100 steps away counts 0.99¹⁰⁰ ≈ 0.37 times as much as an immediate one. The goal is a policy that maximizes $J$, the expected return from the start state. The action-value function of a policy $\pi$ is the expected return after taking $a_t$ in $s_t$ and following $\pi$ afterward:

The subscript $E$ denotes the environment, which may be random. Splitting off the first reward gives the Bellman equation, a recursion between the value at one step and the value at the next:

For a deterministic policy, a function $\mu$ from states to actions, the inner expectation over $a_{t+1}$ has a single term:

The expectation in (3) is over the environment only. A recorded transition $(s_t, a_t, r_t, s_{t+1})$ supplies everything (3) needs, whichever policy chose $a_t$: the next action comes from evaluating $\mu$ at $s_{t+1}$, not from the data. So $Q^{\mu}$ can be learned off-policy, from transitions generated by a different behavior policy $\beta$, such as $\mu$ plus noise, or an older version of $\mu$.

With a neural network $Q(s, a \mid \theta^Q)$, the critic is trained by regression onto the right-hand side of (3):

$\rho^{\beta}$ is the distribution of states the behavior policy visits. The target $y_t$ depends on $\theta^Q$ too, and the gradient through it is ignored: the target is computed, treated as a constant, and only $Q(s_t, a_t)$ is pushed toward it. This is the semi-gradient update of Q-learning. Because the same weights produce the target, every step on $\theta^Q$ also moves the targets that the next step regresses onto, which can make the regression oscillate or diverge; the target networks below address that.

If $\mu(s)$ is replaced by $\arg\max_a Q(s, a)$, (4) and (5) are Q-learning. DDPG uses the actor's output instead, and needs a way to make that output approach the argmax.

How DDPG works: the actor update

The actor $\mu(s \mid \theta^{\mu})$ is a network that takes a state and outputs an action through a final $\tanh$, so each dimension lies in $[-1, 1]$. Its objective is the critic's value of the actions it chooses, averaged over states. Because $Q$ is a differentiable network and $a = \mu(s)$ is its input, the chain rule gives the gradient of that objective with respect to the actor's weights:

Read the second line right to left. $\nabla_{\theta^{\mu}} \mu$ is the Jacobian of the action with respect to the actor's weights: how each action dimension moves when each weight moves. $\nabla_a Q$ is a vector with one entry per action dimension: how the critic's estimated return changes when that dimension of the action is nudged, at the action the actor currently outputs. Their product tells each weight how to move so the action shifts in the direction the critic rates as better. The critic's weights are not changed by this update.

A worked example with a linear actor and a one-dimensional action. Let the state be $s = (0.5, -1.0)$ and the actor $\mu(s) = w^{\top} s + b$ with $w = (0.4, 0.2)$ and $b = 0.1$, so the action is 0.2 − 0.2 + 0.1 = 0.1. Suppose the critic, evaluated at that state and action, has slope 2.0 in the action: a slightly larger action is estimated to give a higher return. The gradient of $\mu$ with respect to $w$ is $s$ and with respect to $b$ is 1, so (6) multiplies the slope 2.0 into each:

A gradient-ascent step of size 0.01 makes $w = (0.41, 0.18)$ and $b = 0.12$, and the new action at this state is 0.205 − 0.18 + 0.12 = 0.145, up by 0.045, which is 0.01 × 2.0 × (0.25 + 1 + 1): step size, times slope, times the squared length of the features $(s, 1)$. The reward never appears in this update; it reaches the actor only through what the critic has learned.

Picture the critic as a terrain map over actions, one map per state, with height equal to estimated return. The actor stands at one point on each map and steps uphill along the local slope. Over many states and steps it becomes a function that outputs, for any state, an action near a peak of that state's map: a learned, amortized argmax that costs one forward pass. The analogy breaks because the map is a learned network, not the terrain, and the actor climbs the map's errors as readily as its true peaks.

Figure 1 runs equation (6) on a toy with one state dimension and one action dimension, with a fixed critic so that only the actor moves. Look at where the white arrows point at the start and how long they are compared with the end, and compare the two starting points.

From $\mu = 0$, the average value $J$ rises from 0.243 to 1.036 by step 60 and 1.040 by step 120, against a maximum of 1.041, and the arrows shrink to nothing as the actor reaches the ridge, because the slope of $Q$ in the action direction is zero at a peak. From $\mu = -0.85$, the nearest uphill direction at every state leads to the lower ridge, and the actor stops there with $J = 0.766$. The update follows the local slope at the current action and never evaluates actions far from it. Exploration noise in DDPG, and a critic that keeps changing, sometimes move the actor out of such a basin; nothing guarantees it.

What the deterministic policy gradient theorem covers

The update in (6) is the deterministic policy gradient (DPG) of Silver et al. (2014), three of whose authors (Heess, Silver, Wierstra) are also on this paper. A reasonable worry is that changing the policy also changes which states the agent visits, and (6) ignores that: it averages over a fixed set of states. Their Theorem 1 shows that for the true action-value function and the states the policy itself visits, ignoring it is exact:

In this theorem $\rho^{\mu}$ is the discounted distribution of states visited by $\mu$ itself. DDPG differs from that statement in three ways. It averages over replay states drawn from $\rho^{\beta}$, the behavior policy's distribution, which Silver et al. introduce as an approximation that drops a term; hence the $\approx$ in (6). It uses a learned critic in place of $Q^{\mu}$, and the DPG paper guarantees the gradient is exact only for a "compatible" critic of a specific linear form, which a neural network is not. And, as the paper's footnote says, it ignores the discount in the state distribution, as most policy gradient implementations do. So (6) is the gradient of a surrogate, the critic's average value over replay states, and it improves the true return to the extent the critic is accurate near the actor's actions.

The stochastic policy gradient that A3C, TRPO and PPO build on has the form $\mathbb{E}[\nabla_{\theta} \log \pi_{\theta}(a \mid s)\, Q(s, a)]$: it samples actions, scores them by their returns, and makes high-scoring ones more likely. It needs no derivative of $Q$, but it learns about the direction of improvement only through sampled actions, and the estimate gets noisier as the action dimension grows. The deterministic version replaces that sampling with the critic's derivative; the DPG paper's Theorem 2 shows it is the limit of the stochastic gradient as the policy's variance goes to zero. The expectation in (6) is over states only, while the stochastic gradient also averages over actions; the DPG authors write that the stochastic version "may require more samples, especially if the action space has many dimensions."

DPG with neural networks had been tried before. NFQCA (Hafner and Riedmiller, 2011) used the same update rules with batch training to keep it stable; the paper notes that a minibatch version of NFQCA is the original DPG, which is what it compares against. That baseline is unstable on hard tasks, and the next three sections are the changes that fix it.

Replay buffer and soft target networks

Two of DDPG's three stabilizers come from DQN. The first is the replay buffer: every transition $(s_t, a_t, r_t, s_{t+1})$ goes into a first-in, first-out store of the last $10^6$ transitions, and each update trains both networks on a minibatch of 64 sampled uniformly from it. Consecutive steps of a physical system are nearly identical; training on them in order would fit the networks to whatever the robot is doing this second. Sampling from a million stored steps makes the minibatch closer to the independent samples that SGD assumes. A buffer is possible only because the critic learns off-policy, as (3) showed.

The second is the target network. In (5) the target $y_t$ is computed with the same critic being trained, so every update that raises $Q(s_t, a_t)$ also tends to raise $Q(s_{t+1}, \cdot)$ through shared weights, which raises the next target, and estimates can chase themselves upward or oscillate. DDPG computes targets with separate copies of both networks, $Q'$ and $\mu'$:

and moves the copies a fraction $\tau$ of the way toward the online networks after every update:

DQN's Nature version froze its target network and replaced it with a full copy every 10,000 updates. The soft update makes the target weights an exponential moving average of the online weights instead. If the online weights jump to new values and stay there, the target closes a fraction $\tau$ of the remaining gap per update, so the gap left after $n$ updates is $(1-\tau)^n$ of the original. With $\tau = 0.001$ half the gap is closed after

63% after 1,000, and 99% after about 4,600. The paper reports that both target copies were needed: "having both a target μ' and Q' was required to have stable targets $y_i$ in order to consistently train the critic without divergence."

Figure 2 applies both rules to a synthetic critic output. Watch how far the amber target trails the teal online value at the paper's $\tau$, and what happens at the two ends of the slider.

At $\tau = 0.001$ the target's update-to-update jitter is about 0.3% of the online value's, and it trails a rising trend by about 1,000 updates ($(1-\tau)/\tau$). The paper names this lag as a cost, since value information propagates through the bootstrap only as fast as the target moves. At $\tau = 10^{-4}$ the target barely leaves its starting value within the plotted 6,000 updates. The hard copy holds every target fixed for 1,000 updates, an average delay of 500, and then changes all of them at once.

In Table 1 the variant labeled cntrl is the original DPG with the replay buffer and batch normalization but no target networks. Its average score over the 26 tasks is 0.22, against 0.72 with target networks; it is below DDPG on 25 of the 26 tasks and below the random policy on 7. The paper's Figure 2 caption puts it as "Target networks are crucial."

Batch normalization for mixed units

The third stabilizer addresses the inputs. A low-dimensional observation mixes quantities in different units: the cart's position in meters, the pole's angle in radians, angular velocities that can be ten times larger. Across tasks the ranges differ again. A network whose first layer sees one input in the hundreds and another near 0.01 learns slowly on the small one, and a learning rate tuned for one task's scales is wrong for another's, which defeats one hyperparameter setting for all tasks.

Instead of scaling each task's features by hand, DDPG applies batch normalization to the state input and to every layer of the actor, and to the critic's layers before the action enters. Each feature is normalized over the minibatch to zero mean and unit variance and then scaled and shifted by learned parameters; running averages of the mean and variance are used when the network acts in the environment. With it, the paper reports learning across tasks with different units without checking their ranges. The re-tuned DDPG baseline in the TD3 paper (2018) dropped batch normalization.

Exploration with Ornstein-Uhlenbeck noise

A deterministic actor takes the same action every time it sees a state, so on its own it never tries anything else and the critic never learns what other actions are worth. Because DDPG is off-policy, exploration can be added to the behavior policy without changing the learning rule. The paper adds noise to the actor's output:

The prime here marks the exploration policy, not the target actor of the previous section; the paper uses the same symbol for both. The noise $\mathcal{N}$ is an Ornstein-Uhlenbeck (OU) process, a random walk pulled back toward zero, with rate $\theta = 0.15$ (unrelated to the network weights $\theta^{\mu}$) and scale $\sigma = 0.2$. Stepped once per action (the paper does not state its time step), each action dimension gets

so each step keeps 85% of the previous noise and adds a fresh Gaussian of standard deviation 0.2. In steady state the noise has standard deviation

on actions in $[-1, 1]$, and its correlation with itself halves every 4.3 steps. A push in one direction therefore tends to last several steps. On a body with inertia, consecutive pushes in the same direction add up to real movement, while independent pushes of the same size mostly cancel before the body responds.

At $\theta = 0.15$ the OU-driven cart ends about three times farther from where it started than the cart driven by independent noise of the same size, so it visits more of the state space per episode. The unstated time step changes the noise by orders of magnitude: public implementations with the same $\theta$ and $\sigma$ step the process with $dt = 1$ (rllab) or $dt = 0.01$ (OpenAI baselines), which gives correlation times of 6.7 and 667 agent steps. For Torcs, the driving game, the paper changed the noise because of the different time scale, without giving the values.

The full DDPG algorithm

Algorithm 1 in the paper interleaves acting and learning: one environment step, then one minibatch update of both networks, then a soft update of both targets. With the minibatch of $N$ transitions, the critic minimizes

and the actor steps along the sampled version of (6):

The paper reuses $N$ for three things: the action dimension ($a \in \mathbb{R}^N$), the minibatch size, and, as $\mathcal{N}$, the noise process. Below, the action dimension is 6 and the minibatch is 64. In an autodiff framework the actor gradient is written as a loss: the negative mean of $Q(s_i, \mu(s_i))$, backpropagated through the critic into the actor, with only the actor's optimizer stepping.

# One DDPG update on cheetah: state 17, action 6, minibatch N = 64.
s, a, r, s2 = buffer.sample(64)          # 64x17, 64x6, 64, 64x17

# critic: regress Q(s, a) onto a target built from the TARGET networks
with torch.no_grad():                    # y is a constant for this step
    a2 = actor_targ(s2)                  # 64x6, in [-1, 1] from tanh
    y = r + 0.99 * critic_targ(s2, a2)   # 64
q = critic(s, a)                         # 64, a = the action actually taken
loss_q = ((y - q) ** 2).mean() + weight_decay_1e-2(critic)
critic_opt.zero_grad(); loss_q.backward(); critic_opt.step()   # lr 1e-3

# actor: ascend Q(s, actor(s)); the gradient passes THROUGH the critic
loss_mu = -critic(s, actor(s)).mean()    # actor's own action, not a
actor_opt.zero_grad(); loss_mu.backward(); actor_opt.step()    # lr 1e-4
# loss_mu.backward() also fills critic grads; only actor_opt steps

# targets trail the online networks
for p_t, p in zip(all_target_params, all_online_params):
    p_t.mul_(1 - 0.001).add_(0.001 * p)  # tau = 0.001

Three different action sources appear in that update. The critic is evaluated at $a_i$, the stored action that was taken, with noise, possibly many updates ago. The target uses $\mu'(s_{i+1})$, the target actor's choice at the next state. The actor loss uses $\mu(s_i)$, the current actor's choice at the stored state, which is never executed. The acting loop around the update:

# Acting in the environment (one episode)
noise = OUProcess(theta=0.15, sigma=0.2, size=6); noise.reset()
s = env.reset()
for t in range(T):
    actor.eval()                         # batch norm uses running stats
    a = actor(s) + noise.sample()        # equation (7); usually clipped
    s2, r, done = env.step(a)            # to the valid action range
    buffer.add(s, a, r, s2)              # buffer keeps the last 1e6
    actor.train()
    ddpg_update()                        # one minibatch update per step
    s = s2

Each transition is sampled into a minibatch about 64 times on average over its stay in a full buffer (a million updates, 64 draws each from a million entries), so every environment step contributes to about 64 updates. An on-policy method such as A3C uses a transition in one update, and PPO in a few epochs over one batch, before discarding it.

Networks and hyperparameters

Every MuJoCo task used the same settings, and Torcs changed only the exploration noise. Both networks have two hidden layers of 400 and 300 ReLU units for low-dimensional input. The actor ends in a tanh layer with one unit per action dimension. The critic takes the state at its first layer and concatenates the action to the 400 features before the second layer. For the cheetah (17 observation dimensions, 6 actions), that is

• actor: 17 × 400 + 400 = 7,200, then 400 × 300 + 300 = 120,300, then 300 × 6 + 6 = 1,806, for 129,306 weights;
• critic: 7,200, then (400 + 6) × 300 + 300 = 122,100, then 301 for the single output, for 129,601 weights,

matching the paper's "≈ 130,000 parameters", not counting batch normalization's learned scales and shifts. The final layers start uniform in $[-3 \times 10^{-3},\, 3 \times 10^{-3}]$ so that initial actions and Q values are near zero; other layers start uniform in $[-1/\sqrt{f},\, 1/\sqrt{f}]$ with $f$ the fan-in. Optimization uses Adam with learning rate $10^{-4}$ for the actor and $10^{-3}$ for the critic, L2 weight decay of $10^{-2}$ on the critic only, $\gamma = 0.99$, $\tau = 0.001$, minibatch 64, buffer $10^6$.

From pixels, each observation is three consecutive renders of 64 by 64 RGB, stacked into 9 channels and scaled to $[0, 1]$. The agent's action is repeated for the three simulator steps, so differences between the frames carry velocity. Three convolutional layers of 32 filters each with no pooling feed two fully connected layers of 200 units, about 430,000 parameters, with actions entering at the fully connected layers. The minibatch is 16 and the final-layer initialization range is ten times smaller.

Results on 26 MuJoCo tasks and Torcs

The tasks are MuJoCo simulations: cart-pole swing-up and balance with up to three linked poles, 1- to 7-joint reaching arms, grippers that pick up and move a block, an arm with a hockey stick that hits a puck (canada), a one-legged hopper, a 2D cheetah and walker, and the 12-joint hyq quadruped. Actions are joint torques in every domain except cheetah. Rewards are dense: distance to a goal, progress forward, and a small action cost in every task.

Each score in Table 1 is normalized per task between two reference policies:

$R_{\text{random}}$ is the mean return of actions drawn uniformly from the valid range. $R_{\text{iLQG}}$ is a model-predictive controller that, at every step, runs one iteration of iterative LQG trajectory optimization over a 250 to 600 ms horizon from the true simulator state, using the simulator's dynamics and their derivatives. A score above 1 means DDPG beat a planner that knows the physics; a negative score means worse than random. Every score is after at most 2.5 million steps, over 5 runs.

Averaged over the 26 tasks, the mean run scores 0.72 from state and 0.50 from pixels. The best run scores 1.08 and 0.93. The best of five beats the planner on 10 tasks from state and 8 from pixels, and on the 7-joint block-lifting task (blockworld3da) the best pixel run scores 2.225. Two average scores from state exceed 1: blockworld1 (1.156) and hardCheetah (1.311), the cheetah variant with its stabilizing joint stiffness removed. The gap between best and average is large on several tasks, 0.303 against 1.735 on canada, so outcomes vary a lot between runs with the same settings.

On 5 tasks the average pixel run beats the average state run (blockworld3da, cart, movingGripper, reacherSingle, walker2d). The paper notes that on some simpler tasks learning from pixels is as fast as from state, and suggests that the action repeats simplify the problem or that convolutional features separate the states well; it does not test either.

On Torcs, a racing simulator with acceleration, braking and steering as actions, the reward is speed along the track with a penalty of 1 per collision. Some runs learned to complete laps, best returns 1,840 from state and 1,876 from pixels, while others failed, which pulls the averages to −393 and −402.

The paper also compared the critic's estimates with the returns actually obtained on test episodes. On pendulum and cart-pole they line up without a systematic bias; on harder tasks the estimates are less accurate, and the policies still perform well. Nearly all tasks were solved within 2.5 million steps, which the paper calls 20 times fewer than DQN needs on Atari. That ratio compares agent steps with DQN's 50 million emulator frames; DQN chose an action every 4th frame, so in decisions the ratio is 5.

Limitations and what came next

The paper names one limitation: like most model-free methods, DDPG needs many training episodes, up to 2.5 million steps of simulated experience per task. It also notes that nonlinear function approximation removes any convergence guarantee; the stability is empirical.

In 2018 Fujimoto, van Hoof and Meger measured DDPG's value estimates on the Gym versions of Hopper and Walker2d and found them above the true returns. The actor is trained to find actions where the critic is high, so it moves toward actions where the critic's error happens to be positive, and those overestimates enter the next targets. They proposed TD3 (Twin Delayed DDPG). TD3 takes the minimum of two critics in the target, updates the actor and targets every second critic update, and adds clipped noise to the target action so that the target does not depend on a narrow peak of $Q$. Their re-tuned DDPG baseline also dropped batch normalization, weight decay and OU noise.

Soft Actor-Critic (2018) kept the off-policy actor-critic structure and the soft target update, and replaced the deterministic actor with a stochastic one trained to maximize return plus entropy. Its actor update still backpropagates the critic through the action, by sampling the action as a differentiable function of noise, which this paper's related-work section anticipated: "The techniques we described here for scaling DPG are also applicable to stochastic policies by using the reparametrization trick".

Implementation notes

DeepMind did not release code for the paper, so details the paper leaves out were filled in differently by each reimplementation. The ones that change behavior:

• The OU time step, discussed above: $dt = 1$ gives noise correlated over a few steps, $dt = 0.01$ over hundreds. rllab's default $\sigma$ is also 0.3, not the paper's 0.2.
• Terminal states. Algorithm 1 bootstraps from $s_{i+1}$ on every transition; implementations multiply the bootstrap term by $1 - \text{done}$ for transitions that end an episode by failure.
• Batch normalization has two modes. The networks use minibatch statistics during updates and running averages when acting; forgetting to switch gives a different action for the same state depending on the batch.
• The noisy action can leave $[-1, 1]$; it is clipped to the valid range before execution (rllab's OU strategy returns the clipped action, which then gets stored).
• Weight decay of $10^{-2}$ applies to the critic only, and the actor's loss must not update the critic: in PyTorch, the backward pass of the actor loss also writes gradients into the critic's parameters, which the critic's next zero_grad() clears.