DQN: Playing Atari with Deep Reinforcement Learning

Q-learning, a convolutional network and a replay memory learn seven Atari games from raw pixels and the score.

Mnih and colleagues at DeepMind trained one network architecture, with one set of hyperparameters, separately on each of seven Atari 2600 games. It beat the best earlier reinforcement-learning methods on all seven games when all were scored by their averages, every previous approach on six, and an expert human on three.

The agent sees a stack of 84×84 grayscale screens and the change in score. It has to work out which button presses, sometimes thousands of steps back, earned each point, from data its own changing behaviour generates.

By 2013 supervised deep learning ran on a fixed recipe: collect labeled examples, attach a fixed target to each, and reduce the error with stochastic gradient descent. AlexNet won ImageNet with it in 2012. The recipe assumes three things about the data: every input comes with its correct output, examples are drawn independently, and the distribution they come from does not change during training.

Reinforcement learning breaks all three, and the paper's introduction lists them. The feedback is a scalar reward that is "frequently sparse, noisy and delayed", with delays that "can be thousands of timesteps long". Consecutive states are highly correlated, because one frame of a game differs from the next by a few pixels. And the data distribution changes as the agent learns new behaviours, because what the agent does decides what it sees next.

This paper trains a convolutional network that reads the raw Atari screen and outputs one number per joystick action, the estimated value of pressing it. The learning rule is a variant of Q-learning, from Chris Watkins's 1989 thesis. Instead of learning from the transition that just happened, the network learns from random samples of its past transitions, a technique called experience replay that Long-Ji Lin introduced in 1992. With those pieces and no per-game tuning beyond one integer, it beat the best published learning methods on all seven games when all were scored as averages, and an expert human on Breakout, Enduro and Pong. Google announced it was acquiring DeepMind 38 days after the arXiv posting; how much this paper had to do with that is not documented.

The sections follow the order the method depends on: the reinforcement-learning loop; the action-value function $Q^*$ and the Bellman equation that defines it; the four-frame input that stands in for a state; the Q-learning update and its moving targets; why a neural network trained with Q-learning was expected to diverge; experience replay; the network and its training settings; and the results.

Screen in, action out, score back

At each time-step the agent sees an image $x_t$, the raw Atari screen: 210×160 pixels drawn from a 128-colour palette. It picks an action $a_t$ from the game's legal joystick moves, between 4 and 18 of them depending on the game. The emulator advances and returns a reward $r_t$, defined as the change in the game score on that step. The paper lists everything the network learns from: "the video input, the reward and terminal signals, and the set of possible actions". It gets no access to the emulator's memory and no hand-built features such as the ball's coordinates.

A policy is the rule that maps what the agent has seen to the button it presses. DQN uses an $\varepsilon$-greedy policy: with probability $\varepsilon$ it presses a random button, and otherwise the button its network currently scores highest. The random presses are exploration. An agent that always presses its current favourite never collects data about the other buttons, so it cannot find out that one of them is better. During training $\varepsilon$ falls linearly from 1 to 0.1 over the first million frames and then stays at 0.1, so halfway through that schedule, at frame 500,000, 55% of presses are random. The 10% floor keeps the data varied for the whole run, and for a reason covered in the Q-learning section it does not bias what the network learns. Evaluation runs use $\varepsilon = 0.05$.

The figure runs this loop on a toy Breakout. Drag $\varepsilon$ to both ends and hold it there. At $\varepsilon = 1$ you are watching the first minutes of training: random presses and quick losses. At $\varepsilon = 0$ the paddle follows the highest bar every step. The bars are action values, the quantity the rest of the paper is about learning.

The reward for a good move can arrive long after the move. In Breakout the points for a brick arrive while the ball is still travelling from a paddle hit made a second earlier; in Seaquest, surfacing for oxygen pays off half a minute later. In supervised learning the label sits next to its input. Here the learner first has to solve credit assignment: deciding which of the recent actions caused the point that just arrived, so that those actions become more likely.

Action values and the Bellman equation

The classical approach to delayed credit starts by defining what an action is worth. Future rewards are summed with a discount factor $\gamma$ between 0 and 1 applied once per step:

$R_t$ is the return, the discounted total reward from step $t$ until the episode ends at step $T$. A reward $k$ steps in the future counts $\gamma^k$ as much as one received now. With $\gamma = 0.99$, a point 100 steps away counts $0.99^{100} \approx 0.37$. Atari episodes end, so the sum would be finite without discounting; discounting bounds it regardless of episode length (with rewards clipped to at most 1, as DQN's are, no return can exceed $1/(1-\gamma)$, which is 100 at $\gamma = 0.99$) and sets an effective planning horizon of about $1/(1-\gamma)$ steps. This paper never states the $\gamma$ it used. The 0.99 in most implementations comes from the 2015 Nature follow-up.

For a state $s$ and action $a$, the optimal action-value function is

which reads: take action $a$ in state $s$, then follow the best possible policy $\pi$ forever after, and average the return. The max over all policies means $Q^*$ does not depend on how anyone plays; it is a property of the game. With $Q^*$ in hand, optimal play needs no search: in every state, press the action with the largest $Q^*(s,a)$.

Why score state-action pairs instead of states? A state value $V(s)$ says how good a situation is, but choosing an action from it requires knowing which state each action leads to, which requires a model of the game's dynamics. DQN never learns such a model. $Q$ folds the one-step lookahead into the function, so choosing an action is a comparison of numbers. A network that outputs $Q$ for every action at once, as DQN's does, picks the greedy action with one forward pass.

$Q^*$ satisfies a consistency condition, the Bellman optimality equation:

The value of an action is the reward it collects now plus $\gamma$ times the value of the best action available in the state it leads to. The expectation over $s' \sim \mathcal{E}$ averages over the emulator's randomness about where you land. A route planner works the same way: the fastest time from your corner is the time of the first leg plus the fastest time from wherever that leg ends. As a worked case, if a move earns $r = 1$, always lands in the same $s'$, and the three actions there have $Q^*$ values 0.5, 0.8 and 0.3, then with $\gamma = 0.99$ that action is worth $1 + 0.99 \times 0.8 = 1.792$. Only $Q^*$ satisfies (1) at every state and action, so an estimate that violates it somewhere is wrong there.

Treating (1) as an assignment rather than an identity, and applying it to every state-action pair in turn, gives value iteration:

and $Q_i \to Q^*$ as $i \to \infty$. Each sweep multiplies the largest error anywhere on the board by at most $\gamma$, so at $\gamma = 0.9$ the worst-case error falls tenfold every 22 sweeps ($0.9^{22} \approx 0.098$). Value also spreads outward one step per sweep from the states where reward occurs. In the figure, press Play and watch the value move back from the +1. Then drag $\gamma$ to 0: only the cells next to the reward ever get a value, because the future is weighted by zero. Near $\gamma = 1$ the board fills almost evenly, because a reward twenty steps away counts nearly as much as one next door.

Value iteration does not solve Atari, for two reasons. It needs a table with one entry per state-action pair and visits every entry on every sweep, and each backup takes an expectation over the emulator's transition probabilities, which nobody has written down. The gridworld above is small and its transitions are known. Atari's states are every screen history the game can produce, far too many to store. The paper calls the tabular approach "totally impractical" and replaces the table with a function approximator, $Q(s,a;\theta) \approx Q^*(s,a)$, here a neural network with weights $\theta$ that the paper calls a Q-network. Two screens that differ by a few pixels of scenery then share most of their computation and get similar values, which is how 10 million frames of play can teach anything about states never seen. The same sharing made people expect the method to diverge, as the section on divergence shows.

One frame is not a state: the four-frame input

The Bellman equation assumes a Markov state: one that contains everything needed to predict what happens next, so that the earlier history adds nothing. A single Atari frame is not Markov. It shows where the ball is but not which way it is moving, so "ball rising" and "ball falling" produce the same image and call for different actions. The paper says such screens are perceptually aliased: it is "impossible to fully understand the current situation from only the current screen".

Formally, the paper defines the state as the whole history of screens and actions so far, $s_t = x_1, a_1, x_2, \ldots, a_{t-1}, x_t$. A history is always Markov, since nothing about the past is missing from it, and because every episode ends after finitely many steps, it defines a large but finite Markov decision process to which the theory of the previous section applies. The implementation truncates the history: a function $\phi$ preprocesses the last 4 frames and stacks them into one 84×84×4 input, and Algorithm 1 treats $\phi(s_t)$ as if it were a Markov state. Two frames give each object's velocity and three give a rough acceleration. The stack also helps with a hardware quirk: many Atari games draw some sprites only on alternate frames, because the console could draw only a few objects per scanline, so any single frame can be missing an object that is in the game.

The truncation sets a limit that is fixed before training. The agent observes every 4th emulator frame (the frame skipping described later), so four stacked frames span about 16 emulator frames, roughly a quarter of a second at 60 frames per second. Anything that develops over seconds, such as an oxygen meter that has been falling for twenty seconds, can only enter the decision through what is visible on the current screens. The results section shows this limit in the games DQN loses.

How Q-learning works: regression on its own guesses

The goal is a network $Q(s,a;\theta)$ that satisfies (1). Its right-hand side cannot be computed directly, because the expectation needs the emulator's transition probabilities. DQN is model-free: it never estimates those probabilities, and learns values directly from sampled experience. Every step of play yields one transition $(s, a, r, s')$, a single observed instance of "from $s$, action $a$ earned $r$ and led to $s'$".

Q-learning turns (1) into a regression problem. It builds a target from the observed reward and the network's own estimate at the next state, and pulls the prediction toward it:

This is a squared error, as in any regression, with two differences. The distribution $\rho$, which the paper calls the behaviour distribution, is whatever mix of states and actions the agent encounters while playing. And the label $y_i$ is computed from the network: one observed reward $r$ plus $\gamma$ times the network's estimate of the best value at the next state. Updating an estimate toward a target built from another estimate is called bootstrapping. The paper states the consequence: in supervised learning the targets are fixed before learning begins, while here "the targets depend on the network weights". Each update to $\theta$ changes the labels too. The subscript $i$ counts rounds of this regression, and $\theta_{i-1}$ is the previous round's weights, held fixed while $\theta_i$ is fitted; in Algorithm 1, which updates after every step, both are the same live weights.

Differentiating (2) with the target held constant gives the update direction:

The bracketed factor is the temporal-difference (TD) error: how far the prediction is from the one-step target. The target also depends on $\theta$, through the max at $s'$, and (3) deliberately does not differentiate through it. That makes (3) a semi-gradient. Differentiating through the target as well gives Baird's residual-gradient method, which has better convergence properties but in practice learns more slowly, and the semi-gradient became standard. Because the target is held constant, (3) is not the gradient of any fixed objective: it aims the prediction at where the target is now, although the step itself moves the target. The sampled reward and next state enter the TD error linearly, so replacing the expectations in (3) by single sampled transitions still gives an unbiased estimate of (3), which is what makes plain SGD on sampled transitions valid.

A worked update, with illustrative numbers. In Breakout the network predicts $Q(s, \text{left}) = 0.42$. The agent presses left, a brick breaks, and the clipped reward is $r = 1$. At the next state the network's largest output is 0.55. With $\gamma = 0.99$ the target is

and the gradient step raises $Q(s,\text{left})$ in proportion to $\delta = 1.1245$, the TD error. No gradient flows into the 0.55: it is treated as a number. If the brick had ended the episode, the target would be $y = r = 1$, since a terminal state has no future. In DQN this is done on 32 transitions at once: 32 stacked inputs of shape 84×84×4 produce a 32×$K$ matrix of Q-values, one row per transition and one column per action; the prediction for each row is the entry in the column of the action taken, the target for each row uses the max over the next state's row, and the loss is the mean of the 32 squared differences.

If the weights are updated after every step with single samples, (3) reduces to the tabular rule Watkins wrote down in 1989, Q-learning:

With one parameter per table entry, $\nabla_\theta Q(s,a)$ is 1 for the entry being updated and 0 for every other, so the SGD step with learning rate $\alpha$ is exactly this rule. For lookup tables, Watkins and Dayan proved in 1992 that it converges to $Q^*$ with probability 1, provided every state-action pair keeps being tried and the step sizes decay at the usual stochastic-approximation rate. Their paper restricts the result to lookup tables and notes that Q-learning "may not converge correctly for other representations". It also notes that the observed transitions "need not form a continuous sequence": the next transition used for learning does not have to start where the previous one ended.

The figure runs the tabular rule on a six-cell corridor with +1 at the right end, one update at a time. Scrub slowly around the first time the agent reaches the goal. That update is the first with nonzero information in its target, because the terminal step has $y = r = 1$. Each later visit to the cell before it copies a discounted share of that value one cell further left, the same backward spread as in Figure 2, built here from sampled transitions instead of full sweeps.

In a network, the target moves in a way it does not in the table. Follow one fixed transition through training. Early on the weights are near random, so the prediction $Q(s,a)$ is noise, and the target $r + \gamma \max_{a'} Q(s',a')$ is built from the equally random estimate at $s'$, which every update is also changing. The target jumps from update to update although the transition never changes. Later, once the estimate at $s'$ has been tied to real rewards many times, the target moves in small steps and the prediction closes the gap. Near convergence the two agree, which is equation (1) holding at that transition. The early jumps have two sources, and the two stabilizers attack them separately: sampling scattered old transitions from a replay memory keeps consecutive, nearly identical frames from pushing the weights the same way many times in a row, and the 2015 paper's frozen target network holds the target's weights still for thousands of updates.

The target's $\max_{a'} Q(s',a')$ scores the best action at $s'$, whichever action the agent took there. That makes Q-learning off-policy: it learns the value of the greedy policy while the data comes from a different, more exploratory one (the $\varepsilon$-greedy behaviour). Picture a food critic travelling with a cautious group: the group decides which restaurants get visited, but the review always records the best dish on the menu, not what the group ordered. Its on-policy sibling, Sarsa, puts the action taken at $s'$ into the target instead, and so learns the value of the exploratory behaviour itself. This is why the 10% of random presses during training do not bias the learned values toward random play, and it is the property that lets DQN learn from a replay memory.

Why a neural network plus Q-learning was expected to diverge

Neural networks and temporal-difference learning had one major success before 2013: Tesauro's TD-Gammon, a backgammon program built on a network with one hidden layer that learned entirely by self-play and, in the paper's words, "achieved a super-human level of play". It differed from DQN in two respects that matter here: it learned a state value $V(s)$, not $Q(s,a)$, and it learned on-policy from its own games. The DQN paper calls it "similar to Q-learning" and its own footnote records both differences.

Attempts to repeat the result on chess, Go and checkers "were less successful", and the paper reports a "widespread belief" that TD-Gammon was a special case, perhaps because the dice force exploration of the state space and make the value function smooth. Pollack and Blair added to the doubt by training a backgammon player with plain hill-climbing, no temporal-difference learning at all, to a competent level, and arguing that backgammon's own dynamics explained much of TD-Gammon's success.

Theory gave concrete reasons to expect failure. Tsitsiklis and Van Roy showed that TD learning with a nonlinear function approximator can diverge, and Baird showed that off-policy updates can diverge even with a linear one. The smallest example of the mechanism takes two states. State A has estimated value $w$ and state B has $2w$: one shared weight, with features 1 and 2. The transition from A to B pays reward 0. Start at $w = 10$ with step size $\alpha = 0.1$ and $\gamma = 0.99$. The A-to-B transition looks like a move from a state worth 10 to a state worth 20, so the TD error is $0.99 \times 20 - 10 = 9.8$ and $w$ rises to 10.98. But B shares the weight, so B's value rises to 21.96 at the same time, and the next update on the same transition sees a larger error, 10.76, and raises $w$ to 12.06. Each such update multiplies $w$ by $1 + \alpha(2\gamma - 1) = 1.098$, and any $\gamma > 0.5$ makes that factor exceed 1.

The error only grows if B's own transition is rarely trained. If B leads to the end of the episode with reward 0, an update there pulls $2w$ toward 0, multiplying $w$ by $1 - 4\alpha = 0.6$. An agent that plays A, B, end trains the two transitions equally often, and a pair of updates multiplies $w$ by $1.098 \times 0.6 \approx 0.66$, so it shrinks to the true value 0. If the data instead contains nine A-to-B transitions for every one from B, ten updates multiply $w$ by $1.098^9 \times 0.6 \approx 1.39$, and it diverges. The figure runs exactly these updates, and its three switches remove one ingredient each: a table with a separate weight per state, a Monte Carlo target (the observed return, 0) instead of a bootstrapped one, or on-policy update frequencies. Removing any one of them makes the error decay. With all three on, drag $\gamma$ down: this update schedule diverges for $\gamma$ above about 0.79.

Sutton and Barto's textbook later named this combination of function approximation, bootstrapping and off-policy training the deadly triad; the term is from 2018 and does not appear in this paper. DQN has all three, in a network with about 680,000 parameters. The paper adds a fourth problem that the triad does not cover: the agent generates its own training data. In the paper's example, if the highest-valued action is to move left, the training samples will be dominated by states on the left-hand side of the screen; if the preferred action then switches to right, the training distribution switches with it. The paper, citing Tsitsiklis and Van Roy, says such feedback loops can leave the parameters "stuck in a poor local minimum" or make them "diverge catastrophically".

Two lines of prior work tried to avoid these failures. Gradient-TD methods perform true gradient descent on a well-defined objective and had been proven convergent, but only for evaluating a fixed policy with a nonlinear approximator or for control with a linear one; the paper notes they "have not yet been extended to nonlinear control". Riedmiller's neural fitted Q-iteration (NFQ) minimizes the same losses as (2), but as a batch method that refits on the whole dataset each iteration, so the cost of an iteration grows with the data; its visual variant first compressed screens with a deep autoencoder and ran NFQ on the compressed representation. DQN's claim to be first is specific: end-to-end from raw pixels, with SGD updates of constant cost. Riedmiller is one of this paper's authors.

How experience replay works

At every step, DQN stores the transition $e_t = (s_t, a_t, r_t, s_{t+1})$ in a replay memory $\mathcal{D}$ that keeps only the most recent entries (the paper's holds "one million most recent frames"). To learn, it does not use the transition that just happened. It draws a minibatch of 32 transitions uniformly at random from the memory and takes one gradient step on those. The newest transition enters the memory like every other and is sampled only when a draw lands on it.

The paper gives three reasons. The first is data efficiency: each transition can be used in many updates. With one new transition stored and one minibatch of 32 drawn per step, a memory of $N$ transitions keeps each one for $N$ steps and samples it with probability $32/N$ per step, so each transition is used about 32 times on average, whatever $N$ is. The second is correlation: a minibatch of 32 consecutive frames holds nearly the same example 32 times, while 32 random draws are close to independent, and averaging independent gradients "reduces the variance of the updates". The third concerns the feedback loop: the training distribution becomes an average over the last million frames of behaviour instead of whatever the current policy is doing. An agent that has just started preferring left still adds left-side frames to the memory, but it trains on a mixture of every phase it has recently been through, so one change of preference cannot flip the whole training set.

In Figure 6 the band is the agent's past on a log scale and the two trays are minibatches. The buffer-size slider runs down to its degenerate edge: at $N = 32$ the memory holds only the last 32 transitions, and replay becomes the online learning it was meant to replace.

Figure 6 shows what goes into a minibatch; Figure 7 shows what that does to a model with shared weights. Two learners fit the same curve from one stream of samples whose position drifts slowly across the input range, the way an agent spends long stretches in one part of a level. The model has five global features, so every weight affects the whole curve, as a network's weights do. The online learner takes each step on the 32 newest samples. The replay learner takes each step on 32 random samples from its memory. Watch the violet curve: it matches the target near the stream position and misses it elsewhere, and its error over the whole range stays several times the replay learner's. Then shrink the buffer to 32.

A transition drawn from the memory was generated by an older version of the agent, possibly a much worse one. An on-policy method such as Sarsa learns the value of the policy that produced its data; trained on old transitions, it would keep estimating the value of policies the agent no longer follows. The paper states the requirement directly: learning by experience replay makes it "necessary to learn off-policy", "which motivates the choice of Q-learning". The Q-learning target does not depend on who chose the action. A stored $(s,a,r,s')$ is a valid sample of what happens when $a$ is taken in $s$, whether the greedy policy or the $\varepsilon$ coin chose it, and the max in the target computes the same quantity for every sample: the best value available from $s'$.

DeepMind did not invent replay. Long-Ji Lin introduced it in 1992 for robots learning with neural networks, and the paper cites his 1993 technical report, noting that Lin's version started from a low-dimensional state rather than raw pixels. The paper also names two limitations of its own memory. It always overwrites the oldest transitions, because its size is finite, and uniform sampling gives a rare, informative transition the same weight as the ten-thousandth frame in which nothing happens. It suggests a sampling scheme that favours transitions "from which we can learn the most", like prioritized sweeping; Schaul and colleagues at DeepMind published prioritized experience replay two years later.

The frozen target network associated with DQN, a second copy of the weights that computes targets and is refreshed only every 10,000 updates, is not in this paper. It is the 2015 Nature paper's addition. In Algorithm 1 the targets come from the same live $\theta$ that is being updated; the $\theta_{i-1}$ in (2) belongs to the round-by-round formulation and becomes the live weights in the per-step algorithm. Experience replay is therefore the only stabilizer in the 2013 method.

The DQN network and training recipe

Preprocessing converts each 210×160 colour frame to grayscale, downsamples it to 110×84, and crops an 84×84 region that roughly covers the playing area. The paper gives the reason for the crop: the team used an existing GPU implementation of 2D convolutions, from Krizhevsky's AlexNet work, which expects square inputs. The last four such frames are stacked into the 84×84×4 input from Figure 3.

The network has two convolutional layers and two fully connected ones. The first convolution applies 16 filters of size 8×8 with stride 4, followed by a ReLU; the output side is $(84-8)/4 + 1 = 20$, so the output is 20×20×16. The second applies 32 filters of size 4×4 with stride 2 and a ReLU, giving $(20-4)/2+1 = 9$ and an output of 9×9×32, which is 2,592 numbers. A fully connected layer maps those to 256 ReLU units, and a final linear layer has one output per valid action. The parameter counts follow:

At $K = 18$ actions the head has 4,626 parameters and the total is 680,770, about 2.7 MB in 32-bit floats; at $K = 4$ the head has 1,028. The fully connected layer holds 97.5% of the weights. The paper names this network, trained with its method, a Deep Q-Network: DQN.

Earlier approaches fed the network a state and an action and read out one value, so scoring 18 actions took 18 forward passes. DQN feeds in only the state and reads out all action values at once, a design later value-based methods kept. The max over actions, which both the target in (3) and the greedy policy need at every step, becomes one forward pass and a max over a vector of length $K$. Across games only this head changes width, from 4 to 18 outputs; every layer before it has the same shape.

The training settings are the same for every game: RMSProp with minibatches of 32; $\varepsilon$ annealed from 1 to 0.1 over the first million frames and held at 0.1 after that; 10 million frames of training; a replay memory of the most recent million frames. RMSProp divides each weight's step by a running average of that weight's recent gradient magnitudes, the same idea the Adam optimizer later combined with momentum. The paper cites no source for RMSProp (at the time it existed as a slide in Hinton's Coursera course) and states no learning rate; the 0.00025 in most implementations is from 2015. A fresh network is trained from scratch for each game; the seven share the architecture and settings but no weights.

Two more settings change the data the network sees. Rewards are clipped during training: every positive reward becomes +1, every negative one −1, and zero stays zero. Score scales differ widely between games (a Breakout brick is worth 1, 4 or 7 points depending on its row; changing a cube's colour in Q*bert is worth 25), and TD targets inherit the reward scale. Without clipping, gradient magnitudes would differ between games as much as the scores do, and a learning rate small enough for the largest would crawl on the smallest. The paper says clipping "limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games", and names the cost: the agent "cannot differentiate between rewards of different magnitude".

The other setting is frame skipping. The agent sees and chooses an action only on every $k$th frame and repeats its last action on the frames in between. Running the emulator one frame is much cheaper than a forward pass of the network, so this lets the agent play roughly $k$ times more games in about the same time. With $k = 4$ the agent makes 15 decisions per second of 60-frame-per-second game time. Skipping and stacking are separate mechanisms: skipping sets which frames the agent observes, and stacking combines the last four observed frames into one input. Space Invaders is the one game with a different value, $k = 3$, because at $k = 4$ the lasers became, in the paper's words, "invisible because of the period at which they blink": the frames on which they were drawn fell on skipped frames, which never reach the stack. The paper calls this "the only difference in hyperparameter values between any of the games".

The 2015 Nature version handled flicker differently: it built each observed frame as the pixel-wise maximum of that frame and the one before it, so a sprite drawn on either of the two frames appears in the input.

Assembled, the algorithm fits on one screen:

# Algorithm 1, deep Q-learning with experience replay (2013 version)
D = ReplayMemory(capacity=N)         # the most recent transitions
theta = init_random()                # ONE set of weights: no frozen copy
for episode in range(M):
    phi = preprocess(first_frames)   # 84x84x4
    for t in range(T):
        if rand() < eps:             # eps: 1.0 -> 0.1 over the first 1M frames
            a = random_action()
        else:
            a = argmax(Q(phi, theta))   # one forward pass scores all K actions
        r, frame, done = emulator.step(a)   # a is held for k frames (skip)
        r = sign(r)                  # reward clipped to -1, 0 or +1
        phi2 = preprocess(frames + [frame])
        D.store(phi, a, r, phi2, done)
        # learn from 32 random past transitions, not from the one just seen
        P, A, R, P2, DONE = D.sample(32)     # P, P2: 32x84x84x4; A, R: 32
        q_next = Q(P2, theta).max(axis=1)    # 32xK -> 32, same live theta
        y = R + gamma * q_next * (1 - DONE)  # terminal transition: y = r
        y = stop_gradient(y)                 # the target is held constant
        q = Q(P, theta)[range(32), A]        # 32xK -> the 32 taken actions
        theta = rmsprop_step(theta, grad(mean((y - q) ** 2), theta))
        phi = phi2
# gamma and the learning rate are not stated in the 2013 paper

The published Algorithm 1 writes the greedy line as max_a Q*(φ(s_t),a;θ). It means argmax, since a max returns a value rather than an action, and the star is a slip, since the learned network is $Q$, not the unknown $Q^*$; the listing renders it corrected. In the line marked same live theta, targets come from the weights being updated; the 2015 version computes them from a frozen copy.

Results on seven Atari games

Before the score table, the paper checks that the network's predicted values rise ahead of rewards, as Bellman values should. It runs a trained Seaquest agent and plots the predicted value over a 30-frame segment. The reward in that segment is a single spike when a torpedo hits an enemy. The predicted value jumps when the enemy appears (point A in the paper's Figure 3), rises while the torpedo travels, peaks just before the hit (B), and falls back once the enemy is gone (C). Equation (1) predicts this shape: the discounted value of the coming +1 has been propagated back into the states that lead to it.

The same quantity solved a practical problem: monitoring training. Average episode score is very noisy, because a small change in the weights can change which states the policy visits. The paper instead collects a fixed set of states with a random policy before training starts and tracks the average of the maximum predicted Q over that set. It plots that curve for Breakout and Seaquest, where it rises much more smoothly than the average score, says the other five games give similarly smooth curves, and reports no divergence "in any of our experiments", while noting that the method is "lacking any theoretical convergence guarantees". Each training epoch of 50,000 minibatch updates took about 30 minutes; the paper does not state the total training time per game.

The learning methods (Sarsa, Contingency, DQN) are scored by average total reward under an $\varepsilon$-greedy policy with $\varepsilon = 0.05$, so an agent also has to survive its own occasional random press. "Human" is the median score of an expert player after about two hours of play per game; the paper notes its human scores are much higher than those in the benchmark's original paper. The benchmark is the Arcade Learning Environment (ALE), which gives every method the same screen, score and controls. Sarsa and Contingency were the best published reinforcement-learning methods on it, and both used hand-engineered visual features, including background subtraction and treating each of the 128 colours as a separate channel. DQN, from raw pixels, beats both on all seven games.

The abstract says DQN "outperforms all previous approaches on six of the games". The seventh game comes from a different comparison. HNeat, a neuroevolution method, was only evaluated on its highest-scoring single episode, in an emulator that then always started from the same state, so a fixed sequence of actions repeats its score exactly. DQN's averages include 5% random presses. Table 1 therefore also reports DQN's best single episode, and in the best-episode view DQN Best is ahead everywhere except Space Invaders, where HNeat Best scores 1,720 to DQN's 1,075. Pong is not an exception: DQN Best scores 21, a shutout, against HNeat Best's 19.

Against the human median, DQN wins on Breakout (168 against 31), Enduro (470 against 368) and Pong (20 against −3, on a scale from −21 to 21), and the paper calls Beam Rider close to human performance (4,092 against 7,456). It loses by large margins on Q*bert (1,952 against 18,900), Seaquest (1,705 against 28,010) and Space Invaders (581 against 3,690). The paper attributes the losses to games that require "a strategy that extends over long time scales". Seaquest is the clearest case: playing it well means tracking oxygen and surfacing on a schedule of tens of seconds, while DQN's input covers about a quarter of a second, and its discount weights rewards beyond roughly $1/(1-\gamma)$ steps lightly.

What the 2015 Nature version changed

The 2015 Nature paper scaled the method to 49 games and added the stabilizer this paper lacks: a target network, a copy $\theta^-$ of the weights used only to compute targets and refreshed every 10,000 updates. Between refreshes, updating θ no longer moves the target for any given transition, so each 10,000-update stretch is a regression toward fixed labels. That version also used three convolutional layers and 512 hidden units, clipped the TD error, trained for 50 million frames, and stated $\gamma = 0.99$. Its ablation on Breakout, at this paper's 10-million-frame scale, separates the two stabilizers: 316.8 with both, 240.7 with replay only, 10.2 with the target network only, and 3.2 with neither. Replay accounts for most of the gain, and the target network adds to it rather than replacing it.

Later work followed the limitations this paper names. Uniform sampling became prioritized experience replay. The max in the target is biased upward when the estimates are noisy, which Double DQN corrected by choosing the action with one set of weights and evaluating it with another. Dueling DQN split the output head into a state value and per-action advantages, and Rainbow combined these changes. DeepMind's A3C paper (2016) decorrelated updates a different way, by running many actors in parallel instead of keeping a replay memory. The logged replay data of DQN agents later became an offline dataset in its own right: the Decision Transformer trains on a sample of it.