Seq2Seq: Sequence to Sequence Learning with Neural Networks

One LSTM reads a sentence into a fixed vector; a second LSTM writes the translation from it.

Sutskever, Vinyals and Le trained two four-layer LSTMs on 12 million English-French sentence pairs. Reversing the word order of every source sentence raised test BLEU from 25.9 to 30.6, and an ensemble of five reversed models scored 34.81, above the 33.30 of a phrase-based statistical system.

The 2014 paper that made the encoder-decoder a standard tool: what the 8000-number sentence vector holds, why reversing the input helps even though the average distance between matching words stays the same, and what 34.81 BLEU does and does not beat.

In 2014 a deep neural network could classify an image or a frame of speech, but its input and its output both had to be vectors of a fixed size. An image is a fixed grid of pixels and a label is one entry from a fixed list. Translation breaks that on both sides. An English sentence can be three words or thirty, its French translation can have a different length, and which French word corresponds to which English word changes from sentence to sentence.

Ilya Sutskever, Oriol Vinyals and Quoc Le, at Google, proposed one recipe for any problem of this shape. One recurrent network reads the input a token at a time and ends in a fixed-size state. A second recurrent network starts from that state and writes the output a token at a time, stopping when it writes an end-of-sentence symbol. Both networks are LSTMs, four layers deep. Nothing in the recipe is specific to language beyond the vocabularies, so it applies the same way to any task that maps a sequence to a sequence.

On the WMT'14 English-to-French task, an ensemble of five of these models scored 34.81 BLEU, above the 33.30 of a phrase-based statistical machine translation (SMT) system, the kind of system built from word-alignment tables and phrase dictionaries that dominated translation at the time. The paper describes this as the first time a pure neural translation system beat a phrase-based baseline on a large-scale task by a sizeable margin.

The sections below cover: the model and what it computes; why it uses LSTMs; the change to the training data, reversing every source sentence, that the authors call one of their key contributions; training and gradient clipping; beam-search decoding; the results; and what the fixed vector learned.

Variable length in, variable length out

A recurrent neural network (RNN) processes a sequence with the same small network applied at every step. It keeps a hidden state $h_t$, a vector that summarizes everything read so far, and updates it once per input:

At step $t$ the network multiplies the new input $x_t$ by one matrix and the previous state $h_{t-1}$ by another, adds them, and passes the sum through a sigmoid to get the new state. The output $y_t$ is a linear read-out of the state: a vector of raw scores, which a softmax later turns into probabilities. The three matrices are the same at every step, so one set of weights handles a sequence of any length.

This maps a sequence to a sequence of the same length, one output per input, which works when the alignment is known in advance: tagging each word with its part of speech, for example. Translation has neither property. The output length differs from the input length, and the first French word can depend on the last English word, so the model should read the whole source before it writes anything. Connectionist temporal classification, the other common sequence method of the time, allows different lengths but assumes the alignment between input and output is monotonic, which word reordering violates.

The paper separates reading from writing. One RNN reads the whole source and keeps only its final state, a fixed-size vector $v$. A second RNN starts from $v$ and generates the target. The reader never has to commit to an output early, and the writer never sees the raw source. The idea was not new: Kalchbrenner and Blunsom (2013) had mapped a sentence to a vector and back with a convolutional encoder, and Cho et al. (2014) had used an RNN encoder-decoder to rescore a phrase-based system. This paper made the version with deep LSTMs translate directly, and found that reversing the source sentences improved it markedly.

How Seq2Seq works

The decoder is a conditional language model. A language model assigns a probability to a sentence; a conditional one assigns it given some context, here the source sentence. The chain rule of probability splits the probability of a whole target sentence into one factor per word:

The source $x_1, \dots, x_T$ has length $T$, the target $y_1, \dots, y_{T'}$ has its own length $T'$, and $v$ is the encoder's final state, which stands in for the whole source. The equation says: the probability of the translation is the probability of its first word given the source, times the probability of the second word given the source and the first word, and so on to the last. The factorization is an exact identity that holds for any distribution over sequences. The modeling decision is how each factor is computed: here the decoder LSTM, started from $v$, outputs one factor per step.

Each factor is a softmax over the full target vocabulary of 80,000 French words. At step $t$ the decoder's top layer produces a 1000-dimensional state, a weight matrix turns it into one score $z_i$ per word, and the softmax exponentiates and normalizes the scores so they sum to one:

Every sentence in the data ends with a special <EOS> (end-of-sentence) token, and <EOS> is one of the words the softmax can choose. The decoder writes words until it chooses <EOS>, so the model defines a probability for outputs of every length, and the length of each translation is predicted along with its words.

The following example uses the model's real shapes and made-up probabilities. Take the source "the cat sleeps" and the target "le chat dort". The encoder is fed the reversed source, sleeps cat the <EOS>. Each token is looked up as a row of the 160,000 × 1000 source embedding table, giving a 1000-dimensional vector, and each of the four LSTM layers updates its 1000-number hidden state and 1000-number cell state. After the fourth token the encoder stops; its eight state vectors, 8000 numbers in total, are $v$. The decoder starts from $v$ and at each step multiplies its top-layer state by the 80,000 × 1000 output matrix to get 80,000 scores. Suppose the softmax gives "le" probability 0.61, then "chat" 0.72 given "le", then "dort" 0.35, then <EOS> 0.88. Equation (1) multiplies them, and in log space the product becomes a sum, which is the form training and decoding both use:

Press Play to watch the source flow into $v$ and the target come back out, and switch the source between its written and reversed order:

The figure draws the two networks as one chain because backpropagation runs through them as one chain. The paper's actual model uses two separate LSTMs with their own weights, for two reasons it gives: separate weights add parameters at negligible compute cost, and they make it natural to train one model on several language pairs at once. The decoder is an ordinary LSTM language model whose initial state is $v$ instead of zeros, and that initial state is its only access to the source. Training is end to end: the loss on the target words backpropagates through $v$ into every encoder weight, so the encoder learns whatever summary of the source helps the decoder predict the translation.

The 8000-number vector and the 384M parameters

Both LSTMs have four layers of 1000 cells, with 1000-dimensional word embeddings. An LSTM layer carries two state vectors, a hidden state and a cell state (the next section explains why), so the encoder's final state is $4 \times 1000 \times 2 = 8000$ numbers. The paper states the total, "8000 real numbers to represent a sentence"; the factor of 2 comes from counting both vectors in each layer. A three-word sentence and a forty-word sentence both arrive at the decoder as the same 8000 numbers. This is the fixed-vector bottleneck: whatever the decoder will know about the source has to be in those numbers when the encoder stops.

Depth was a measured choice. Deep LSTMs did significantly better than shallow ones, and each added layer reduced perplexity by nearly 10%. Perplexity is the exponential of the average negative log-probability per word; a perplexity of 5 means the model is as uncertain as if it were choosing uniformly among 5 words. The paper suggests the gain may come from the larger hidden state that each extra layer brings, and it does not compare against a wider shallow model, so depth and state size are not separated.

The model has 384M parameters in total, and most of them are vocabulary tables. A worked count:

• source embeddings: 160,000 words × 1000 = 160M
• target embeddings: 80,000 words × 1000 = 80M
• output softmax matrix: 80,000 words × 1000 = 80M
• recurrent weights: 64M, which the paper splits as 32M in the encoder and 32M in the decoder. Each LSTM layer has four gate blocks, each reading a 1000-number input and the 1000-number previous state, so a layer holds $4 \times 2000 \times 1000 = 8\text{M}$ weights and four layers hold 32M (biases and peephole weights add about 7,000 per layer).

160M + 80M + 80M + 64M = 384M. The vocabulary tables account for 320M of it, five-sixths, and the machinery that actually reads and writes sequences is the remaining 64M. In 32-bit floats the full model is about 1.5 GB. When the introduction says "384M parameters and 8,000 dimensional state each", the "each" refers to each of the five models in the ensemble.

Why an LSTM, not a plain RNN

Training any RNN uses backpropagation through time: unroll the network over the sequence, as in Figure 1, and send the error from each output back through every step that led to it. In a plain RNN each backward step multiplies the signal by the derivative of the sigmoid and by the recurrent matrix $W^{hh}$. For a single unit that is a scalar factor $\gamma$, and after $k$ steps the signal has been scaled by $\gamma^k$. The sigmoid's derivative is at most 0.25, so $\gamma$ is often well below 1. With $\gamma = 0.9$, 30 steps leave $0.9^{30} = 0.042$ of the signal; with $\gamma = 1.1$ the same 30 steps multiply it by 17. These are the vanishing and exploding gradient problems. When the gradient vanishes, the network cannot learn that the first source word should influence a target word 30 steps later, because almost none of the error reaches that far back.

The LSTM (long short-term memory) adds a second state vector, the cell state $c_t$, which is updated by addition instead of being recomputed through a squashing function. The paper uses the formulation from Graves (2013):

The three gates $i_t$, $f_t$, $o_t$ are sigmoids, so each entry is between 0 and 1, and they multiply element-wise. The input gate $i_t$ sets how much of the new candidate to write into the cell, the forget gate $f_t$ sets how much of the old cell to keep, and the output gate $o_t$ sets how much of the cell to expose as the hidden state $h_t$. The weights $W_{ci}, W_{cf}, W_{co}$ are diagonal "peephole" connections that let the gates see the cell. The forget gate and the peepholes were added after the original 1997 LSTM.

The third line explains the gradient behaviour. Along the direct path from $c_{t-1}$ to $c_t$, the derivative is the forget gate $f_t$ itself, with no sigmoid derivative and no full matrix in between. Over $k$ steps the signal on that path is scaled by the product of the forget gates, and the network learns those values per unit: a unit that needs to hold a fact for 30 steps can keep $f$ near 1. With $f = 0.98$, 30 steps leave $0.98^{30} = 0.55$ of the signal, against 0.042 for the plain RNN above. Drag both factors and compare the curves:

The cell path cannot explode, because $f \le 1$, but the other paths can: the gates and the hidden state are computed through ordinary weight matrices at every step. The paper's wording is that LSTMs "tend to not suffer from the vanishing gradient problem" but "can have exploding gradients", which is why training clips the gradient, as covered in the training section.

Why reversing the source sentence helps

The authors fed every source sentence to the encoder in reverse order while leaving the target in order: to translate $a, b, c$ into $\alpha, \beta, \gamma$, the model reads $c, b, a$ and writes $\alpha, \beta, \gamma$. The model is unchanged. With this one change to the data, test perplexity dropped from 5.8 to 4.7 and test BLEU rose from 25.9 to 30.6, a gain of 4.7 points. The forward model trained and produced usable translations; reversing made it substantially better.

To see why, count time steps. Lay out the source and target as one chain of time steps, the path backpropagation follows: source words at steps 1 to $L$, target words at steps $L+1$ to $2L$ (ignoring the <EOS> step, and assuming for the count that word $i$ of the source translates to word $i$ of the target). The time lag of a pair is the number of steps between a source word and its translation, which is how far back the gradient has to travel to connect them.

With the source in order, word $i$ sits at step $i$ and its translation at step $L+i$, so every pair has lag $L$. Reversed, word $i$ sits at step $L+1-i$ and its lag is

The first pair is 1 step apart, the second 3, the last $2L-1$. For a five-word sentence the forward lags are 5, 5, 5, 5, 5 and the reversed lags are 1, 3, 5, 7, 9. Both lists average 5. In general:

The average lag is $L$ either way, as the paper states. What reversing changes is the minimal lag, from $L$ down to 1, at the cost of a maximal lag of $2L-1$. Drag the length and toggle the order:

The paper's explanation, which it presents as a belief rather than a proof, is about optimization. At the start of training the weights are random, and an error signal that must cross 30 steps has passed through 30 multiplications by those random weights before it reaches the source word. With the source reversed, the first few target words have their source words one or a few steps away, so backpropagation gets short, strong paths between the two languages from the first updates. The authors describe this as making it easier to "establish communication" between source and target. The explanation is not that the LSTM's memory is too short: the same paper says LSTMs tend not to suffer from vanishing gradients.

The authors expected reversing to help the start of each translation and hurt the end, where the lags are now longest. Instead, reversed models did much better on long sentences than forward ones, which the paper takes as a sign that reversing produces LSTMs that make better use of their memory. Its conclusion goes further: the authors could not train a standard RNN on the non-reversed problem at all, and believe one would train with the source reversed, a claim they did not test.

Training: the objective, SGD and gradient clipping

The training data is a cleaned subset of WMT'14 English-French: 12 million sentence pairs, 304 million English words and 348 million French words. Words outside the two vocabularies are replaced by an UNK token. Training maximizes the average log-probability of the correct translation $T$ given its source $S$, over the training set $\mathcal{S}$:

The letters are reused: here $S$ and $T$ are whole sentences, not the lengths from (1), and $\mathcal{S}$ is the set of pairs. By (1), $\log p(T \mid S)$ is a sum of per-word log-probabilities, and the negative log-probability of the correct word is the cross-entropy loss for that step. Maximizing (3) is therefore minimizing the summed cross-entropy over all target words. Each factor of (1) conditions on the true previous words, so during training the decoder is fed the reference word at each step, not its own prediction. This is called teacher forcing, and it lets every step of a target sentence be scored in one pass. At test time there is no reference, so the decoder is fed its own previous outputs, and an early wrong word becomes the input to every later step, a situation the model never saw in training. Later work calls this mismatch exposure bias; the paper does not discuss it.

The optimizer is stochastic gradient descent without momentum. The paper does not say why momentum was left out. Every parameter starts uniform in $[-0.08, 0.08]$. The learning rate is 0.7 for the first 5 epochs; after that it halves every half epoch, and training stops at 7.5 epochs, so the last half epoch runs at about 0.02 to 0.04 depending on whether the first halving falls at epoch 5 or 5.5. Each gradient is computed on 128 sequences and divided by 128. Adam, the adaptive optimizer, was posted to arXiv in December 2014, three months after this paper, so the schedule was tuned by hand.

Because LSTM gradients can still explode, the paper enforces a hard limit on the gradient's norm: take $g$, the gradient of every parameter stacked into one vector and already divided by 128, compute its length $s = \lVert g \rVert_2$, and if $s$ exceeds 5, scale the whole vector down to length 5:

This is global-norm clipping. Every coordinate is multiplied by the same scalar, $5/s$, so the update points in the same direction and only its length is capped. If a batch produces a gradient with $s = 12.37$, every coordinate is multiplied by 0.404; a batch with $s = 3$ passes through unchanged. The alternative a reader might reach for, clamping each coordinate to a range, is what Graves (2013) applied to LSTM derivatives. It limits each coordinate separately, so a gradient with one large coordinate and several small ones is turned toward the small ones. Drag the gradient past the circle and compare:

A training step in Python-style pseudocode:

# one SGD step on a batch of 128 (source, target) pairs, teacher-forced
loss = 0
for source, target in batch:                # 128 pairs of similar length
    src = reverse(tokenize(source))         # reverse the SOURCE only
    tgt = tokenize(target) + ["<EOS>"]      # target stays in order

    h = c = zeros(4, 1000)                  # 4 layers of LSTM state
    for w in src + ["<EOS>"]:               # encoder: one token per step
        h, c = enc_lstm(src_embed[w], h, c) # src_embed: 160,000 x 1000
    # v = (h, c): 4 * 1000 * 2 = 8000 numbers, the only link to the decoder

    prev = "<EOS>"                          # first decoder input (a choice)
    for w in tgt:                           # decoder starts from state v
        h, c = dec_lstm(tgt_embed[prev], h, c)
        z = W_out @ h[3] + b                # 80,000 scores from the top layer
        loss += -log_softmax(z)[w]          # cross-entropy on the TRUE word
        prev = w                            # teacher forcing: feed the truth

g = grad(loss) / 128                        # average over the batch
s = norm(g)                                 # one global L2 norm over all params
if s > 5:
    g = 5 * g / s                           # rescale; direction unchanged
params -= lr * g                            # plain SGD, no momentum; lr = 0.7

Two engineering choices set the speed. Most training sentences are 20 to 30 words long but some exceed 100, so a random batch of 128 is mostly short sentences padded out to its longest one. Grouping sentences of roughly equal length into each batch gave a 2× speedup. And a single-GPU C++ implementation processed about 1,700 words per second, too slow, so the model was spread over an 8-GPU machine: each of the four LSTM layers on its own GPU, passing activations to the next as soon as they were ready, and the 80,000-word softmax split across the other four, each multiplying by a 1000 × 20,000 slice of the output matrix. That reached 6,300 words per second (English and French combined) at batch size 128, and training took about ten days.

Decoding with beam search

Training scores known translations. To translate a new sentence, the model should output the most likely target:

This cannot be computed exactly. The number of possible targets is the vocabulary size raised to the output length: for a 10-word sentence over 80,000 words, $80{,}000^{10}$, about $1.1 \times 10^{49}$ candidates. The cheapest approximation is greedy decoding: at each step, emit the single most probable next word. Greedy decoding can miss the best sentence because a sentence's score is the product of all its factors, and the most probable first word can lead only to continuations with low probabilities.

Beam search keeps the $B$ best partial translations (prefixes) at every step. It extends each kept prefix by every word in the vocabulary, scores each extension by its total log-probability, and keeps the $B$ best. A prefix that ends in <EOS> is removed from the beam and added to the set of complete translations, and the best complete one is the output. With $B = 12$ the decoder scores 12 × 80,000 = 960,000 extensions per step and keeps 12. Greedy decoding is the case $B = 1$. In the toy tree below, set $B = 2$ and step through: "the" leads after word 1, but after word 2 both "a cat" (−1.1) and "a fox" (−1.7) beat "the cat" (−1.8), so "the" is pruned.

On the real task the beam did not need to be wide. The paper reports that a beam of 1 already performs well and a beam of 2 gives most of the benefit: for the five-model ensemble, BLEU was 33.00 at beam 1, 34.50 at beam 2 and 34.81 at beam 12. Cost grows with both the beam and the ensemble, since every model runs once per kept hypothesis per step, and the paper notes that five models at beam 2 (10 decoder runs per step) are cheaper than one model at beam 12 (12 runs) while scoring 34.50 against 30.59. The decoder sums raw log-probabilities and the paper describes no length normalization; since every added word adds a negative term, the raw score favors shorter outputs. The loop:

# beam search of width B over the decoder, as described in Sec. 3.2
h, c = encode(reverse(tokenize(source)))    # the same fixed vector v
beam = [(["<EOS>"], 0.0, (h, c))]           # (tokens, log-prob, state)
done = []
while beam:
    cand = []
    for toks, lp, (h, c) in beam:
        h2, c2 = dec_lstm(tgt_embed[toks[-1]], h, c)
        logp = log_softmax(W_out @ h2[3] + b)   # 80,000 next-word log-probs
        for w in vocab:                         # extend by EVERY word
            cand.append((toks + [w], lp + logp[w], (h2, c2)))
    cand.sort(key=lambda t: -t[1])              # rank by total log-prob
    beam = []
    for hyp in cand[:B]:                        # keep the B best
        if hyp[0][-1] == "<EOS>":
            done.append(hyp)                    # finished: leaves the beam
        else:
            beam.append(hyp)
return max(done, key=lambda t: t[1])        # no length normalization

What it scored on WMT'14

Translation quality is measured in BLEU (Papineni et al., 2002). For each candidate it counts how many of its 1-, 2-, 3- and 4-word sequences also appear in the reference translation, clipped so a repeated word cannot be counted more often than it occurs in the reference. It takes the geometric mean of those four precisions and multiplies by a brevity penalty, $e^{1 - r/c}$ for a candidate of total length $c$ shorter than the reference length $r$. Scores are reported on a 0 to 100 scale. BLEU measures word overlap, not meaning; Papineni et al. showed it correlates with human judgments, and it was the standard automatic metric for comparing translation systems. For scale, on this test set the phrase-based baseline scores 33.30 and the best WMT'14 system 37.0, a gap of 3.7 points. The paper uses cased BLEU computed with the multi-bleu.pl script on tokenized text, which reproduces the baseline's published 33.3.

Toggle between the direct-translation results and the rescoring results, and hover a bar for its description:

The headline 34.81 comes from an ensemble of five reversed LSTMs that differ only in random initialization and the random order of minibatches, decoded with a beam of 12. A single reversed LSTM scored 30.59 and a single forward LSTM 26.17; neither single model beat the baseline, and an ensemble of two reversed models (33.27) fell just short of it. The best WMT'14 system scored 37.0 under the same evaluation (the authors' recomputation with multi-bleu.pl; statmt.org reported it as 35.8), and the ensemble did not beat that, with an output vocabulary limited to 80,000 words.

The model was also used to help the baseline instead of replacing it. The baseline's 1000-best lists are public: for each test sentence, the 1000 highest-scoring candidate translations the SMT system produced. The authors computed each candidate's log-probability under the LSTM, took an even average of that and the SMT system's own score, and output the top candidate. Rescoring with a single forward LSTM gave 35.61, a single reversed LSTM 35.85, and the five-model ensemble 36.5: 3.2 points above the baseline and within 0.5 of the best WMT'14 result. For reference, picking the best candidate from each 1000-best list with knowledge of the reference (oracle rescoring) would reach about 45, and Cho et al.'s neural rescoring reached 34.54.

How it handled long sentences and rare words

A single fixed vector was expected to fail on long sentences, and other researchers had reported poor long-sentence performance with similar models. The paper's Figure 3 shows otherwise. Its left panel sorts the test sentences by length and reports BLEU per group; the caption says there is no degradation on sentences shorter than 35 words and only a minor one on the longest. Its right panel sorts sentences by the average frequency rank of their words, where a higher rank means rarer words. The figure below reproduces both panels with the points read off the published plot:

By length, the LSTM is ahead of the baseline in every group from 7 words up, and its score generally rises with length, to about 36.4 at 35 words against the baseline's 33.5. On the longest group (up to 79 words) it falls to about 34.2, level with the baseline's 34.1. The shortest groups are noisy for both systems. The authors credit the reversed source for this behaviour: it let SGD find LSTMs that had little trouble with long sentences.

By rarity, the LSTM's lead is largest on sentences of common words (about 40.4 against 36.9 at rank 100) and shrinks as the words get rarer, until at the rarest bin it scores about 28.8 against the baseline's 30.1. The decline fits the vocabulary limit: a rarer word is more likely to fall outside the 80,000 target words, the model can only emit UNK for it, and the paper notes its score is penalized whenever the reference contains such a word, whereas the phrase-based system's vocabulary is unlimited. The first bin, rank near 0, is an exception the paper does not discuss: both systems score low there and the LSTM trails, 23.7 against 26.1.

What the sentence vector encodes

The paper's Figure 2 looks at the vectors directly. The authors fed a handful of phrases through the trained encoder, took the LSTM hidden states after each phrase, and projected them to two dimensions with principal component analysis (PCA), which keeps the two directions along which those particular points vary most. The paper does not say whether the projection used all 8000 state numbers or only the hidden states, and a 2-D projection of a few points is a qualitative check, not a measurement.

Two comparisons are in the figure. Sentences that use the same words in a different order should land far apart if the vector encodes who did what to whom; a bag-of-words model, which only counts words, would put them in the same place. Sentences that say the same thing in the active and the passive voice should land close if the vector encodes meaning more than surface form. Click a sentence to see its distances:

In the word-order panel, "Mary admires John" is about 1.8 units from "Mary is in love with John" and about 8.5 from "John admires Mary", which contains exactly the same words. In the voice panel, "She gave me a card in the garden" is about 7 units from "I was given a card by her in the garden", a passive sentence with the same meaning, and about 24 from "I gave her a card in the garden", which reverses who gave what. The paper reads this as representations that are sensitive to word order and "fairly insensitive" to replacing the active voice with the passive (the abstract says "relatively invariant").

Attention and what came next

Every word of the source reaches the decoder through the same 8000 numbers. Attention removes that limit: instead of starting from one vector and never looking back, the decoder computes, for each word it writes, a weighted average of the encoder's states at every source position, with weights it learns to put on the relevant words. This paper credits the attention mechanism to Graves (2013) and calls the work of Bahdanau, Cho and Bengio "an elegant variant of this idea" applied to translation. That paper, explained in Bahdanau attention, was posted to arXiv on 1 September 2014, nine days before this one, and this paper already lists its 28.45 BLEU in Table 1. The two appeared side by side, and over the following years attention became the standard way to remove the fixed-vector bottleneck.

The encoder-decoder structure, the <EOS>-terminated autoregressive decoder and beam-search decoding all carried forward. The Transformer (2017) kept the encoder and decoder but replaced recurrence with attention throughout. With attention, the decoder reaches any source position in one step, so the distance between a source word and its translation no longer depends on where the word sits in the input, and the lag argument behind reversing the source no longer applies.