Neural Machine Translation by Jointly Learning to Align and Translate

The attention mechanism, born in a translator.

A decoder that reads from one fixed summary of the source forgets the start of a long sentence by the time it reaches the end. This paper lets it look back at every source word, weighted, for each word it writes. That weighted look is attention.

Instead of cramming a whole sentence into one vector, let the model look back at the source for every word it writes. That look is where attention starts.

In 2014 the new way to translate with a neural network was to read the whole source sentence into a single vector and then write the translation out of that vector. One network, the encoder, consumed the English word by word and ended with a fixed list of numbers meant to hold the entire meaning. A second network, the decoder, generated the French from that list alone. Train the pair end to end on millions of sentence pairs and it worked well enough to rival systems built from a dozen hand-tuned parts. The whole sentence, however long, had to fit in that one fixed list.

This paper is where that constraint broke, and it is where the attention mechanism that now runs every large language model first appears. Bahdanau et al. stop forcing the source into one vector. They keep a separate vector for every source word, and let the decoder decide, fresh for each word it emits, which of those vectors to read and how much. The authors call it learning to align (which source words matter now) and translate (what to write) at the same time. To see why it works, a few ideas carry it: the encoder–decoder it extends, the per-word context vector that replaces the fixed one, the softmax that decides the weights, the bidirectional encoder that supplies the per-word vectors, and why the whole thing still trains with plain gradient descent.

One vector, however long the sentence

Start with the thing the paper set out to fix. In the encoder–decoder setup, the encoder reads the source and produces one fixed-length vector, written $c$, and the decoder generates the entire translation conditioned on that one $c$. The vector has the same size whether the source is four words or forty. For a short sentence that is plenty of room. For a long one it is a funnel: everything the decoder will ever need about the source has to survive being compressed into a list of numbers whose length never grows.

The authors are careful about how strongly they claim this hurts. They conjecture that the fixed-length vector is a bottleneck, and they point to a companion study (Cho et al., 2014b) that measured the symptom: a basic encoder–decoder translates short sentences well and then degrades sharply as the source gets longer. So the bottleneck is a hypothesis motivated by evidence, not a proven theorem, and the rest of the paper is the experiment that confirms it. Drag the length below and switch between the two designs. In the fixed-vector design the box never widens, so each word's share of it falls as $1/L$; in the soft-search design the encoder keeps one annotation per word, and that strip grows with the sentence:

That contrast is the paper in one picture. The fixed vector is a budget that does not grow; the annotations are a budget per word that does. Everything that follows is the machinery for the right panel: building one vector per source word, and teaching the decoder to read the right ones.

The encoder–decoder it extends

To change the model precisely, name the baseline precisely. An RNN reads the source one symbol at a time, folding each new word into a running hidden state:

Here $x_t$ is the $t$-th source word, $h_t$ is the hidden state after reading it, and $f$ is the recurrent cell that updates the state. After the last word the encoder summarizes the whole run of states into the context $c$ through some function $q$. In the systems this paper builds on, $q$ just takes the final state: Sutskever and colleagues used an LSTM for $f$ and set $c = h_{T_x}$, the state after the last word. One vector, end of source.

The decoder turns that vector back into a sentence. It defines the probability of a translation $\mathbf{y}$ by emitting one word at a time, each conditioned on the words so far and on the same $c$:

Read the second piece slowly, because the fix lives in one symbol. The next-word distribution depends on the previous word $y_{t-1}$, the decoder's own hidden state $s_t$, and the context $c$. That $c$ carries no subscript: it is identical for the first word and the fortieth. The decoder is reading from a frozen summary it cannot refresh. The change the paper makes is to give $c$ a subscript, and to recompute it for every word. Once you see that the design turns on a missing index, the rest is working out how to build a good $c_i$.

A context vector for every word

The new decoder uses a distinct context vector $c_i$ for each target word $y_i$:

Same shape as before, one index richer. The decoder state $s_i$ and the output for word $i$ now see $c_i$, a context built specifically for this step, rather than the one frozen $c$. So the question becomes: where does $c_i$ come from? It is a weighted sum of the source representations. Suppose the encoder has turned the source into a sequence of vectors $(h_1, \dots, h_{T_x})$, one per word, which the paper calls annotations. Each $h_j$ summarizes the whole sentence with a focus on the words around position $j$ (the next section builds them; for now take them as given, one rich vector per source word). The context for target word $i$ is then:

The weights $\alpha_{ij}$ are nonnegative and sum to one across the source (the next section is how), so $c_i$ is a convex combination of the annotations: a weighted average that lands somewhere inside the cloud of source vectors, pulled toward whichever ones carry the most weight right now. Put all the weight on one annotation and $c_i$ is essentially that source word; spread it evenly and $c_i$ is a blur of the whole sentence. The paper calls $c_i$ an expected annotation, the average annotation under the weight distribution $\alpha_{i\cdot}$. Worth keeping straight: that expectation is over the model's own learned weights, a deterministic mean it computes, not a Bayesian posterior over some true hidden alignment.

The point of (5) is that the decoder is no longer stuck with one summary. At every step it gets to mix a new one, drawn from the parts of the source that matter for the word it is about to write. The fixed funnel is gone; what remains is to decide the mix.

The weights come from a softmax over alignment scores

The decoder needs, at each step $i$, a number for each source word saying how relevant it is right now. Call that raw score the energy $e_{ij}$, and turn the energies into weights with a softmax:

The softmax does two jobs at once. It makes every weight positive and forces the row to sum to one, so the weights are a genuine distribution over source positions and the context (5) is a proper average. And it is smooth: a small change in an energy nudges all the weights a little, which is what lets gradients flow back into whatever produced the energies. The energy itself is an alignment model $a$, a small network that scores how well the source around position $j$ matches what the decoder is about to do. It reads two things: the decoder's hidden state $s_{i-1}$, the state just before emitting $y_i$, and the annotation $h_j$.

Notice which state it scores against. The query is $s_{i-1}$, the previous decoder state, available before $y_i$ exists, so the model can choose where to look and only then commit to a word. (A later variant, Luong's, scores against the current state instead; the Transformer drops the recurrence and scores a query against keys with a scaled dot product. Same family, different wiring, and worth not conflating.) The alignment model is parametrized as a one-hidden-layer network with a $\tanh$:

This is additive attention: combine the query $s_{i-1}$ and the annotation $h_j$ inside a $\tanh$, then project to a single number with $v_a$. The same annotation $h_j$ plays two roles, the thing being scored (the key) and the thing being averaged in (the value), with no separate projections for the two. A practical detail makes this affordable: $U_a h_j$ does not depend on the target step $i$, so it is computed once per source word and reused for every output word. The alignment model and the rest of the translator are trained together, from the same loss; alignment is not a separate pre-processing step or a fixed table.

Below is one decoder step under your control. Step through the target sentence and watch the energies become a row of weights that sums to one, then watch those weights blend the annotations into the context. The distribution slides along the source as the decoder advances, mostly tracking the matching word:

With (5), (6) and (7) the decoder has what it needs: a relevance score per source word, a normalized set of weights, and a context that is their weighted blend, all recomputed every step and all differentiable. The encoder still owes us the annotations $h_j$.

Annotations that read from both directions

Each annotation $h_j$ should describe its own source word together with the context it sits in, and context runs both ways: the words before $x_j$ and the words after. A plain forward RNN only sees the words before, so the paper reads the source with a bidirectional RNN. A forward RNN reads left to right and produces states $\overrightarrow{h}_1, \dots, \overrightarrow{h}_{T_x}$; a backward RNN reads right to left and produces $\overleftarrow{h}_1, \dots, \overleftarrow{h}_{T_x}$. The annotation for word $j$ is the two stuck together:

The forward half has read everything up to and including word $j$; the backward half has read everything from the end back to word $j$. Together they summarize the whole sentence. And because an RNN leans on its most recent input, each half is sharpest near $j$, so the concatenated $h_j$ stays focused on the words around position $j$ while still carrying the global picture. That local focus is exactly what makes the alignment weights interpretable later: when the decoder puts weight on $h_j$, it is mostly looking at the region around the $j$-th source word. One consequence of concatenation: $h_j$ is twice as wide as a single direction's state, which is why the alignment model's $U_a$ and the decoder's context weights act on a $2n$-wide vector. Hover a word to see its annotation assembled from the two passes:

One small asymmetry is easy to get backwards, so catch it now. The decoder's first state $s_0$ is initialized from the first backward annotation, $s_0 = \tanh(W_s \overleftarrow{h}_1)$. The backward RNN reaches position 1 last, after sweeping the entire sentence right to left, so $\overleftarrow{h}_1$ is the state that has just seen all of the source. The decoder, in other words, begins from a vector that already glimpsed the whole input, and then refines its view word by word through attention.

Why soft alignment trains at all

What makes the alignment learnable at all is the word soft. Older alignment ideas in translation treated the alignment as a hidden choice: word $i$ comes from source word $j$, a discrete decision you would have to search over or sample. A discrete choice is a step function, and you cannot push a gradient through a step, so you are forced into sampling and high-variance estimators (this is what later hard attention does, training the picked location with a REINFORCE-style estimator). This paper makes the alignment soft instead. The softmax (6) produces a smooth distribution, the context (5) is a differentiable average, and the alignment is explicitly not a latent variable. So the gradient of the translation loss flows straight back through the weights into the alignment model and the encoder. Alignment and translation are trained by the same backprop, on the same objective, with nothing special bolted on.

That single decision, soft rather than hard, is doing several jobs. It removes the need for a separate aligner or a fixed alignment table. It lets the model hedge, splitting weight across two or three source words when it is unsure, instead of betting everything on one. And it keeps the entire system one differentiable function from source to loss, which is the property that let a single network match systems assembled from many tuned parts. It does run up a bill: the alignment model is evaluated once for every (source word, target word) pair, so the work scales with the product $T_x \times T_y$ of the two sentence lengths. The paper judges that acceptable for sentences of 15 to 40 words and flags it as a possible limit elsewhere. It is also the first appearance of the quadratic cost that attention research has been fighting ever since.

The recurrent cell $f$ is the gated hidden unit from Cho et al. (the unit later called the GRU), not the LSTM that Sutskever used. It keeps a running state with two gates: an update gate $z_i$ that decides how much of the old state to carry forward unchanged, and a reset gate $r_i$ that decides how much of the old state to ignore when proposing a new one. Carrying the old state through nearly untouched is what lets gradients survive over many steps, the same long-memory trick the LSTM buys with more machinery. The output layer is a single maxout hidden layer (it takes the max over pairs of pre-activations, a cheap way to learn a flexible nonlinearity) feeding a softmax over the target vocabulary. In code, one decoder step is the five lines we have been describing:

# one decoder step i: emit target word y_i (bias terms omitted)
# h[j]: source annotations (2n-dim), j = 1..Tx ; s_prev: state s_{i-1}
e   = [v_a @ tanh(W_a @ s_prev + U_a @ h[j]) for j in range(Tx)]  # (7)
a   = softmax(e)                       # weights alpha_ij, sum to 1   (6)
c   = sum(a[j] * h[j] for j in range(Tx))   # context c_i = blend     (5)
s   = gru(s_prev, embed(y_prev), c)    # new decoder state s_i
y_i = softmax(maxout(s, embed(y_prev), c))  # next-word distribution  (4)

None of these pieces is the contribution; they are the solid 2014 parts the contribution is built from. The contribution is the three lines in the middle: score, normalize, blend. Swap the fixed $c$ for that per-step blend and the translator stops forgetting the start of long sentences.

What the alignments show

The test is WMT'14 English-to-French, trained on 348M words with a 30,000-word vocabulary per language. Two models, matched in size at 1000 hidden units: the fixed-vector baseline (RNNencdec) and the soft-search model (RNNsearch), each trained once on sentences up to 30 words and once up to 50. Quality is BLEU, a modified n-gram precision against reference translations, with a penalty for being too short; higher is better.

On the full test set RNNsearch-50 scores 26.75 BLEU against RNNencdec-50's 17.82, a gap of nearly nine points, roughly the difference between a translation you can read and one you have to decode. The robustness shows up most clearly against sentence length: drag the marker below and the curves tell the story the bottleneck predicted. The fixed-vector model climbs for short sentences and then falls off as the source outgrows its one vector; the soft-search model trained to length 50 stays essentially flat past 50 words, while the one trained only to 30 holds until its training length and then declines. The gap is near zero for short sentences (where one vector is enough) and widens past ten BLEU for long ones (where it is not):

Two smaller numbers make the point sharper. RNNsearch-30, trained only on short sentences, still beats RNNencdec-50 overall (21.50 versus 17.82): the better architecture trained on less wins. And trained longer, RNNsearch-50's best run reaches 28.45 on all sentences, 36.15 once sentences with unknown words are excluded. That second figure edges the conventional phrase-based system Moses, at 35.63, which had years of engineering and an extra 418M words of monolingual text behind it. Be careful with that comparison, though: on the full test set, unknown words and all, Moses still wins 33.30 to 28.45. The honest claim, and the one the paper makes, is that a single neural network drew level with phrase-based translation on known-word sentences, not that it beat it outright.

The most telling result is not a number, it is a picture. Because the weights $\alpha_{ij}$ are a distribution over source positions, you can lay them out as a heatmap and read off, for each target word, which source words it attended to. The alignments come out mostly monotonic, a bright diagonal, which is what you expect between two fairly similar languages. The interesting cells are off the diagonal, where French reorders the words. Hover the rows below: when the model writes "zone économique européenne" for "European Economic Area," it jumps to "Area" first and then walks the adjectives back, producing the anti-diagonal block instead of a straight line:

Nobody told the model that "européenne" aligns with "European." It learned the alignment as a by-product of learning to translate, because reading the right source word is what lets it predict the right target word. That the learned weights agree with how a person would align the sentences, reordering and all, is the evidence that the mechanism is doing what its name claims, attending to the relevant source.

Trace the fix back and it is one substitution with consequences. A fixed-length context vector chokes on long sentences; replace it with a fresh context per output word, built as a softmax-weighted blend of one annotation per source word, and keep the weighting soft so the whole system still trains by ordinary backprop. That bought a translator that stopped forgetting the start of long sentences and drew level with a decade of phrase-based engineering. The mechanism it introduced, a learned soft search over a set of vectors, proved far more general than translation. Three years later it would be the only thing a Transformer is made of, and the encoder and decoder RNNs around it would be thrown away.