VerifiedarXiv:1901.0286028 min
Architecture · Language modeling

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

A Transformer language model that keeps the previous segment's hidden states as extra context and encodes positions as distances, so it can use text far beyond the segment it trains on.

Transformer-XL trains on fixed-length segments like an ordinary Transformer language model, but each layer also attends to the hidden states it computed for the previous segment, cached and excluded from the gradient. Position enters each attention score as the distance between query and key, so a cached state means the same thing when it is read again one segment later.

Explaining the paperTransformer-XL: Attentive Language Models Beyond a Fixed-Length ContextDai, Yang, Yang, Carbonell, Le, Salakhutdinov · ACL 2019 · arXiv:1901.02860 ↗

A 2019 paper from Carnegie Mellon and Google Brain. It set new results on five language-modeling benchmarks, including 18.3 perplexity on WikiText-103 and the first score under 1.0 bits per character on enwik8, and it evaluates up to 1,874 times faster than the sliding-window Transformer it compares against.

A language model assigns a probability to the next token given the tokens before it. In 2018 the strongest Transformer language models, such as the 64-layer character-level model of Al-Rfou et al., were trained on separate segments of a few hundred tokens, and nothing crossed from one segment to the next. LSTMs carried a state across segments, but in practice used about 200 tokens of context. Transformer-XL (the XL is for "extra long") keeps segment-by-segment training and adds two pieces, which the paper presents as a pair because neither works alone:

  1. segment-level recurrence: every layer caches the hidden states it computed for the segment it just processed, and the next segment's attention uses them as extra keys and values;
  2. relative positional encodings: no position vector is added to the input; instead, each attention score gets terms that depend on the distance between the query and the key.

The page sets up the fixed-context model and its two problems, then covers the recurrence and how far back it reaches, why it breaks absolute positions, the relative attention score and the matrix shift that makes it cheap, a full layer with code, the evaluation speedup, and the results and ablations.

Language modeling with a fixed-length context

Given a corpus of tokens x=(x1,…,xT)\mathbf{x} = (x_1, \dots, x_T), a language model factorizes its probability one token at a time:

P(x)=∏tP(xt∣x<t)P(\mathbf{x}) = \prod_{t} P(x_t \mid x_{<t})

A neural language model encodes the context x<tx_{<t} into a vector, multiplies it with the word embeddings to get one logit per vocabulary entry, and applies a softmax. Training minimizes the average negative log-probability of the true next token. Two units report it. Perplexity is the exponential of that average in nats: 18.3 on WikiText-103 means the model is, on average, as uncertain as a uniform choice among 18.3 words. Bits per character (bpc) is the same average in bits, per character or byte: 0.99 on enwik8 means the model would compress the text to about one bit per byte, an eighth of its raw size.

A Transformer with unlimited memory could attend over the entire prefix of the corpus. Attention cost grows with the square of the length, so instead the corpus is cut into consecutive segments of LL tokens and the model is trained on each segment independently. The paper calls this the vanilla model; Al-Rfou et al. used L=512L = 512 characters.

The vanilla model has two problems. First, no dependency can be longer than LL. Information does not flow across a segment boundary in the forward pass or the backward pass, so the self-attention that lets a Transformer connect distant tokens in one step only ever sees one segment. Second, segments are cut at fixed offsets without regard to sentences, so the first token of every segment is predicted from no context at all, the second from one token, and so on. On average a token is predicted from (L−1)/2(L-1)/2 earlier tokens, and the first few tokens of each segment are predicted badly. The paper calls this context fragmentation.

At evaluation, Al-Rfou et al. avoid fragmentation by sliding the window one token at a time and scoring only its last position: every token is predicted from L−1L - 1 earlier tokens, but every prediction costs a full forward pass over LL positions, and nothing is reused from the previous step. For a 512-character window, that is 512 positions through 64 layers for each character of the test set.

Figure 1 shows the three regimes on four segments. The bars are the number of earlier tokens each token is predicted from. Click a bar to see which tokens it can attend to; the readout gives the mean over segments 2 to 4.

Figure 1 · context per token
8
Each bar is the number of earlier tokens a token is predicted from; the highlighted strip below shows them. Vanilla training restarts at every segment boundary, sliding evaluation gives every token L−1L-1 at LL forward passes per segment, and Transformer-XL with a one-segment memory gives LL to 2L−12L-1 at one pass. Drag LL.

Transformer-XL gets the sliding window's context at the vanilla model's cost: one forward pass per segment, with every token seeing at least LL earlier tokens (outside the first segment), because the previous segment is still available in a cache.

How segment-level recurrence works

Transformer-XL processes the segments of the corpus in order. While it computes segment τ\tau, it keeps the input to every layer, the sequence hτn−1∈RL×d\mathbf{h}_\tau^{n-1} \in \mathbb{R}^{L \times d} for layer nn. When segment τ+1\tau+1 arrives, layer nn prepends the stored sequence to its current input and computes keys and values from the concatenation, while the queries still come from the current segment alone:

h~τ+1n−1=[SG(hτn−1)∘hτ+1n−1]qτ+1n, kτ+1n, vτ+1n=hτ+1n−1Wq⊤, h~τ+1n−1Wk⊤, h~τ+1n−1Wv⊤hτ+1n=Transformer-Layer(qτ+1n,kτ+1n,vτ+1n)\begin{aligned} \widetilde{\mathbf{h}}_{\tau+1}^{n-1} &= \left[\mathrm{SG}(\mathbf{h}_{\tau}^{n-1}) \circ \mathbf{h}_{\tau+1}^{n-1}\right] \\ \mathbf{q}_{\tau+1}^{n},\, \mathbf{k}_{\tau+1}^{n},\, \mathbf{v}_{\tau+1}^{n} &= \mathbf{h}_{\tau+1}^{n-1}\mathbf{W}_{q}^{\top},\, \widetilde{\mathbf{h}}_{\tau+1}^{n-1}\mathbf{W}_{k}^{\top},\, \widetilde{\mathbf{h}}_{\tau+1}^{n-1}\mathbf{W}_{v}^{\top} \\ \mathbf{h}_{\tau+1}^{n} &= \text{Transformer-Layer}(\mathbf{q}_{\tau+1}^{n}, \mathbf{k}_{\tau+1}^{n}, \mathbf{v}_{\tau+1}^{n}) \end{aligned}h~τ+1n−1=[SG(hτn−1)∘hτ+1n−1]qτ+1n, kτ+1n, vτ+1n=hτ+1n−1Wq⊤, h~τ+1n−1Wk⊤, h~τ+1n−1Wv⊤hτ+1n=Transformer-Layer(qτ+1n,kτ+1n,vτ+1n)\begin{aligned} \widetilde{\mathbf{h}}_{\tau+1}^{n-1} &= \left[\mathrm{SG}(\mathbf{h}_{\tau}^{n-1}) \circ \mathbf{h}_{\tau+1}^{n-1}\right] \\ \mathbf{q}_{\tau+1}^{n},\, \mathbf{k}_{\tau+1}^{n},\, \mathbf{v}_{\tau+1}^{n} &= \mathbf{h}_{\tau+1}^{n-1}\mathbf{W}_{q}^{\top},\, \\ &\quad \widetilde{\mathbf{h}}_{\tau+1}^{n-1}\mathbf{W}_{k}^{\top},\, \\ &\quad \widetilde{\mathbf{h}}_{\tau+1}^{n-1}\mathbf{W}_{v}^{\top} \\ \mathbf{h}_{\tau+1}^{n} &= \text{Transformer-Layer}(\mathbf{q}_{\tau+1}^{n}, \\ &\quad \mathbf{k}_{\tau+1}^{n}, \mathbf{v}_{\tau+1}^{n}) \end{aligned}
(1)

[ ⋅∘⋅ ][\,\cdot \circ \cdot\,] concatenates two sequences along the length dimension, and SG\mathrm{SG} is stop-gradient: in the forward pass it passes its input through unchanged, and in the backward pass it passes no gradient. In PyTorch it is .detach(). The only change from a standard layer is that the keys and values, and through them the layer's output, depend on the cached states of the previous segment.

The stop-gradient keeps the cost of a training step fixed. Without it, the loss on segment τ+1\tau+1 would backpropagate into the computation of segment τ\tau, then τ−1\tau-1, and so on, and every step would have to keep the activations of all of them. With it, each step backpropagates through one segment, like truncated backpropagation through time for recurrent networks, which carries an RNN's hidden state forward from chunk to chunk but backpropagates only within the current chunk. The obvious alternative, training on longer segments, costs time and activation memory that grow with the square of the segment length; the cache adds keys and values without adding anything to backpropagate.

The recurrence differs from a recurrent network's in two ways. It caches a whole sequence of states per layer, not one final state. And it runs diagonally: the output of layer nn for segment τ+1\tau+1 depends on layer n−1n-1 of segment τ\tau, one layer lower, where an RNN connects the same layer across time steps. The next section shows what that diagonal does to the reach.

The cache need not be exactly one segment. The paper keeps the last MM positions per layer, a memory mτn∈RM×d\mathbf{m}_\tau^{n} \in \mathbb{R}^{M \times d} that can span several segments; the code keeps the last mem_len rows of the concatenated memory and new states. Training uses M=LM = L, and evaluation uses a memory several times longer. The WikiText-103 large model trains with L=M=384L = M = 384 and evaluates with M=1600M = 1600.

A worked layer, with the WikiText-103 base model from the released scripts: 16 layers, d=410d = 410, 10 heads of 41 dimensions, and L=M=150L = M = 150 in training. Layer nn receives the current input, 150 × 410, and its cached memory, another 150 × 410. The concatenation h~\widetilde{\mathbf{h}} is 300 × 410. The queries are 150 × 410, the keys and values 300 × 410, and each head computes a 150 × 300 score matrix. A causal mask leaves query ii (counting from 0) the 150 memory positions plus positions 0 to ii of its own segment: 151 keys for the first query, 300 for the last. The loss covers the 150 next-token predictions of the segment, and its gradient reaches the weights only through the computation on these 150 positions. The cache for one sequence holds 16 × 150 × 410 = 984,000 numbers, 3.9 MB in 32-bit floats.

How far back the model can see

One layer of attention lets the state at position pp read the layer below at positions back to the start of the memory: MM positions before its segment begins, so between MM and M+L−1M + L - 1 positions back depending on where pp sits in its segment. Each cached state among them was itself computed one segment earlier, from the layer below it and from that segment's own memory. Going down NN layers, the reach adds up to roughly N×MN \times M tokens. The paper, with M=LM = L, writes this as O(N×L)O(N \times L).

For the base model that is 16 × 150 = 2,400 tokens during training and 16 × 640 = 10,240 at evaluation, while a single query in training attends over at most 300 positions. The same per-layer growth, one window of reach added per layer, appears in sliding-window attention, where Mistral 7B reaches about 32 × 4,096 tokens back with a 4,096-token window.

Figure 2 lays out layers (rows) against positions (columns), cut into segments. The outlined state's dependencies are lit: bright cells are computed in its own segment and receive gradient from its loss, dim cells are cached states that contribute only through the forward pass. At the default N=3N = 3, L=4L = 4, M=4M = 4, the top state at the last position reaches 15 tokens back: 7 in the first layer down (it is the last token of its segment, so M+L−1M + L - 1), then 4 for each layer below. Set MM to 0 and the lit region stops at the segment's left edge, which is the vanilla model. Click any cell to trace its cone instead.

Figure 2 · the reach through cached states
3
4
4
Layers (rows, embeddings at the bottom) by token positions, in segments. The outlined state depends on the lit cells: bright ones are recomputed in its segment with gradient, dim ones are read from the cache through the stop-gradient. Drag the number of layers, the segment length and the memory length; click a cell to select it.

The dim cells receive no gradient from this loss. The weights still learn to use the memory, since the keys and values computed from cached states pass through Wk\mathbf{W}_k and Wv\mathbf{W}_v, which do receive gradient. What is never trained by a later loss is the computation that wrote a cached state: no gradient from segment τ+1\tau+1's loss reaches segment τ\tau's computation, so nothing trains segment τ\tau to store what segment τ+1\tau+1 will need.

Why absolute positions break when states are reused

A standard Transformer adds a positional encoding to each input: row ii of a matrix U∈RLmax⁡×d\mathbf{U} \in \mathbb{R}^{L_{\max} \times d} is added to the embedding of the token at position ii of the segment. With the recurrence, both segments get the same rows. Schematically, with Esτ\mathbf{E}_{\mathbf{s}_\tau} the embeddings of segment τ\tau and ff the whole network,

hτ+1=f(hτ,Esτ+1+U1:L)hτ=f(hτ−1,Esτ+U1:L)\begin{aligned} \mathbf{h}_{\tau+1} &= f(\mathbf{h}_{\tau}, \mathbf{E}_{\mathbf{s}_{\tau+1}} + \mathbf{U}_{1:L}) \\ \mathbf{h}_{\tau} &= f(\mathbf{h}_{\tau-1}, \mathbf{E}_{\mathbf{s}_{\tau}} + \mathbf{U}_{1:L}) \end{aligned}
(2)

Token xτ,jx_{\tau,j} in the memory and token xτ+1,jx_{\tau+1,j} in the current segment carry the same encoding Uj\mathbf{U}_j, so a query has no positional way to tell whether a key is jj tokens into its own segment or into the previous one. The paper says this would cause a large loss in performance; Table 6 contains no run that combines recurrence with an absolute encoding. Renumbering does not help: the cached states were computed when their tokens had other positions, and after the first layer that encoding is mixed into every number of the stored state. Changing it means recomputing the states, which is the cost the cache was meant to avoid.

Transformer-XL adds no position to the input: hτ0\mathbf{h}^0_\tau is the word embeddings alone, so no hidden state contains a position. Position enters only inside each attention score, as a function of the distance i−ji - j between the query and the key, computed when the score is computed. A cached key is further away: a query at index ii of the current segment is M+i−jM + i - j tokens from memory key jj and i−ji - j tokens from current key jj. Absolute order is still recoverable, since distances to every earlier token determine where a token sits.

In Figure 3 a query in the current segment looks at six cached keys and its own segment. Under the absolute encoding, the cached and current keys with the same index share a code, joined by a line. Switch to the relative encoding and each visible key shows its distance from the query. Click a token of the current segment to move the query.

Figure 3 · position codes with a memory
Each cell is a key the query can attend to (future keys are masked). With absolute encodings, a cached key and a current key with the same index carry the same code; with relative encodings every key has its own distance i−ji-j. Toggle the encoding; click the current segment to move the query.

How the relative attention score works

The paper derives its encoding by expanding the score of a standard Transformer. With column vectors, query qi=Wq(Exi+Ui)\mathbf{q}_i = \mathbf{W}_q(\mathbf{E}_{x_i} + \mathbf{U}_i) and key kj=Wk(Exj+Uj)\mathbf{k}_j = \mathbf{W}_k(\mathbf{E}_{x_j} + \mathbf{U}_j), the dot product splits into four terms:

Ai,jabs=Exi⊤Wq⊤WkExj⏟(a)+Exi⊤Wq⊤WkUj⏟(b)+Ui⊤Wq⊤WkExj⏟(c)+Ui⊤Wq⊤WkUj⏟(d)\begin{aligned} \mathbf{A}_{i,j}^{\mathrm{abs}} &= \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{E}_{x_j}}_{(a)} + \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{U}_j}_{(b)} \\ &\quad + \underbrace{\mathbf{U}_i^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{E}_{x_j}}_{(c)} + \underbrace{\mathbf{U}_i^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{U}_j}_{(d)} \end{aligned}Ai,jabs=Exi⊤Wq⊤WkExj⏟(a)+Exi⊤Wq⊤WkUj⏟(b)+Ui⊤Wq⊤WkExj⏟(c)+Ui⊤Wq⊤WkUj⏟(d)\begin{aligned} \mathbf{A}_{i,j}^{\mathrm{abs}} &= \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{E}_{x_j}}_{(a)} \\ &\quad + \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{U}_j}_{(b)} \\ &\quad + \underbrace{\mathbf{U}_i^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{E}_{x_j}}_{(c)} \\ &\quad + \underbrace{\mathbf{U}_i^{\top}\mathbf{W}_q^{\top}\mathbf{W}_k\mathbf{U}_j}_{(d)} \end{aligned}
(3)

(a) is content against content, (b) is the query's content against the key's position, (c) the query's position against the key's content, and (d) position against position. Transformer-XL rewrites the four terms so that only the distance i−ji - j appears:

Ai,jrel=Exi⊤Wq⊤Wk,EExj⏟(a)+Exi⊤Wq⊤Wk,RRi−j⏟(b)+u⊤Wk,EExj⏟(c)+v⊤Wk,RRi−j⏟(d)\begin{aligned} \mathbf{A}_{i,j}^{\mathrm{rel}} &= \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_{k,E}\mathbf{E}_{x_j}}_{(a)} + \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_{k,R}\mathbf{R}_{i-j}}_{(b)} \\ &\quad + \underbrace{u^{\top}\mathbf{W}_{k,E}\mathbf{E}_{x_j}}_{(c)} + \underbrace{v^{\top}\mathbf{W}_{k,R}\mathbf{R}_{i-j}}_{(d)} \end{aligned}Ai,jrel=Exi⊤Wq⊤Wk,EExj⏟(a)+Exi⊤Wq⊤Wk,RRi−j⏟(b)+u⊤Wk,EExj⏟(c)+v⊤Wk,RRi−j⏟(d)\begin{aligned} \mathbf{A}_{i,j}^{\mathrm{rel}} &= \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_{k,E}\mathbf{E}_{x_j}}_{(a)} \\ &\quad + \underbrace{\mathbf{E}_{x_i}^{\top}\mathbf{W}_q^{\top}\mathbf{W}_{k,R}\mathbf{R}_{i-j}}_{(b)} \\ &\quad + \underbrace{u^{\top}\mathbf{W}_{k,E}\mathbf{E}_{x_j}}_{(c)} \\ &\quad + \underbrace{v^{\top}\mathbf{W}_{k,R}\mathbf{R}_{i-j}}_{(d)} \end{aligned}
(4)

Three changes take (3) to (4).

  1. The key's absolute position Uj\mathbf{U}_j in terms (b) and (d) becomes a relative encoding Ri−j\mathbf{R}_{i-j}, the sinusoid of the original Transformer evaluated at the distance. It has no learned parameters.
  2. The query's position term Ui⊤Wq⊤\mathbf{U}_i^{\top}\mathbf{W}_q^{\top} in (c) and (d) becomes a learned vector, uu in (c) and vv in (d), both in Rd\mathbb{R}^d. If only distances matter, the query's own position should not change how much it favors a word or a distance, so one vector serves every query position.
  3. The key projection splits in two: Wk,E\mathbf{W}_{k,E} for content keys and Wk,R\mathbf{W}_{k,R} for position keys.

Each term of (4) then has a reading. (a) is content-based addressing: does this word answer this query. (b) is a content-dependent positional bias: how far back this particular query tends to look. (c) is a global content bias: some words are worth attending to from anywhere. (d) is a global positional bias: nearby tokens are usually more useful than distant ones. In the paper's attention maps of the trained model (its Appendix D), term (b) produces most of the focus on nearby tokens, and term (d) is flatter, favoring a longer context.

The sinusoid follows Attention Is All You Need: component 2m2m of Rk\mathbf{R}_k is sin⁡(k/100002m/d)\sin(k / 10000^{2m/d}) and component 2m+12m+1 is the cosine of the same angle (the code puts all the sines first and all the cosines second, a fixed reordering). With d=410d = 410 the fastest pair repeats every 2π≈6.32\pi \approx 6.3 tokens and the slowest about every 60,000, so every distance a memory can hold gets a distinct code. Shaw et al. (2018) instead learn one vector per distance up to a clipping value (16 in their translation models), give every longer distance the clipped vector, and have no terms (c) and (d). A learned table of that kind gives all distances beyond its range one shared vector, while the sinusoid gives each distance its own code and Wk,R\mathbf{W}_{k,R} is a linear map of it. The paper credits this for a result in Table 6, below: a model trained with a memory of one length runs with a memory several times longer and gets better, which the Shaw variant does not.

The code computes the four terms as two products. Terms (a) and (c) share the content key Wk,EExj\mathbf{W}_{k,E}\mathbf{E}_{x_j}, so they are (qi+u)⊤kj(\mathbf{q}_i + u)^{\top}\mathbf{k}_j; terms (b) and (d) share the position key, so they are (qi+v)⊤Wk,RRi−j(\mathbf{q}_i + v)^{\top}\mathbf{W}_{k,R}\mathbf{R}_{i-j}. The scores are scaled by 1/dhead1/\sqrt{d_{\text{head}}} as in any Transformer, a factor the paper's equations leave out. uu and vv are the parameters r_w_bias and r_r_bias, one 41-number vector per head in the base model. The PyTorch code shares one pair across all layers; the TensorFlow scripts for WikiText-103 and One Billion Word set untie_r=True, which gives every layer its own pair, while the enwik8 and text8 scripts keep them shared.

# Relative attention scores for one layer (official PyTorch code, simplified).
# L = 150 queries, M = 150 cached positions, H = 10 heads of dh = 41 (d = 410).
cat = torch.cat([mem, h], 0)             # [M+L, B, d]   mem was detached
q = (h @ W_q).view(L, B, H, dh)          # queries: current segment only
k = (cat @ W_kE).view(M + L, B, H, dh)   # content keys over memory + segment
v = (cat @ W_v).view(M + L, B, H, dh)
pos = sinusoid(arange(M + L - 1, -1, -1))  # [M+L, d]  distances M+L-1 ... 0
r = (pos @ W_kR).view(M + L, H, dh)      # one position key per distance
AC = einsum('ibhd,jbhd->ijbh', q + u, k)   # terms (a) + (c)
BD = einsum('ibhd,jhd->ijbh', q + v, r)    # terms (b) + (d), column = distance
BD = rel_shift(BD)                         # move each row to distance M + i - j
score = (AC + BD) / sqrt(dh)               # [L, M+L, B, H]
score = score.masked_fill(j > M + i, -inf)   # causal mask over memory + segment
out = softmax(score, dim=1) @ v              # weighted sum of values

Computing the position term in linear time

Computed literally, terms (b) and (d) need the vector Wk,RRi−j\mathbf{W}_{k,R}\mathbf{R}_{i-j} for every pair of query and key, L×(M+L)L \times (M + L) projected vectors of dd numbers each, 45,000 of them for the base model. But i−ji - j only takes the M+LM + L values from 0 to M+L−1M + L - 1, so Appendix B projects each distance once. It stacks the projected encodings in reverse order,

Qk=Wk,RRM+L−1−k,k=0,…,M+L−1\mathbf{Q}_k = \mathbf{W}_{k,R}\mathbf{R}_{M+L-1-k}, \qquad k = 0, \dots, M+L-1Qk=Wk,RRM+L−1−k,k=0,…,M+L−1\begin{gathered} \mathbf{Q}_k = \mathbf{W}_{k,R}\mathbf{R}_{M+L-1-k}, \\ k = 0, \dots, M+L-1 \end{gathered}

and multiplies all queries by all of them at once, B~=qQ⊤\widetilde{\mathbf{B}} = \mathbf{q}\mathbf{Q}^{\top}, an L×(M+L)L \times (M + L) matrix in which column kk holds distance M+L−1−kM + L - 1 - k in every row. The matrix the score needs, B\mathbf{B}, wants distance M+i−jM + i - j in row ii, column jj. Query ii sits at position M+iM + i of the concatenated sequence, so its row of B\mathbf{B} is its row of B~\widetilde{\mathbf{B}} shifted left by L−1−iL - 1 - i: the last query is not shifted, the first is shifted by L−1L - 1. Term (d) works the same way with vv in place of each query, and its unshifted version is a single matrix-vector product Qv\mathbf{Q}v. The projections drop from L(M+L)L(M + L) vectors to M+LM + L, 300 instead of 45,000, which is the linear cost the paper refers to; the dot products, one per query-key pair, cost the same as the content scores.

The code does the shift without a loop or an index gather. It prepends a column of zeros, reinterprets the buffer as (M+L+1)×L(M + L + 1) \times L, drops the first row, and reinterprets the rest as L×(M+L)L \times (M + L). Dropping LL entries from the front of a row-major buffer whose rows are M+L+1M + L + 1 long moves the start of row ii forward by L−1−iL - 1 - i, which is the required shift. Figure 4 runs those four steps on small matrices. The numbers are distances and the colors are the rows of B~\widetilde{\mathbf{B}} the entries came from; at step 4, hover a cell to compare the distance it needs with the one it holds.

Figure 4 · the relative shift
step 0
4
3
B~=qQ⊤\widetilde{\mathbf{B}} = \mathbf{q}\mathbf{Q}^{\top} has the same distance in every row of a column. Pad, reinterpret, drop and reinterpret, and every unmasked cell of row ii holds distance M+i−jM + i - j. Play or scrub the steps; drag the number of queries and the memory length.

At step 4 every cell left of the white line has the right distance from the right row. The cells to its right hold padding zeros or entries wrapped in from the next row, and they are exactly the cells the causal mask removes, so the shift is only correct together with the mask.

def rel_shift(x):                      # x: [L, K, ...], K = M + L
    pad = zeros(L, 1, *x.shape[2:])
    x = cat([pad, x], dim=1)            # [L, K+1]: zero column in front
    x = x.view(K + 1, L, *x.shape[2:])  # same buffer, read as (K+1) x L
    return x[1:].view(L, K, *x.shape[2:])  # drop L entries, read as L x K
# Row i of the result is row i of x shifted left by L - 1 - i. Cells with
# j > M + i hold zeros or values wrapped in from row i + 1: the mask hides them.

One Transformer-XL layer, end to end

Combining the recurrence with the relative score gives the paper's summary of an NN-layer model, written for one attention head. For n=1,…,Nn = 1, \dots, N:

h~τn−1=[SG(mτn−1)∘hτn−1]qτn, kτn, vτn=hτn−1Wqn⊤, h~τn−1Wk,En⊤, h~τn−1Wvn⊤Aτ,i,jn=qτ,in⊤kτ,jn+qτ,in⊤Wk,RnRi−j+u⊤kτ,j+v⊤Wk,RnRi−jaτn=Masked-Softmax(Aτn) vτnoτn=LayerNorm(Linear(aτn)+hτn−1)hτn=Positionwise-Feed-Forward(oτn)\begin{aligned} \widetilde{\mathbf{h}}_{\tau}^{n-1} &= \left[\mathrm{SG}(\mathbf{m}_{\tau}^{n-1}) \circ \mathbf{h}_{\tau}^{n-1}\right] \\ \mathbf{q}_{\tau}^{n},\, \mathbf{k}_{\tau}^{n},\, \mathbf{v}_{\tau}^{n} &= \mathbf{h}_{\tau}^{n-1}{\mathbf{W}_{q}^{n}}^{\top},\, \widetilde{\mathbf{h}}_{\tau}^{n-1}{\mathbf{W}_{k,E}^{n}}^{\top},\, \widetilde{\mathbf{h}}_{\tau}^{n-1}{\mathbf{W}_{v}^{n}}^{\top} \\ \mathbf{A}_{\tau,i,j}^{n} &= {\mathbf{q}_{\tau,i}^{n}}^{\top}\mathbf{k}_{\tau,j}^{n} + {\mathbf{q}_{\tau,i}^{n}}^{\top}\mathbf{W}_{k,R}^{n}\mathbf{R}_{i-j} + u^{\top}\mathbf{k}_{\tau,j} + v^{\top}\mathbf{W}_{k,R}^{n}\mathbf{R}_{i-j} \\ \mathbf{a}_{\tau}^{n} &= \text{Masked-Softmax}(\mathbf{A}_{\tau}^{n})\,\mathbf{v}_{\tau}^{n} \\ \mathbf{o}_{\tau}^{n} &= \text{LayerNorm}(\text{Linear}(\mathbf{a}_{\tau}^{n}) + \mathbf{h}_{\tau}^{n-1}) \\ \mathbf{h}_{\tau}^{n} &= \text{Positionwise-Feed-Forward}(\mathbf{o}_{\tau}^{n}) \end{aligned}h~τn−1=[SG(mτn−1)∘hτn−1]qτn, kτn, vτn=hτn−1Wqn⊤, h~τn−1Wk,En⊤, h~τn−1Wvn⊤Aτ,i,jn=qτ,in⊤kτ,jn+qτ,in⊤Wk,RnRi−j+u⊤kτ,j+v⊤Wk,RnRi−jaτn=Masked-Softmax(Aτn) vτnoτn=LayerNorm(Linear(aτn)+hτn−1)hτn=Positionwise-Feed-Forward(oτn)\begin{aligned} \widetilde{\mathbf{h}}_{\tau}^{n-1} &= \left[\mathrm{SG}(\mathbf{m}_{\tau}^{n-1}) \circ \mathbf{h}_{\tau}^{n-1}\right] \\ \mathbf{q}_{\tau}^{n},\, \mathbf{k}_{\tau}^{n},\, \mathbf{v}_{\tau}^{n} &= \mathbf{h}_{\tau}^{n-1}{\mathbf{W}_{q}^{n}}^{\top},\, \\ &\quad \widetilde{\mathbf{h}}_{\tau}^{n-1}{\mathbf{W}_{k,E}^{n}}^{\top},\, \\ &\quad \widetilde{\mathbf{h}}_{\tau}^{n-1}{\mathbf{W}_{v}^{n}}^{\top} \\ \mathbf{A}_{\tau,i,j}^{n} &= {\mathbf{q}_{\tau,i}^{n}}^{\top}\mathbf{k}_{\tau,j}^{n} \\ &\quad + {\mathbf{q}_{\tau,i}^{n}}^{\top}\mathbf{W}_{k,R}^{n}\mathbf{R}_{i-j} \\ &\quad + u^{\top}\mathbf{k}_{\tau,j} \\ &\quad + v^{\top}\mathbf{W}_{k,R}^{n}\mathbf{R}_{i-j} \\ \mathbf{a}_{\tau}^{n} &= \text{Masked-Softmax}(\mathbf{A}_{\tau}^{n})\,\mathbf{v}_{\tau}^{n} \\ \mathbf{o}_{\tau}^{n} &= \text{LayerNorm}(\text{Linear}(\mathbf{a}_{\tau}^{n}) \\ &\quad + \mathbf{h}_{\tau}^{n-1}) \\ \mathbf{h}_{\tau}^{n} &= \text{Positionwise-Feed-Forward}(\mathbf{o}_{\tau}^{n}) \end{aligned}
(5)

with hτ0=Esτ\mathbf{h}^0_\tau = \mathbf{E}_{\mathbf{s}_\tau}, the word embeddings of the segment. The first line is the recurrence of (1) with a memory of length MM; the third is (4), written with the projected query and key. The last two lines are a standard post-norm Transformer block: a residual connection and layer normalization after attention, then a feed-forward network, which in the code has its own residual connection and LayerNorm. The released code runs several heads, applies the 1/dhead1/\sqrt{d_{\text{head}}} scale, and offers a pre-norm option that the reported configurations do not use.

The training loop below runs that layer stack over the corpus segment by segment. The memory for each layer is refreshed from its current input and detached, so the next segment reads it as a constant. The data loader (LMOrderedIterator in data_utils.py) splits the corpus into BB contiguous streams, one per batch row, so row bb of each segment continues row bb of the previous one; shuffled segments would give every row a memory from unrelated text.

# Training loop with a memory (WikiText-103 base model: L = M = 150, B = 60).
mems = [empty(0, B, d) for _ in range(n_layer)]   # one cache per layer input
for x, y in segments(corpus, L):          # x, y: [L, B] tokens and next tokens
    h = embed(x)                          # [L, B, d], no position vector added
    new_mems = []
    for n in range(n_layer):
        new_mems.append(cat([mems[n], h])[-M:].detach())  # cache this input
        h = layer[n](h, mem=mems[n])      # attention over [mems[n] ; h], then FFN
    loss = cross_entropy(softmax_head(h), y)   # 150 x 60 targets
    loss.backward()        # reaches the weights through this segment only
    opt.step(); opt.zero_grad()
    mems = new_mems        # the next segment reads these, without gradient

The base model has 151.1M parameters, matching the 151M in the paper's Table 1, but 109.8M of them are the word embedding of the 267,735-word vocabulary, shared with the softmax. The 16 layers hold 41.1M, about 2.57M each; the position projection Wk,R\mathbf{W}_{k,R} adds 168,100 per layer, and uu and vv add 820 numbers to the whole model when shared across layers.

Why evaluation is up to 1,874 times faster

The vanilla model evaluates one token per forward pass: to score token tt it runs the whole window of the CC tokens before it through every layer, then shifts by one and runs the window again. Almost all of that work recomputes states it computed one step earlier. Transformer-XL evaluates in segments (64 or 128 tokens in the released evaluation scripts) and computes every position once; each new query attends to the cached states instead of recomputing them. The evaluation scripts also set same_length, which masks every query to exactly mem_len keys, so each token is scored with the same amount of context. Per scored token, the vanilla model computes C+1C + 1 positions through the network and Transformer-XL computes one, although its attention still reads all CC cached positions.

Figure 5 plays that out on a short strip and plots the paper's measurements underneath. Table 9 reports per-token evaluation time on one GPU (Section 3.2 places the measurement on enwik8) as how many times slower the vanilla model of Al-Rfou et al. is: 363 times at an attention length of 800, 773 at 1,800, 1,409 at 2,800 and 1,874 at 3,800, the length the enwik8 model is evaluated with.

Figure 5 · recomputation versus reuse at evaluation
token 6
6
Top: to score the current token, the vanilla model recomputes its whole context while Transformer-XL reads it from the cache. Play or scrub the tokens, and drag the context length. Bottom: the measured slowdown of the vanilla model in Table 9, with a dashed line at half the attention length.

The measured ratios grow in proportion to the attention length, at 0.43 to 0.50 times the length across the four settings, the scaling the count of recomputed positions predicts. The paper does not break the time down further. The comparison is against the sliding-window protocol, which is the expensive way to evaluate a fixed-context model; a vanilla model evaluated on disjoint segments would be fast but would suffer the fragmentation of Figure 1.

Results on five benchmarks

The paper evaluates on five corpora. WikiText-103 has 103M training tokens from 28K Wikipedia articles, 3.6K tokens per article on average. enwik8 is 100M bytes of unprocessed Wikipedia text, markup included; text8 is 100M characters of Wikipedia reduced to the 26 letters and space. One Billion Word has its sentences shuffled, so it contains no dependency longer than a sentence. Penn Treebank has only 1M training tokens. The WikiText-103 models use adaptive softmax and adaptive input representations for the 267,735-word vocabulary: words are grouped into clusters by frequency so that most of the softmax cost falls on the small cluster of frequent words, and the large model's script also gives the rarer clusters smaller embedding vectors.

Benchmark (metric)Previous bestTransformer-XLParams
WikiText-103 (ppl)20.518.3257M
enwik8 (bpc)1.060.99277M
text8 (bpc)1.131.08277M
One Billion Word (ppl)23.721.80.8B
Penn Treebank (ppl)55.354.5224M

On WikiText-103 the previous best, 20.5, was the adaptive-input Transformer of Baevski and Auli, a contemporary paper, with 247M parameters. At equal size the gap is wider: the 151M standard Transformer-XL reaches 24.0 against 33.0 for a 151M QRNN. On enwik8, the 24-layer model is the first to go below 1.0 bits per character on a widely used character benchmark; it is evaluated with a memory of 3,800. A 12-layer Transformer-XL with 41M parameters reaches 1.06, the score of Al-Rfou et al.'s 64-layer, 235M-parameter model, with 17% of the parameters and without that model's auxiliary losses; Al-Rfou's 12-layer model scores 1.11. The text8 model reuses the enwik8 hyper-parameters unchanged.

One Billion Word tests the claim that the method helps beyond long dependencies. Its sentences are shuffled, so there is little to gain from a long memory, and Transformer-XL still improves the best single model from 23.7 to 21.8. On Penn Treebank, with the variational dropout and weight averaging of AWD-LSTM, it reaches 54.52, the best result without two-step finetuning; a mixture-of-softmaxes model with finetuning reports 54.44.

What the ablations show

The ablation on WikiText-103 (Table 6) uses a 128M-parameter model trained with a backpropagation length of 128, and reports two perplexities per variant: evaluated with the training attention length, and at the shortest evaluation attention length that gives the best result. Two terms need defining. A "half" loss applies the cross-entropy only to the second half of the positions in a segment, skipping the positions with very little context; a "full" loss uses all of them. The encodings compared are the paper's, Shaw et al.'s (relative, learned), the original sinusoid of Vaswani et al. (absolute) and Al-Rfou et al.'s learned per-layer embeddings (absolute). Click a row of Figure 6 to read it.

Figure 6 · the WikiText-103 ablation
Perplexity of nine 128M-parameter variants from Table 6, at the training attention length and at the best evaluation length, which is listed on the right. ✓ marks recurrence. Click a row.

The full model reaches 27.02 at the training length and 26.77 when evaluated with an attention length of 500. Recurrence with Shaw et al.'s encoding reaches 27.94 at its training length, and a longer evaluation context does not improve it (256 is the shortest length that reaches 27.94). Without recurrence, the best variant is 29.02, and the absolute encodings appear only with a half loss, because the paper found they work only when the shortest contexts are excluded from training. The setting closest to Al-Rfou et al. (without their auxiliary losses) is last, at 31.16. In the larger 151M model, perplexity keeps falling as the evaluation attention length grows: 23.43 at the training length, 23.35 at 300, 23.16 at 450 and 23.09 at 640. The paper gives the backpropagation length of these runs as 128; the released script for the 151M base model uses 150.

Recurrence also costs memory, so Appendix A repeats the comparison at equal GPU memory: Transformer-XL with backpropagation length 128 reaches 26.77, and the best model without recurrence, which can afford a length of 176, reaches 27.98. On One Billion Word (Table 7), where long context should not matter, removing recurrence moves perplexity from 25.2 to 27.1, and Shaw et al.'s encoding gives 25.7. The paper attributes the recurrence gain there to removing context fragmentation, since only the first tokens of each segment lack context in the vanilla model.

To measure how much context a model uses, the paper defines the relative effective context length (RECL). It extends Khandelwal et al.'s effective context length, the longest context whose extension still improves the loss by more than a threshold, by comparing a group of models against a shared baseline. For each token the baseline is the lowest loss any model in the group achieves with a short context cc, and a model only gets credit for doing better than that baseline with a longer context c′c'. With fi(c,c′)f_i(c, c') the average of the smaller of the baseline and model ii's loss at c′c', the relative gain is the relative drop in perplexity,

gi(c,c′)=exp⁡fi(c,c)−exp⁡fi(c,c′)exp⁡fi(c,c)g_i(c, c') = \frac{\exp f_i(c, c) - \exp f_i(c, c')}{\exp f_i(c, c)}

and RECL is the length at which extending the context by one more step stops gaining 1%. The average can be restricted to the fraction rr of tokens with the highest baseline loss. With r=0.1r = 0.1, the 151M Transformer-XL reaches 900 words, against 500 for a QRNN and 400 for an LSTM in the same group; in the 128M group, Transformer-XL reaches 700, without recurrence 300, with Shaw et al.'s encoding 400, and the vanilla Transformer 128. The abstract's "80% longer than RNNs" is 900 against 500, and "450% longer than vanilla Transformers" is 700 against 128, a factor of 5.47 within the second group.

Limits

The memory makes information from far back available, but training never optimizes what gets stored: no gradient crosses a segment boundary, so a cached state holds whatever helped predictions within its own segment. The measured RECL of 900 words for the 151M model is far below its 10,240-token reach at evaluation.

The memory costs M×dM \times d numbers per layer per sequence, and every query attends to M+LM + L keys, so attention per segment costs L(M+L)L(M + L) scores per head. Training uses M=LM = L; the long memories only appear at evaluation, where nothing is backpropagated. The evaluation code also clamps every distance above 1,000 to the encoding of 1,000 for the large models, so the reported 1,600- and 3,800-token memories do not test whether the sinusoid separates distances beyond 1,000.

The results are perplexity and bits per character. The text samples in Appendix E, 500 to 1,000 tokens generated from WikiText-103 test-set contexts with top-40 sampling, stay on topic and keep the section structure of a Wikipedia article, but they are uncurated examples, not a measurement.

Transformer-XL became a backbone for later work. XLNet, from an overlapping group of authors, carries its recurrence and relative encoding into pretraining. The Compressive Transformer (Rae et al., 2019) adds a second, compressed memory for states that fall out of the first, and reports 17.1 perplexity on WikiText-103 and 0.97 bits per character on enwik8. Putting position into the attention score instead of the input is now the common choice: T5 adds a learned bias per bucket of distances, and rotary embeddings rotate queries and keys so that their dot product depends on the offset between them.

Provenance Verified against primary literatureHow we verify
Dai et al. (2019), arXiv v3Every equation on this page (the paper numbers none of them; the numbers here are this page's), Sections 3.1-3.3, Tables 1-10, Appendix B (relative shift), Appendix C (RECL), Appendix D (attention visualisation). Read from the arXiv HTML render.
kimiyoung/transformer-xl, pytorch/mem_transformer.pyPositionalEmbedding (lines 15-31): inv_freq = 1/10000^(2i/d), output cat[sin, cos]. RelMultiHeadAttn (lines 142-161): scale = 1/sqrt(d_head). _rel_shift (lines 194-207): pad, view, drop first row, view. RelPartialLearnableMultiHeadAttn.forward (lines 218-258): queries from the last qlen rows of cat(mems, w); AC = (q + r_w_bias) k; BD = rel_shift((q + r_r_bias) r). Lines 586-588: r_w_bias and r_r_bias have shape (n_head, d_head), one pair shared by all layers. The TensorFlow scripts for WikiText-103 (base and large) and One Billion Word pass untie_r=True (one pair per layer); the enwik8 and text8 scripts do not. _update_mems (lines 619-640): under no_grad, keep the last mem_len rows of cat(mems, hids), detached. Lines 649-668: causal mask triu(1 + mlen), optional same_length mask, pos_seq = klen-1 ... 0, clamp_len.
kimiyoung/transformer-xl, run scriptspytorch/run_wt103_base.sh: 16 layers, d_model 410, 10 heads of 41, d_inner 2100, tgt_len = mem_len = 150, batch 60; evaluation tgt_len 64, mem_len 640, clamp_len 400. run_wt103_large.sh and tf/scripts/wt103_large_tpu.sh: 18 layers, d_model 1024, 16 heads of 64, d_inner 4096, 384/384; evaluation 128/1600, clamp_len 1000. enwik8 large (PyTorch and TF): 24 layers, d_model 1024, 8 heads of 128, d_inner 3072, 768/768; evaluation 128/3800, clamp_len 1000. pytorch/data_utils.py, LMOrderedIterator: data.view(bsz, -1).t(), so each batch row is one contiguous stream.
Parameter count, reproducedThe base configuration gives 151.1M: tied embedding and softmax 267,735 x 410 = 109.8M, 16 layers x 2,566,650 = 41.1M (q, k, v, output and position projections, feed-forward, two LayerNorms), plus 0.27M of softmax biases and cluster weights. Table 1 lists 151M.
Al-Rfou et al. (2018)The vanilla baseline: 64 layers, inputs of 512 characters, a learned absolute position embedding added before every layer, auxiliary losses at intermediate layers and positions. At evaluation only the last position is scored and the context is processed from scratch for every character, with no state passed between predictions. T12 has 41M parameters (44M including the training-only heads).
Shaw, Uszkoreit & Vaswani (2018)Relative position representations a_ij^K = w^K_clip(j-i, k), learned, 2k + 1 labels, k = 16 in their translation models; also a value-side term a_ij^V that Transformer-XL does not use.
Vaswani et al. (2017)Sinusoidal encoding PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(...), interleaved; the Transformer-XL code concatenates the sin and cos halves instead, a fixed permutation of the same features.
Khandelwal et al. (2018)An LSTM language model uses about 200 tokens of context on average, and ignores word order beyond about 50, the figure the paper cites in its introduction.
This page's checksThe relative shift was checked exhaustively for L = 1-6 and M = 0-6 with a script: after the shift, every unmasked cell (i, j) holds the score for distance M + i - j from row i. The 80% and 450% RECL figures were recomputed from Table 8 (900/500 = 1.8; 700/128 = 5.47).
correctionOne printing error and several scope notes. (1) Appendix B, matrix B: in the last row (query L-1), the two entries around the middle are printed as R_{M+L-1} and R_{L-1}; at key positions M and M+1 they should be R_{L-1} and R_{L-2}. The Q-indexed form printed directly beneath it is correct, and it is what this page teaches. (2) The enwik8 text gives a training attention length of 784; the released TensorFlow and PyTorch enwik8 scripts use tgt_len = mem_len = 768. (3) Appendix D calls the visualised model "the SoTA model" with 16 layers of 10 heads and a memory of 640. That is the configuration of the 151M standard model (24.0 test perplexity in Table 1), whose released evaluation script uses mem_len 640, not of the 18-layer 257M model behind the 18.3 result. (4) The abstract's "80% longer than RNNs and 450% longer than vanilla Transformers" draws on Table 8's two separate model groups: 900 vs 500 tokens (151M Transformer-XL vs QRNN) and 700 vs 128 (128M Transformer-XL vs the 128M Transformer, 5.47 times, rounded to 450% longer). (5) The 1,874-times speedup is against the sliding-window evaluation of Al-Rfou et al., which recomputes the whole window for every token; it is not a comparison with a Transformer evaluated on disjoint segments. (6) The equations omit the 1/sqrt(d) scale that the code applies, and at evaluation the code clamps every distance above clamp_len (1000 for the large models, 400 for the base model) to that value, so the reported long attention lengths run with all distances beyond 1000 sharing one encoding.

Questions you might still have

?

Is the memory the same thing as a KV cache?
No. A KV cache stores each layer's keys and values; Transformer-XL stores each layer's input hidden states and recomputes keys and values from them for every segment, with the current weights. During training the weights change between segments, so cached keys would be stale. The memory also has a fixed length M per layer: when a segment is appended, the oldest positions drop out. At evaluation the weights are fixed, so content keys could be cached instead; the released code does not do that.

?

Does any gradient flow into the previous segment?
No. The cached states enter through a stop-gradient, so the loss on segment tau reaches the weights only through the computation of segment tau. The weights still learn to read the memory, because the keys and values computed from it depend on W_k and W_v, which receive gradient. What never gets a gradient is the computation that wrote a state, so nothing trains an earlier segment to store information a later one needs.

?

Why not just train on longer segments?
Attention over a segment of length L costs time and activation memory proportional to L squared, and backpropagation has to keep all of it. The cache adds M keys per query without storing any activations for backpropagation. The paper also compares at equal GPU memory (Table 10): Transformer-XL with backpropagation length 128 plus recurrence reaches 26.77 perplexity, against 27.98 for a model without recurrence that can afford a backpropagation length of 176.

?

Could absolute positions work if the memory and segment were numbered 1 to M + L?
Not with cached states. Each cached state was computed when its token had a different absolute position, and in a deep model the position vector added at the input is mixed into every later layer's state. Renumbering the tokens does not change those stored vectors; recomputing them is the cost the cache exists to avoid. With relative encodings the stored states never contain a position, and the distance is computed fresh inside each attention score.

?

What are u and v, and why do they not depend on the query's position?
They are two learned vectors per head (shared across layers, or one pair per layer when the TensorFlow flag untie_r is set) that replace the query's own position term U_i W_q in the expanded attention score. u multiplies the content key, so it adds the same score for a given key token whatever the query is; v multiplies the position key, so it adds the same score for a given distance whatever the content is. If only distances matter, the query's absolute position should not change either term, so a single learned vector replaces a position-dependent one.

?

How big is the memory in practice?
For the WikiText-103 base model in training, 16 layer inputs of 150 positions by 410 numbers, 984,000 numbers per sequence, about 3.9 MB in 32-bit floats. At evaluation the memory is 640 positions, about 16.8 MB per sequence. The large models evaluate with 1,600 positions of 1,024 numbers in each of 18 layers (WikiText-103) and 3,800 positions in each of 24 layers (enwik8).

?

How does this relate to sliding-window attention and later position schemes?
Sliding-window attention as in Mistral 7B uses the same per-layer growth: each layer reaches one window further back, so N layers reach about N times the window. Later models also put position into the attention score instead of the input: T5 adds a learned scalar per bucket of distances, and rotary embeddings (RoPE) rotate queries and keys so their dot product depends only on the offset. The Mistral, T5 and RoPE explainers cover those.

Footnotes & further reading

  1. The paper: Dai, Yang, Yang, Carbonell, Le & Salakhutdinov, Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (ACL 2019; arXiv January 2019). Code, pretrained models and hyper-parameters: kimiyoung/transformer-xl, in TensorFlow and PyTorch.
  2. The vanilla baseline: Al-Rfou, Choe, Constant, Guo & Jones, Character-Level Language Modeling with Deeper Self-Attention (AAAI 2019). LSTM context use: Khandelwal, He, Qi & Jurafsky, Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (ACL 2018).
  3. Positional encodings: Vaswani et al., Attention Is All You Need (NeurIPS 2017), covered in the Transformer explainer; Shaw, Uszkoreit & Vaswani, Self-Attention with Relative Position Representations (NAACL 2018); Huang et al., Music Transformer (ICLR 2019), the other relative scheme the paper cites.
  4. The WikiText-103 previous best: Baevski & Auli, Adaptive Input Representations for Neural Language Modeling (ICLR 2019).
  5. Follow-ups: Yang et al., XLNet: Generalized Autoregressive Pretraining for Language Understanding (NeurIPS 2019); Rae, Potapenko, Jayakumar & Lillicrap, Compressive Transformers for Long-Range Sequence Modelling (ICLR 2020). The same per-layer growth of reach in sliding-window attention is in the Mistral 7B explainer.