VerifiedarXiv:2112.0442630 min
LLMs · Retrieval

RETRO: Improving language models by retrieving from trillions of tokens

A 7.5-billion-parameter language model looks up text similar to each 64-token chunk of its input in a database of 1.75 trillion tokens, and predicts text better than a 178-billion-parameter model on 10 of 16 parts of the Pile benchmark.

DeepMind's Retrieval-Enhanced Transformer. A frozen BERT model finds the stored chunks nearest to each chunk of the input, a small encoder reads them, and a cross-attention layer lets every token use the text retrieved for the chunk before its own, so the model still predicts one token at a time from the past.

Explaining the paperImproving language models by retrieving from trillions of tokensBorgeaud, Mensch, Hoffmann, Cai, Rutherford, et al. (DeepMind) · ICML 2022 · arXiv:2112.04426 ↗

A model that can read its training data at test time also scores well on any test text that leaked into that data. The paper builds an evaluation that removes leaked chunks and reports how much of the gain is left.

What RETRO adds to a language model

A transformer language model keeps everything it has learned in its weights. Making it bigger buys two things, and the paper separates them: more computation per token, and more capacity to memorize the training text. RETRO (Retrieval-Enhanced Transformer) tries to get the second without the first. It keeps a 7.5B-parameter model and gives it a database of text, which it searches while it reads and while it writes.

The unit of lookup is a chunk of 64 tokens. After each chunk of input, the model embeds that chunk with a frozen BERT, finds the nearest stored chunks in a database of 28 billion of them, and fetches each one together with the 64 tokens that followed it in its source document. A 2-layer encoder reads the fetched text, and cross-attention layers in the decoder let the next 64 tokens use it. At evaluation the database is a multilingual version of the MassiveText corpus from the Gopher paper, about 1.75 trillion tokens of web pages, books, news, Wikipedia and GitHub code.

Results are reported in bits per byte: the model's total negative log-likelihood of a document, in bits, divided by the document's length in UTF-8 bytes. Lower is better, and unlike perplexity the number does not depend on the tokenizer. On the C4 web text corpus the 7.5B RETRO reaches 0.66 bits per byte. A 7B transformer trained the same way without retrieval reaches 0.78, so retrieval removes 15% of the bits needed to encode the text. The smallest RETRO, 172M parameters, reaches 0.82, which is better than the baseline in the 1.5B column of the paper's Table 14 (0.84, a 1.3B-parameter model). On the Pile, a benchmark of 16 text domains used here, RETRO beats its own baseline on all 16, the 178B-parameter Jurassic-1 on 10 and the 280B-parameter Gopher on 9.

Four pieces make it work, and the page follows them in order: a likelihood that splits the sequence into chunks, a retrieval step with a frozen BERT, a chunked cross-attention layer that keeps the model autoregressive, and an evaluation that measures how much of the gain comes from copying test text that is also in the database. Sections on training, results, retrofitting of existing models and limits follow those four.

Chunks and the retrieval-enhanced likelihood

A training example is n=2048n = 2048 tokens X=(x1,…,xn)X = (x_1, \dots, x_n), split into l=32l = 32 consecutive chunks C1,…,ClC_1, \dots, C_l of m=64m = 64 tokens each. Every chunk CuC_u gets a set Ret(Cu)\mathrm{Ret}(C_u) of kk neighbors from the database D\mathcal{D}, computed by a fixed procedure with no trainable weights. The model, with weights θ\theta, is trained to maximize this log-likelihood:

L(X∣θ,D)≜∑u=1l∑i=1mℓθ(x(u−1)m+i∣(xj)j<(u−1)m+i, (Ret(Cu′))u′<u)L(X\mid\theta,\mathcal{D})\triangleq\sum_{u=1}^{l}\sum_{i=1}^{m}\ell_{\theta}\bigl(x_{(u-1)m+i}\mid(x_{j})_{j<(u-1)m+i},\,(\mathrm{Ret}(C_{u'}))_{u'<u}\bigr)L(X∣θ,D)≜∑u=1l∑i=1mℓθ(x(u−1)m+i∣(xj)j<(u−1)m+i, (Ret(Cu′))u′<u)\begin{aligned}L(X\mid\theta,\mathcal{D})&\triangleq\sum_{u=1}^{l}\sum_{i=1}^{m}\ell_{\theta}\bigl(x_{(u-1)m+i}\mid\\&\qquad(x_{j})_{j<(u-1)m+i},\,(\mathrm{Ret}(C_{u'}))_{u'<u}\bigr)\end{aligned}
(1)

The term ℓθ\ell_\theta is the model's log-probability of one token. Token ii of chunk uu sits at position (u−1)m+i(u-1)m+i, and its probability may depend on every earlier token and on the neighbors of every earlier chunk, u′<uu' < u, but never on the neighbors of its own chunk. The first chunk has no earlier chunk, so the paper sets Ret(C1)=∅\mathrm{Ret}(C_1) = \emptyset, and the first 64 tokens are predicted exactly as a plain language model would predict them.

Take token 100. It is token 36 of chunk 2, which covers positions 65 to 128. Its probability conditions on tokens 1 to 99 and on Ret(C1)\mathrm{Ret}(C_1), the neighbors found for positions 1 to 64. It cannot condition on Ret(C2)\mathrm{Ret}(C_2) because the lookup for chunk 2 is computed from all 64 of its tokens, including tokens 100 to 128, which the model has not produced yet when it predicts token 100. Equation (1) is therefore an ordinary next-token likelihood with extra inputs that only ever come from the past, and it can be sampled one token at a time like any autoregressive model: write a chunk, look up its neighbors, write the next chunk.

Why chunks? Earlier retrieval language models worked at one of two granularities. Token-level methods such as kNN-LM store one key per token of the database, which at 1.75 trillion tokens means 1.75 trillion keys. Prompt-level methods such as RAG and REALM retrieve once for the whole input, which is fine for a question but not for a 2,048-token document whose topic moves. Chunks give 64 times fewer keys than tokens, and a fresh lookup every 64 tokens.

How RETRO retrieves neighbors from the database

The database is a key-value store. Each value is two consecutive 64-token chunks from one document, written [N,F][N, F]: the neighbor chunk NN and its continuation FF, the 64 tokens that came after it. Each key is computed from NN alone: BERT outputs one vector per token, and averaging those vectors over the 64 tokens of NN gives one vector per chunk, written Bert(N)\mathrm{Bert}(N). For a query chunk CC, the distance to a stored chunk is the squared Euclidean distance between the two averages,

d(C,N)=∥Bert(C)−Bert(N)∥22d(C,N)=\lVert\mathrm{Bert}(C)-\mathrm{Bert}(N)\rVert_{2}^{2}

and the model receives the values of the kk nearest keys:

Ret(C)≜([N1,F1],…,[Nk,Fk])\mathrm{Ret}(C)\triangleq([N^{1},F^{1}],\dots,[N^{k},F^{k}])

Each value is r=128r = 128 tokens, so Ret(C)\mathrm{Ret}(C) is a k×128k \times 128 block of token ids. The model is trained with k=2k = 2 and can be evaluated with more. The continuation FF matters more for prediction: NN resembles the chunk just read, so FF is what followed similar text somewhere else, which is a guess at the chunk about to be written. The ablations measure the split: continuations alone recover 56% of the gain from retrieval, neighbor chunks alone 22%.

The BERT is frozen: its weights never change, so every key can be computed once and stored. A trainable retriever, as in REALM, would change its embeddings as it learns and force the whole database to be re-embedded every so often, which at 28 billion chunks means running BERT over the full 1.75 trillion tokens. Freezing also lets the paper precompute the neighbors of every training chunk before training starts. The paper reports that retrieving on the fly could not keep up with the training computation, so the neighbor ids are stored as part of the training data, like the tokens.

During training, the database contains the training documents themselves, which creates a leak. The nearest stored chunk to CuC_u would usually be CuC_u itself, taken from the same document, and its continuation FF would be Cu+1C_{u+1}, the exact tokens the model is being trained to predict next. The paper therefore discards any neighbor that comes from the same document as the training sequence XX.

The search is approximate. The ScaNN library (Guo et al., 2020) returns approximate nearest neighbors in time that grows with the logarithm of the database size, and the paper reports 10 milliseconds per query on the 2-trillion-token database. One query serves 64 tokens, about 0.16 milliseconds per token. When the search is restricted to Wikipedia, the retrieved articles are typically 2 to 3 links away from the query's article in the Wikipedia link graph, against more than 5 for random pairs of articles.

There are two databases. For training, 600 billion tokens are drawn from MassiveText in the same proportions as the training mixture. For evaluation, the database is the union of all of MassiveText except books, of which a 4% sample is kept: 1.75 trillion tokens, and 28 billion keys at 64 tokens per key. To limit leakage from training into test sets, any training document with 13-gram Jaccard similarity of 0.8 or more to a validation or test document is removed (estimated with MinHash, a hashing scheme that approximates set overlap without comparing every pair), as are the Wikitext103 validation and test articles.

Chunk-level keys keep the index small. kNN-LM stores a 1,024-number vector for every token: for the 4 billion tokens of Wikipedia that is 4 billion × 1,024 × 4 bytes = 16.4 terabytes, which the paper rounds to 15. RETRO's index for the same Wikipedia is 215 gigabytes, and for all 1.75 trillion tokens of MassiveText it is 93 terabytes: about six times the kNN-LM Wikipedia index, for roughly 440 times as many tokens.

How the RETRO architecture works

RETRO is an encoder-decoder transformer. The decoder is the language model: a GPT-2-style transformer with RMSNorm in place of LayerNorm and with the relative position encodings of Transformer-XL. The encoder reads retrieved text. The decoder is a stack of two kinds of block, built from three residual operators that each map an n×dn \times d activation HH to another: a feed-forward layer Ffw, causal self-attention Attn, and the chunked cross-attention Cca that reads the encoded neighbors EE.

Retro(H,E)≜Ffw(Cca(Attn(H),E)),Lm(H)≜Ffw(Attn(H))\mathrm{Retro}(H,E)\triangleq\mathrm{Ffw}(\mathrm{Cca}(\mathrm{Attn}(H),E)),\qquad\mathrm{Lm}(H)\triangleq\mathrm{Ffw}(\mathrm{Attn}(H))Retro(H,E)≜Ffw(Cca(Attn(H),E)),Lm(H)≜Ffw(Attn(H))\begin{gathered}\mathrm{Retro}(H,E)\triangleq\mathrm{Ffw}(\mathrm{Cca}(\mathrm{Attn}(H),E)),\\\mathrm{Lm}(H)\triangleq\mathrm{Ffw}(\mathrm{Attn}(H))\end{gathered}
(2)

A Retro block is an ordinary transformer block with one cross-attention layer inserted between self-attention and the feed-forward layer. The set PP of layers that use it is a hyperparameter; the paper uses every third layer starting at layer 6, so the 12-layer 172M model has Retro blocks at layers 6, 9 and 12. Each operator includes its residual connection, so a cross-attention layer that outputs zero leaves the activations unchanged.

The encoder is a bidirectional transformer of 2 layers and width d′=896d' = 896 in every model size. It runs on each retrieved neighbor separately, with weights shared across all neighbors and all chunks, and it is conditioned on the chunk that retrieved the neighbor: its one cross-attention layer lets the neighbor's 128 tokens attend to HuH_u, the decoder's activations for the 64 tokens of chunk CuC_u. Those activations are taken at the first Retro layer, min⁡P\min P, just after its self-attention. The encoder therefore runs once per forward pass, partway up the decoder, and its output EE is reused by every Retro block above that point.

For one training sequence of the 172M model, EE has shape l×k×r×d′l \times k \times r \times d' = 32 × 2 × 128 × 896. The encoder processes 32 × 2 × 128 = 8,192 retrieved tokens per 2,048-token sequence, four times as many tokens as the decoder, which is affordable because it has 2 layers against the decoder's 12 (32 in the 7.5B model). Figure 1 shows the stack with the shape at each step.

Figure 1 · the 172M RETRO, layer by layer
Layer 6 = min(P): Attn, then the encoder runs on H [32, 64, 896];Cca: 31 attending chunks x 64 rows, each over 2 x 128 = 256 rows of E; Ffw.
Decoder layers on the left, the neighbor encoder on the right. Click a layer for the operations it runs and the tensor shapes for one 2,048-token sequence. The toggle switches between training from scratch, where every weight is trained, and retrofitting, where the pretrained decoder is frozen and only the teal encoder and cross-attention weights are trained.

The figure's two parameter counts come from Table 2 of the paper, which excludes embedding matrices. The four baselines have 132M, 368M, 1,309M and 6,982M parameters; the matching RETRO models have 172M, 425M, 1,451M and 7,532M, overheads of 30%, 15%, 11% and 8%. The encoder is the same size for all of them, which is why the relative overhead falls as the decoder grows. For the smallest model the paper attributes about 19M to the encoder and 12M to the cross-attention layers (three in the decoder, one in the encoder).

Chunked cross-attention

Chunked cross-attention is the operator that lets the decoder use EE without breaking equation (1). It splits the n×dn \times d activation into l−1l - 1 attending chunks Hu+H_u^+, shifted one position earlier than the data chunks: Hu+H_u^+ holds positions umum to um+m−1um + m - 1, which are the last token of CuC_u and the first m−1m - 1 tokens of Cu+1C_{u+1}. Each attending chunk cross-attends to EuE_u, the encoded neighbors of CuC_u. For every chunk uu and every ii in 1 to mm:

Cca(H,E)um+i−1≜Ca(hum+i−1,Eu)\mathrm{Cca}(H,E)_{um+i-1}\triangleq\mathrm{Ca}(h_{um+i-1},E_{u})
(3)

The shift comes from how a transformer predicts. The output at position jj is used to predict token j+1j + 1. Position umum is the first position that has read all of CuC_u, so it is the first that may use Ret(Cu)\mathrm{Ret}(C_u), and its output predicts xum+1x_{um+1}, the first token of Cu+1C_{u+1}, which equation (1) allows to depend on Ret(Cu)\mathrm{Ret}(C_u). Position um+m−1um + m - 1 predicts the last token of Cu+1C_{u+1}, still allowed. Position (u+1)m(u+1)m has read all of Cu+1C_{u+1} and moves on to Eu+1E_{u+1}.

Two edge cases complete the definition. Positions 1 to m−1m - 1 have no completed chunk before them, so Cca is the identity there. The last position, lmlm, attends to ElE_l; during training its output predicts nothing inside the sequence, and during sampling it predicts the first token of the next chunk.

Ca is ordinary cross-attention. In its simplest form, with query, key and value matrices Q,K∈Rd×cQ, K \in \mathbb{R}^{d \times c} and V∈Rd×dV \in \mathbb{R}^{d \times d}, for one activation h∈Rdh \in \mathbb{R}^d and TT attended rows Y∈RT×dY \in \mathbb{R}^{T \times d}:

Ca(h,Y)≜softmax(YKQTh) YV\mathrm{Ca}(h,Y)\triangleq\mathrm{softmax}(YKQ^{T}h)\,YV
(4)

YKYK are the keys of the attended rows, QThQ^T h is the query, their product is one score per row, the softmax turns the scores into weights, and the output is the weighted sum of the value rows YVYV. The real model uses several heads, and the encoder width d′=896d' = 896 differs from dd in every model but the smallest, so in practice the key and value matrices take d′d'-dimensional rows; (4) writes the single-width case. The attended rows are the encoded tokens of all kk neighbors of CuC_u together: EuE_u has shape k×r×d′k \times r \times d', and the neighbor and time dimensions are merged into T=krT = kr rows before one softmax. With k=2k = 2 that is 256 rows, and the two neighbors compete for the same attention weights.

Each cross-attention reads only the neighbors of one chunk, yet a token can depend on all earlier neighbor sets. In layer 6, the first 63 tokens of C3C_3 attend to E2E_2. In layer 7, self-attention lets a token of C4C_4 read those C3C_3 activations, which already carry information from E2E_2, while its own cross-attention in layer 9 reads E3E_3. The paper notes that this gives the dependence on every Ret(Cu′)\mathrm{Ret}(C_{u'}) with u′<uu' < u in equation (1) without the cost of attending to all of them. One Cca layer computes n×krn \times kr attention scores, 2,048 × 256 = 524,288 for a training sequence. Attending instead to every earlier chunk's neighbors would make attending chunk uu read uu neighbor sets rather than one, 16 times as many on average over the 31 attending chunks. That count grows with the square of the sequence length, while the cost of Cca grows only linearly with the amount retrieved.

Figure 2 · which retrieved text each position reads
p = 6
The toy sizes of Figure 2 in the paper: n=12n = 12, m=4m = 4, l=3l = 3, k=2k = 2. Drag the slider to pick the position whose output is computed. The amber block is the neighbor set its cross-attention reads directly, dim amber blocks reach it through self-attention from earlier tokens, and the outlined token is the one its output predicts. Toggle the brackets to compare data chunks with attending chunks.

Slide to position 3 and nothing is retrieved: positions 1 to 3 come before the first completed chunk. Position 4 is the first to read Ret(C1)\mathrm{Ret}(C_1), and it predicts token 5. Positions 4 to 7 all read Ret(C1)\mathrm{Ret}(C_1), and at position 8 the direct source switches to Ret(C2)\mathrm{Ret}(C_2). Position 12 reads Ret(C3)\mathrm{Ret}(C_3) to predict a token beyond the sequence.

Cross-attention scores also get a relative position term, because a neighbor is expected to line up with the chunk that retrieved it: token i′i' of NN is treated as if it sat at the same offset as token i′i' of CuC_u. Token ii of the attending chunk Hu+H_u^+ is at absolute position um+i−1um + i - 1, and token i′i' of the neighbor is aligned with position (u−1)m+i′(u-1)m + i', so their distance is

d(i,i′)≜i−i′+m−1d(i,i')\triangleq i-i'+m-1
(6)

with ii in 1 to mm and i′i' in 1 to 2m2m (the FF half of a value continues past the chunk). The paper prints this equation with ll, its symbol for the number of chunks, in place of mm; the Provenance panel records the slip. In the encoder's cross-attention, where neighbor token i′i' attends to token ii of CuC_u itself (no shift), the distance is denc(i′,i)≜i′−id_{\mathrm{enc}}(i',i)\triangleq i'-i, equation (7). The positional logits are a learned linear map of a cosine encoding of these distances, added to the content scores. Without them the ablation model ends at 0.826 bits per byte instead of 0.822.

The paper's own implementation of Cca (Listing 1, JAX) is short. Rewritten with the 172M model's sizes, and with the multi-head and positional terms left out as in the original:

# Listing 1 of the paper (Appendix B.1.3), one head, no positional logits.
# Sizes of the 172M model: n = 2048, m = 64, l = 32, k = 2, r = 128, d = 896.
def cross_attention(chunk, neighbours):           # [m, d], [k, r, d]
    q = chunk @ Q                                 # [m, d]
    keys = neighbours @ K                         # [k, r, d]
    logits = einsum("md,krd->mkr", q, keys)       # [64, 2, 128]
    logits = logits.reshape(m, k * r)             # one softmax over 256 tokens
    values = (neighbours @ V).reshape(k * r, d)   # [256, d]
    return softmax(logits) @ values               # [64, d]

def chunked_cross_attention(H, E):                # H [n, d], E [l, k, r, d]
    # attending chunk u (1-indexed) = positions u*m .. u*m + m - 1:
    # the last token of C_u and the first m - 1 tokens of C_{u+1}
    att = pad(H[m - 1:], ((0, m - 1), (0, 0))).reshape(l, m, d)
    out = stack([cross_attention(att[u], E[u]) for u in range(l)])
    # shift back: positions 1 .. m - 1 get 0, so the residual add leaves them as is
    return pad(out.reshape(n, d), ((m - 1, 0), (0, 0)))[:n]

The padding does the alignment. Dropping the first m−1m - 1 rows and reshaping makes row 0 of attending chunk uu the last token of CuC_u; after the per-chunk attention, the output is shifted back down by m−1m - 1 rows with zeros in front, and the zeros become the identity once the residual is added.

Generating text with RETRO

Sampling interleaves generation and lookup. The prompt is the first chunk. When a chunk is complete, its 64 tokens are embedded with BERT, ScaNN returns the neighbors, the encoder encodes them (conditioned on the chunk's activations), and the next 64 tokens are sampled one at a time while their cross-attention reads that encoding. The cost of sampling stays quadratic in the length of the text, as for any transformer, and the retrieval adds a cost linear in the number of chunks, which the paper calls negligible next to sampling the tokens.

The paper's samples show what retrieval changes. Given the opening of Hamlet, the 7.5B model with retrieval continues with the actual text of the play, robust to formatting differences between the prompt and the retrieved copies, while the same model with retrieval switched off writes plausible but invented dialogue and ends by repeating one character's line (Table 7). Prompted with the first digits of π, the model with retrieval copies a long run of correct digits from its neighbors, and without retrieval it diverges two digits after the prompt (Table 21).

Training setup and ablations

Every model trains for 419,430,400,000 tokens, which is 204.8 million sequences of 2,048 tokens, with a batch of 256 sequences (1,024 for the largest model), so 800,000 steps for the three smaller models and 200,000 for the largest. The optimizer is AdamW (the decoupled weight decay variant of Adam) with weight decay 0.1; the learning rate warms up linearly from 10−710^{-7} over 750 steps to a maximum of 2×10−42 \times 10^{-4} (10−410^{-4} for the largest model), then follows a cosine decay to a tenth of the maximum. Optimizer state is sharded with ZeRO. Retrieval during training uses k=2k = 2 neighbors from the 600B-token training database. The tokenizer is SentencePiece with a 128,000-token vocabulary. One training step for the 172M model, in pseudocode:

# one training sequence for the 172M RETRO (Sections 2.2 to 2.5, Algorithm 1)
X = sample_sequence()              # [2048] token ids, vocabulary 128,000
nbrs = neighbours[X]               # [32, 2, 128] token ids, computed OFFLINE:
                                   #   chunk u -> 2 nearest [N, F] by frozen BERT,
                                   #   same-document neighbours removed
H = emb(X)                         # [2048, 896]
for p in range(1, 13):             # 12 decoder layers; each op adds its residual
    H = attn(H)                    # causal self-attention
    if p == 6:                     # min(P): the encoder runs once, here
        E = encoder(nbrs, H.reshape(32, 64, 896))
                                   # [32, 2, 128, 896]: 2 bidirectional layers,
                                   # d' = 896, cross-attends to its chunk's H_u
    if p in (6, 9, 12):            # P: the Retro blocks
        H = chunked_cross_attention(H, E)
    H = ffw(H)
logits = read(H)                   # [2048, 128000]
loss = cross_entropy(logits[:-1], X[1:])
# outputs at positions 1..63 never saw retrieval; position 64 predicts token 65
# from Ret(C_1). Gradients reach emb, all decoder layers, the encoder and the
# cross-attention weights. BERT and the neighbour indices are constants.

The ablations in Appendix D use a model the paper calls 247M, which is the smallest configuration as Table 10 counts it, embedding matrix included. Table 2 excludes embeddings and Table 10 does not: Table 10's sizes minus Table 2's baseline sizes (247 − 132 = 115M, then 196M, 265M and 523M) each match one 128,000-row embedding at the model's width. The ablations train on a compressed 157-billion-token schedule and report C4 bits per byte. The default RETRO ends at 0.822 and the same model with no retrieval at 0.987, a gap of 0.165:

Results: how RETRO scales with model and database size

The main comparison trains four pairs of models: a baseline and a RETRO model at each of four sizes, on the same data and schedule. Table 14 of the paper gives C4 bits per byte for each, and also for RETRO evaluated with retrieval switched off, RETRO[OFF]. Retrieval improves the four sizes by 0.16, 0.15, 0.13 and 0.12 bits per byte. In relative terms the gain is almost flat, 16.3% at the two smaller sizes and 15.4% at 7.5B. With retrieval switched off RETRO matches the baseline exactly on C4 at every size, so training with retrieval did not cost the model its ability to predict without it.

The same table sweeps the evaluation database from 2 billion to 900 billion tokens, and the number of neighbors from 1 to 100, with the models trained as before. Figure 3 plots all three sweeps. In the model-size view, look at the vertical distance between the curves; in the database view, at how much of the gain arrives only at the largest sizes; in the neighbors view, at where each model's curve turns back up.

Figure 3 · C4 bits per byte against model size, database size and neighbors
Data from Table 14 of the paper. The tabs switch the swept quantity. In the model-size view the second row of tabs switches the evaluation set; Wikitext103 is perplexity and LAMBADA is accuracy, higher is better. Hover or tap a point for its value. The rightmost database point is the full 1.75T-token database.

Three things show up. First, the 172M RETRO at 0.82 sits below the 1.5B baseline at 0.84, which is the basis of the paper's claim that retrieval is worth about a tenfold increase in parameters on C4. Second, the database curve is steepest at the large end: shrinking the database from 1.75T to 900B tokens costs the 172M model 0.06 bits per byte, while going from 2B to 90B gains it only 0.02. C4 is also one of the evaluation sets with chunks that overlap the training data, which the leakage section returns to. Third, the models were trained with 2 neighbors and still improve with more at evaluation: the 172M model is flat from 3 to 30 neighbors and degrades beyond, from 0.82 to 0.85 at 100, while the 7.5B model keeps improving to 0.65 at 30 and 40 neighbors.

On the other evaluation sets the gain varies with how much of the set resembles the database. On the September 2021 Wikipedia set, 23 articles added or heavily edited after the database was collected, the 7.5B model goes from 0.65 to 0.61 bits per byte. On Curation Corpus, whose summaries are not in the database, the gain is 0.02. On LAMBADA, last-word prediction in novels, accuracy goes from 69% to 73%. On Wikitext103, Wikipedia articles that overlap the database, perplexity drops from 10.65 to 2.22.

The Pile comparison is against other people's models: Jurassic-1 (178B parameters) and Gopher (280B), trained on different data. Figure 4 shows bits per byte on each of the 16 subsets used (the paper excludes Enron Emails and YouTube Subtitles), relative to the 7B baseline.

Figure 4 · the Pile, relative to the 7B baseline
github: 7B baseline 0.420 · GPT-3 0.645 · Jurassic-1 0.358 · Gopher 0.367 · RETRO 0.199
Data from Table 15 of the paper; bars are the change in bits per byte against the 7B baseline, so left is better. Teal is the 7.5B RETRO, amber the comparison model chosen with the tabs. Click a row for the absolute values.

RETRO is ahead of its baseline on all 16 subsets, by 1.1% on dm_mathematics and 1.8% on ubuntu_irc at the low end and by 52.6% on GitHub and 50.2% on Project Gutenberg books at the high end. It beats Jurassic-1 on 10 subsets and Gopher on 9, and the unweighted mean over the 16 is 0.670 for RETRO, 0.694 for Gopher and 0.705 for Jurassic-1 (our average of Table 15). The two largest gains are on code and public-domain books, and the database contains a GitHub subset and a books subset. On dm_mathematics and ubuntu_irc the paper suspects the nearest-neighbor search returns chunks that do not help.

Wikitext103 gives a controlled comparison with kNN-LM, which interpolates the language model's next-token distribution with a distribution built from the nearest stored tokens:

pkNN-LM=λ pkNN+(1−λ) pLMp_{k\mathrm{NN\text{-}LM}}=\lambda\,p_{k\mathrm{NN}}+(1-\lambda)\,p_{\mathrm{LM}}

The paper trains a baseline on the Wikitext103 training set (validation perplexity 21.53), reimplements kNN-LM on it with λ = 0.118, and retrofits the same baseline into RETRO. Retrieving from the 4-billion-token Wikipedia, the two are close: 18.52 for kNN-LM and 18.46 for RETRO on validation, 19.54 and 18.97 on test. RETRO can also swap databases at evaluation, which kNN-LM at this scale cannot afford: with C4 it reaches 12.87 validation perplexity, with 10% of MassiveText 13.54, and with all of it 3.21 on validation and 3.92 on test. The paper attributes the last two numbers partly to Wikitext103 text that survived deduplication.

On question answering, the 7.5B RETRO is fine-tuned on Natural Questions with the 20 passages that DPR retrieves for each question, fed through the encoder, and the input left-padded so that "answer:" ends exactly at the end of the first 64-token chunk, the point where the decoder starts reading neighbors. It reaches 45.5% exact match, against 30.4% for the 7B baseline answering from its weights alone, 44.5% for RAG and 51.4% for FiD, a T5-based model that reads 100 retrieved passages with its encoder. More than 20 passages did not help RETRO. The paper's hypothesis is that T5's encoder-decoder pre-training makes FiD rely more on its encoder.

Retrofitting a trained transformer with retrieval

A baseline that was trained without retrieval can be turned into a RETRO model afterwards. The paper freezes all of the pretrained weights, adds a freshly initialized encoder and cross-attention layers (every third layer from layer 6), and trains only those on 6 million sequences, 3% of the 204.8 million used in pre-training. For the 7.5B model the new weights are 550M of 7,532M, 7.3%. The learning rate decays from 2×10−42 \times 10^{-4} to 2×10−52 \times 10^{-5} (half of each for the 7.5B model) at a batch of 256.

Retrofitted models pass the baseline quickly and end close to RETRO trained from scratch (Figure 5 of the paper). Because the original weights are frozen, the retrofitted model with retrieval switched off is exactly the baseline. The authors also tried unfreezing everything and found it worse: the retrieval-off performance degraded. On Wikitext103 retrofitting is the only option the paper found to work, because a RETRO trained from scratch on that small dataset overfit and, retrieving from Wikipedia, ended below the baseline.

How much of RETRO's gain is copying from leaked text

A retrieval model can do two different things with its neighbors. It can copy: when the test document, or a near-duplicate of it, is in the database, the neighbors contain the answer token for token. Or it can use related text: facts, names and phrasing from other documents on the same subject. Table 19 shows the first case, a Wikitext103 article about Daniel Radcliffe whose second and third chunks are almost entirely present, up to formatting, in the neighbors retrieved from C4. Table 17 shows the second: the name of the Tyrone football team appears in a neighbor of chunk 1 and then in chunk 2 of a September 2021 article that did not exist when the database was built.

Section 2.6 measures overlap chunk by chunk. For an evaluation chunk CC, the 10 nearest training chunks are retrieved, and ss is the length of the longest run of consecutive tokens that CC shares with any of them. The overlap ratio r(C)=s/mr(C) = s/m is 0 for a chunk with no shared token run and 1 for a chunk found in full. Given each chunk's log-likelihood ℓ(C)\ell(C) in bits and its length N(C)N(C) in bytes, the filtered bits per byte keeps only the chunks with overlap at most α\alpha:

Cα≜{C∈C, r(C)⩽α},bpb(α)≜∑C∈Cαℓ(C)∑C∈CαN(C)\mathcal{C}_{\alpha}\triangleq\{C\in\mathcal{C},\,r(C)\leqslant\alpha\},\qquad\mathrm{bpb}(\alpha)\triangleq\frac{\sum_{C\in\mathcal{C}_{\alpha}}\ell(C)}{\sum_{C\in\mathcal{C}_{\alpha}}N(C)}Cα≜{C∈C, r(C)⩽α},bpb(α)≜∑C∈Cαℓ(C)∑C∈CαN(C)\begin{gathered}\mathcal{C}_{\alpha}\triangleq\{C\in\mathcal{C},\,r(C)\leqslant\alpha\},\\\mathrm{bpb}(\alpha)\triangleq\frac{\sum_{C\in\mathcal{C}_{\alpha}}\ell(C)}{\sum_{C\in\mathcal{C}_{\alpha}}N(C)}\end{gathered}
(5)

At α=1\alpha = 1 every chunk is kept and bpb(1)\mathrm{bpb}(1) is the ordinary score. Lowering α\alpha removes the chunks with the longest shared runs first. A model that gains mostly by copying loses its advantage as α\alpha falls; a model whose gain survives at small α\alpha is helped on chunks it could not have copied. For ℓ(C)\ell(C) to give positive bits per byte it has to be read as the negative log-likelihood here, the opposite sign of ℓθ\ell_\theta in equation (1).

Figure 5 · the overlap ratio of equation (5) on example chunks
α = 12.5%
Illustrative 16-word chunks, one word per token (the paper uses 64 tokens and the best of 10 neighbors). Pick an evaluation chunk with the tabs; the amber run is the longest one it shares with the neighbor, and teal marks other words they share. Drag α\alpha to see whether the chunk is counted in bpb(α)\mathrm{bpb}(\alpha).

The reordered example in Figure 5 shares 11 of its 16 words with the neighbor and still has an overlap ratio of 3/16, because ss counts only consecutive tokens. The measure catches copying and near-copying. It does not catch a chunk that states the same facts in a different order, and a model can still gain on such a chunk from its neighbors.

Figure 6 of the paper plots bpb(α)\mathrm{bpb}(\alpha) for C4, Curation Corpus and Wikitext103. On C4 and Wikitext103 both baseline and RETRO improve as α\alpha grows, since even the baseline has memorized some leaked text, and RETRO's curve falls more steeply, so RETRO exploits leakage more. On Curation Corpus, which by design does not overlap the training data, RETRO is better by a constant offset. At α=12.5%\alpha = 12.5\%, which keeps only chunks that share fewer than 8 consecutive tokens with any neighbor, RETRO is still better than the baseline on all three datasets. The paper reads this as RETRO generalizing from its neighbors and concludes that only a modest fraction of its gains come from leakage; it gives the curves but no per-dataset numbers at 12.5%.

A later study tested that reading. Norlund et al. (2023) reimplemented RETRO and concluded that its gains from retrieval come largely from tokens that overlap between the database and the test data, and that even limited overlap can lower test loss substantially. A chunk with fewer than 8 consecutive shared tokens can still share many shorter runs, as in Figure 5.

Limits and later work

A model that reads its training data at test time scores better on any test text duplicated in that data, deduplication by 13-gram Jaccard similarity catches only near-exact copies, and the strongest headline numbers (Wikitext103 at 3.92 test perplexity when retrieving from all of MassiveText, the largest Pile gains) come from datasets whose content the database plausibly contains. The filtered evaluation of equation (5) is a method any language model can use; the paper cautions about such leakage in large-scale datasets and calls for further work on its role.

Retrieval quality is fixed. The frozen BERT was not trained for this task, nothing in RETRO's loss can improve it, and the weakest Pile subsets (mathematics problems and chat logs) are where the paper suspects it retrieves unhelpful chunks. On Natural Questions RETRO trails FiD by 5.9 points and stops improving at 20 passages.

The paper argues that a database is easier to change than weights: documents can be removed at inference time to stop the model from copying private or toxic text, or added to keep it current, without retraining. It does not run these experiments. It also notes the other side: a model that can copy its training data verbatim makes memorization of private text a more direct risk.

The main replication at scale is Wang et al. (2023), who rebuilt RETRO in NVIDIA's Megatron-LM up to 9.5B parameters with a 330-billion-token database. They found that RETRO generated text with less repetition and somewhat higher factual accuracy than a GPT model of the same size, scored much better on knowledge-intensive tasks and about the same on others, and that a variant they call RETRO++ scored 8.6 exact-match points above RETRO on Natural Questions.

Provenance Verified against primary literatureHow we verify
Borgeaud et al. (2021), v3Abstract; Section 2.1 and Table 1 (MassiveText, SentencePiece vocabulary 128,000, 600B-token training database, 1.75T-token evaluation database with books at 4%, 13-gram MinHash deduplication at Jaccard 0.8); Section 2.2 and equation (1) (n = 2048, m = 64, Ret(C_1) empty); Section 2.3 (key = time-averaged BERT embedding of N, squared L2 distance, value [N, F] with r = 128, same-document filter, ScaNN at 10 ms, precomputed training neighbors, Wikipedia link distance 2 to 3); Section 2.4, equation (2), Algorithm 1, equations (3) and (4), the one-token overlap footnote; Section 2.5 and Table 2 (RMSNorm, relative positions, encoder d' = 896 with 2 layers adding about 19M, cross-attention in layers 6, 9, 12 plus once in the encoder adding 12M); Section 2.6 and equation (5); Sections 4.1 to 4.5 with Tables 4 and 5; Appendix B.1.2 equations (6) and (7); Listing 1; B.2; C.1 to C.4 and Tables 10 to 12; Appendix D and Table 13; Tables 14 and 15; Tables 16 to 21.
Listing 1 (the authors' code)The paper releases no repository; the JAX listing in Appendix B.1.3 is the only code from the authors. multi_chunk_cross_attention pads observation[m-1:] with m - 1 zero rows and reshapes it into l chunks, so attending chunk u covers 1-indexed positions u m to u m + m - 1, the alignment of equation (3); multi_neighbour_cross_attention reshapes the logits to (m, r k) before one softmax, so all neighbors of a chunk share one softmax; the output is shifted back by m - 1 rows with zero padding, which is the identity at positions 1 to m - 1 once the residual is added.
Khandelwal et al. (2020)Generalization through Memorization: Nearest Neighbor Language Models, Section 2: p(y|x) = λ p_kNN(y|x) + (1 - λ) p_LM(y|x), keys are 1024-dimensional final-layer states, one per token; λ = 0.25 is best on Wikitext-103 in their setup.
Norlund et al. (2023)On the Generalization Ability of Retrieval-Enhanced Transformers, abstract: the gains from retrieval largely originate from tokens that overlap between the database and the test data.
Wang et al. (2023)Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study, abstract: RETRO reproduced up to 9.5B parameters with a 330B-token database; large gains on knowledge-intensive tasks, on par with GPT elsewhere; RETRO++ adds 8.6 exact-match points on Natural Questions.
Our recomputationTraining tokens 419,430,400,000 / 2,048 = 204.8M sequences; / 256 per batch = 800,000 steps (200,000 at batch 1,024); 3% of 204.8M = 6.1M, the paper's 6 million retrofitting sequences. Evaluation database 5,026,463M - 3,423,740M + 4% of 3,423,740M = 1.74T tokens. Table 4 keys: 1,792B / 64 = 28B. kNN-LM index: 4B tokens x 1,024 floats x 4 bytes = 16.4 TB = 14.9 TiB. Table 10 minus Table 2 sizes: 115M, 196M, 265M, 523M, which match one 128,000-row embedding at d = 896, 1,536, 2,048, 4,096. Retro parameter overhead of the 7.5B model: 550M of 7,532M = 7.3%. Ablation shares: (0.987 - 0.950) / (0.987 - 0.822) = 22.4% and (0.987 - 0.895) / 0.165 = 55.8%. C4 gains from Table 14: 0.16, 0.15, 0.13, 0.12 bits per byte (16.3%, 16.3%, 15.5%, 15.4%). Pile counts and the unweighted means (baseline 0.774, GPT-3 0.811, Jurassic-1 0.705, Gopher 0.694, RETRO 0.670) are from Table 15.
correctionSeveral slips in the paper, each checked against its own tables or the cited work. (a) Equation (6) writes the cross-attention distance as i - i' + l - 1 with i in [1, l] and i' in [1, 2l], where l is the number of chunks everywhere else; the ranges and the offset are those of the chunk length m, and i - i' + m - 1 is the distance that follows from the one-token shift of the attending chunk. (b) Appendix C.2 writes kNN-LM as λ p_LM + (1 - λ) p_kNN, the reverse of Section 4.1 and of Khandelwal et al.; with λ = 0.118, λ is the weight on p_kNN. (c) Section 4.1 says RETRO does not outperform the 7B baseline on dm_mathematics and ubuntu_irc; Table 15 has it lower on both (1.164 against 1.177, 1.178 against 1.200), small gains of 1.1% and 1.8%, and the Figure 4 caption says it beats the baseline on all test sets. (d) Table 10 lists the Retro layers of the 1.5B and 7.5B models as starting at layer 9, while Section 2.5 and Table 12 say every third layer from layer 6; the paper does not resolve this. (e) The abstract's "25x fewer parameters" is 23.3x against GPT-3 and 23.7x against Jurassic-1, and its "order of magnitude more data than what is typically consumed during training" is a 1.75T-token database against 419B training tokens, about 4x. Separately, Table 13 shows training with 4 neighbors at 0.847 bits per byte, worse than the 0.822 of the 2-neighbor default, where the text says only that it gives no substantial improvement; and Table 2's 172M exceeds 132M plus the 19M and 12M the text itemizes.

Questions you might still have

?

How is RETRO different from retrieval-augmented generation (RAG)?
RAG retrieves once per input, from 21 million Wikipedia passages, with a retriever fine-tuned for the task, and it was built for question answering. RETRO retrieves again for every 64-token chunk of the text it reads or writes, from a database of up to 28 billion chunks, with a frozen BERT, and the retrieval pathway is in place for the whole of pre-training rather than added for one downstream task. Both read retrieved text with cross-attention from an encoder. The RAG explainer on this site covers its marginalization over passages, which RETRO does not do: RETRO gives all neighbors to one cross-attention softmax at once.

?

Why is the retriever frozen? Would a trained one be better?
A trained retriever changes its embeddings as it learns, so every key in the database goes stale and has to be recomputed. REALM does this periodically for 13 million Wikipedia passages; RETRO has 28 billion chunks, and recomputing them means running BERT over 1.75 trillion tokens. A frozen BERT also lets the paper compute every training chunk's neighbors once, before training, because retrieving on the fly could not keep up with the accelerators. The paper does not test a trained retriever at this scale, so whether one would help is open. Its weakest Pile subsets, dm_mathematics and ubuntu_irc, are where it suspects the nearest-neighbor search returns unhelpful chunks.

?

Can RETRO answer questions about events after its training data?
Only if the new text is in the database, and only as well as cross-attention can use it. The paper does not add new documents and measure the effect. Its September 2021 Wikipedia set tests text written after the database was collected: the 7.5B model gains 0.04 bits per byte there (0.65 to 0.61), against 0.12 on C4. In Table 18 the retrieved neighbors for the 2020 Summer Paralympics were written before the games were postponed, and they carry the original dates.

?

Does RETRO need the database at inference time?
No. With retrieval switched off (RETRO[OFF]) the 7.5B model scores 0.78 bits per byte on C4, the same as the baseline trained without retrieval. On the other datasets in Table 14 it is better than the baseline on LAMBADA and at most 2.5% worse elsewhere. With retrieval on it scores 0.66. A retrofitted model keeps its original weights frozen, so its retrieval-off performance is exactly the baseline's.

?

Why does a chunk use the neighbors of the previous chunk and not its own?
Because its own neighbors are looked up from its own 64 tokens. While the model is predicting token 30 of a chunk, tokens 31 to 64 do not exist yet, so a lookup that depends on them would leak the future during training and could not be computed during sampling. The neighbors of the previous chunk are ready as soon as that chunk is complete, which is why the attending chunks start one token early: the last token of chunk u is the first position that has read all of chunk u.

?

How much does retrieval cost at inference?
The paper reports 10 milliseconds per query on the 2-trillion-token database with the ScaNN library, one query per 64 tokens, which is about 0.16 milliseconds per generated token. Each query also requires a BERT pass over 64 tokens and the 2-layer encoder over k neighbors of 128 tokens each. The paper calls the retrieval cost negligible next to sampling, but it does not report end-to-end latency, and the database is 93 terabytes, so it has to live on some storage system next to the accelerators.

?

Why not paste the retrieved text into the prompt instead of using an encoder?
For a 2,048-token sequence with 2 neighbors of 128 tokens per chunk, the retrieved text is 32 x 2 x 128 = 8,192 tokens, four times the sequence. Prepended to the prompt, self-attention would run over about 10,000 tokens, roughly 25 times the attention cost of 2,048 tokens (our arithmetic), and the neighbors for later chunks would have to be in the prompt before the chunks that retrieve them exist. RETRO encodes each neighbor separately with a 2-layer encoder and lets each query attend only 256 retrieved tokens, so the cost grows linearly with the amount retrieved.

?

Is "25x fewer parameters" a fair summary?
The 7.5B RETRO against the 178B Jurassic-1 is a factor of 23.7, against the 175B GPT-3 23.3, and against the 280B Gopher 37. On the Pile, RETRO has lower bits per byte than Jurassic-1 on 10 of 16 subsets and than Gopher on 9. The comparison counts parameters only: RETRO also reads a 1.75-trillion-token database at evaluation, and its largest wins are on GitHub code and Project Gutenberg books, two kinds of text the database contains.

Footnotes & further reading

  1. The paper: Borgeaud, Mensch, Hoffmann, Cai, Rutherford, Millican, van den Driessche, Lespiau, Damoc, Clark, de Las Casas, Guy, Menick, Ring, Hennigan, Huang, Maggiore, Jones, Cassirer, Brock, Paganini, Irving, Vinyals, Osindero, Simonyan, Rae, Elsen and Sifre, Improving language models by retrieving from trillions of tokens (ICML 2022; this page follows arXiv v3). The authors released no repository; Listing 1 in Appendix B.1.3 is their implementation of chunked cross-attention.
  2. The corpus and the Gopher comparison: Rae et al., Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021). The Pile: Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling (2020). GPT-3, whose Pile scores appear in Table 15, is explained at GPT-3.
  3. Token-level retrieval: Khandelwal, Levy, Jurafsky, Zettlemoyer and Lewis, Generalization through Memorization: Nearest Neighbor Language Models (ICLR 2020).
  4. Prompt-level retrieval: Guu et al., REALM: Retrieval-Augmented Language Model Pre-Training (2020); Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020), explained at RAG; Izacard and Grave, Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (FiD, EACL 2021).
  5. The retriever and the search: Devlin et al., BERT (2019), explained at BERT; Guo et al., Accelerating Large-Scale Inference with Anisotropic Vector Quantization (ScaNN, ICML 2020).
  6. Later work: Norlund, Doostmohammadi, Johansson and Kuhlmann, On the Generalization Ability of Retrieval-Enhanced Transformers (2023); Wang et al., Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023).