RETRO: Improving language models by retrieving from trillions of tokens
A 7.5-billion-parameter language model looks up text similar to each 64-token chunk of its input in a database of 1.75 trillion tokens, and predicts text better than a 178-billion-parameter model on 10 of 16 parts of the Pile benchmark.
DeepMind's Retrieval-Enhanced Transformer. A frozen BERT model finds the stored chunks nearest to each chunk of the input, a small encoder reads them, and a cross-attention layer lets every token use the text retrieved for the chunk before its own, so the model still predicts one token at a time from the past.
Explaining the paperImproving language models by retrieving from trillions of tokensA model that can read its training data at test time also scores well on any test text that leaked into that data. The paper builds an evaluation that removes leaked chunks and reports how much of the gain is left.
What RETRO adds to a language model
A transformer language model keeps everything it has learned in its weights. Making it bigger buys two things, and the paper separates them: more computation per token, and more capacity to memorize the training text. RETRO (Retrieval-Enhanced Transformer) tries to get the second without the first. It keeps a 7.5B-parameter model and gives it a database of text, which it searches while it reads and while it writes.
The unit of lookup is a chunk of 64 tokens. After each chunk of input, the model embeds that chunk with a frozen BERT, finds the nearest stored chunks in a database of 28 billion of them, and fetches each one together with the 64 tokens that followed it in its source document. A 2-layer encoder reads the fetched text, and cross-attention layers in the decoder let the next 64 tokens use it. At evaluation the database is a multilingual version of the MassiveText corpus from the Gopher paper, about 1.75 trillion tokens of web pages, books, news, Wikipedia and GitHub code.
Results are reported in bits per byte: the model's total negative log-likelihood of a document, in bits, divided by the document's length in UTF-8 bytes. Lower is better, and unlike perplexity the number does not depend on the tokenizer. On the C4 web text corpus the 7.5B RETRO reaches 0.66 bits per byte. A 7B transformer trained the same way without retrieval reaches 0.78, so retrieval removes 15% of the bits needed to encode the text. The smallest RETRO, 172M parameters, reaches 0.82, which is better than the baseline in the 1.5B column of the paper's Table 14 (0.84, a 1.3B-parameter model). On the Pile, a benchmark of 16 text domains used here, RETRO beats its own baseline on all 16, the 178B-parameter Jurassic-1 on 10 and the 280B-parameter Gopher on 9.
Four pieces make it work, and the page follows them in order: a likelihood that splits the sequence into chunks, a retrieval step with a frozen BERT, a chunked cross-attention layer that keeps the model autoregressive, and an evaluation that measures how much of the gain comes from copying test text that is also in the database. Sections on training, results, retrofitting of existing models and limits follow those four.
Chunks and the retrieval-enhanced likelihood
A training example is tokens , split into consecutive chunks of tokens each. Every chunk gets a set of neighbors from the database , computed by a fixed procedure with no trainable weights. The model, with weights , is trained to maximize this log-likelihood:
The term is the model's log-probability of one token. Token of chunk sits at position , and its probability may depend on every earlier token and on the neighbors of every earlier chunk, , but never on the neighbors of its own chunk. The first chunk has no earlier chunk, so the paper sets , and the first 64 tokens are predicted exactly as a plain language model would predict them.
Take token 100. It is token 36 of chunk 2, which covers positions 65 to 128. Its probability conditions on tokens 1 to 99 and on , the neighbors found for positions 1 to 64. It cannot condition on because the lookup for chunk 2 is computed from all 64 of its tokens, including tokens 100 to 128, which the model has not produced yet when it predicts token 100. Equation (1) is therefore an ordinary next-token likelihood with extra inputs that only ever come from the past, and it can be sampled one token at a time like any autoregressive model: write a chunk, look up its neighbors, write the next chunk.
Why chunks? Earlier retrieval language models worked at one of two granularities. Token-level methods such as kNN-LM store one key per token of the database, which at 1.75 trillion tokens means 1.75 trillion keys. Prompt-level methods such as RAG and REALM retrieve once for the whole input, which is fine for a question but not for a 2,048-token document whose topic moves. Chunks give 64 times fewer keys than tokens, and a fresh lookup every 64 tokens.
How RETRO retrieves neighbors from the database
The database is a key-value store. Each value is two consecutive 64-token chunks from one document, written : the neighbor chunk and its continuation , the 64 tokens that came after it. Each key is computed from alone: BERT outputs one vector per token, and averaging those vectors over the 64 tokens of gives one vector per chunk, written . For a query chunk , the distance to a stored chunk is the squared Euclidean distance between the two averages,
and the model receives the values of the nearest keys:
Each value is tokens, so is a block of token ids. The model is trained with and can be evaluated with more. The continuation matters more for prediction: resembles the chunk just read, so is what followed similar text somewhere else, which is a guess at the chunk about to be written. The ablations measure the split: continuations alone recover 56% of the gain from retrieval, neighbor chunks alone 22%.
The BERT is frozen: its weights never change, so every key can be computed once and stored. A trainable retriever, as in REALM, would change its embeddings as it learns and force the whole database to be re-embedded every so often, which at 28 billion chunks means running BERT over the full 1.75 trillion tokens. Freezing also lets the paper precompute the neighbors of every training chunk before training starts. The paper reports that retrieving on the fly could not keep up with the training computation, so the neighbor ids are stored as part of the training data, like the tokens.
During training, the database contains the training documents themselves, which creates a leak. The nearest stored chunk to would usually be itself, taken from the same document, and its continuation would be , the exact tokens the model is being trained to predict next. The paper therefore discards any neighbor that comes from the same document as the training sequence .
The search is approximate. The ScaNN library (Guo et al., 2020) returns approximate nearest neighbors in time that grows with the logarithm of the database size, and the paper reports 10 milliseconds per query on the 2-trillion-token database. One query serves 64 tokens, about 0.16 milliseconds per token. When the search is restricted to Wikipedia, the retrieved articles are typically 2 to 3 links away from the query's article in the Wikipedia link graph, against more than 5 for random pairs of articles.
There are two databases. For training, 600 billion tokens are drawn from MassiveText in the same proportions as the training mixture. For evaluation, the database is the union of all of MassiveText except books, of which a 4% sample is kept: 1.75 trillion tokens, and 28 billion keys at 64 tokens per key. To limit leakage from training into test sets, any training document with 13-gram Jaccard similarity of 0.8 or more to a validation or test document is removed (estimated with MinHash, a hashing scheme that approximates set overlap without comparing every pair), as are the Wikitext103 validation and test articles.
Chunk-level keys keep the index small. kNN-LM stores a 1,024-number vector for every token: for the 4 billion tokens of Wikipedia that is 4 billion × 1,024 × 4 bytes = 16.4 terabytes, which the paper rounds to 15. RETRO's index for the same Wikipedia is 215 gigabytes, and for all 1.75 trillion tokens of MassiveText it is 93 terabytes: about six times the kNN-LM Wikipedia index, for roughly 440 times as many tokens.
How the RETRO architecture works
RETRO is an encoder-decoder transformer. The decoder is the language model: a GPT-2-style transformer with RMSNorm in place of LayerNorm and with the relative position encodings of Transformer-XL. The encoder reads retrieved text. The decoder is a stack of two kinds of block, built from three residual operators that each map an activation to another: a feed-forward layer Ffw, causal self-attention Attn, and the chunked cross-attention Cca that reads the encoded neighbors .
A Retro block is an ordinary transformer block with one cross-attention layer inserted between self-attention and the feed-forward layer. The set of layers that use it is a hyperparameter; the paper uses every third layer starting at layer 6, so the 12-layer 172M model has Retro blocks at layers 6, 9 and 12. Each operator includes its residual connection, so a cross-attention layer that outputs zero leaves the activations unchanged.
The encoder is a bidirectional transformer of 2 layers and width in every model size. It runs on each retrieved neighbor separately, with weights shared across all neighbors and all chunks, and it is conditioned on the chunk that retrieved the neighbor: its one cross-attention layer lets the neighbor's 128 tokens attend to , the decoder's activations for the 64 tokens of chunk . Those activations are taken at the first Retro layer, , just after its self-attention. The encoder therefore runs once per forward pass, partway up the decoder, and its output is reused by every Retro block above that point.
For one training sequence of the 172M model, has shape = 32 × 2 × 128 × 896. The encoder processes 32 × 2 × 128 = 8,192 retrieved tokens per 2,048-token sequence, four times as many tokens as the decoder, which is affordable because it has 2 layers against the decoder's 12 (32 in the 7.5B model). Figure 1 shows the stack with the shape at each step.
The figure's two parameter counts come from Table 2 of the paper, which excludes embedding matrices. The four baselines have 132M, 368M, 1,309M and 6,982M parameters; the matching RETRO models have 172M, 425M, 1,451M and 7,532M, overheads of 30%, 15%, 11% and 8%. The encoder is the same size for all of them, which is why the relative overhead falls as the decoder grows. For the smallest model the paper attributes about 19M to the encoder and 12M to the cross-attention layers (three in the decoder, one in the encoder).
Chunked cross-attention
Chunked cross-attention is the operator that lets the decoder use without breaking equation (1). It splits the activation into attending chunks , shifted one position earlier than the data chunks: holds positions to , which are the last token of and the first tokens of . Each attending chunk cross-attends to , the encoded neighbors of . For every chunk and every in 1 to :
The shift comes from how a transformer predicts. The output at position is used to predict token . Position is the first position that has read all of , so it is the first that may use , and its output predicts , the first token of , which equation (1) allows to depend on . Position predicts the last token of , still allowed. Position has read all of and moves on to .
Two edge cases complete the definition. Positions 1 to have no completed chunk before them, so Cca is the identity there. The last position, , attends to ; during training its output predicts nothing inside the sequence, and during sampling it predicts the first token of the next chunk.
Ca is ordinary cross-attention. In its simplest form, with query, key and value matrices and , for one activation and attended rows :
are the keys of the attended rows, is the query, their product is one score per row, the softmax turns the scores into weights, and the output is the weighted sum of the value rows . The real model uses several heads, and the encoder width differs from in every model but the smallest, so in practice the key and value matrices take -dimensional rows; (4) writes the single-width case. The attended rows are the encoded tokens of all neighbors of together: has shape , and the neighbor and time dimensions are merged into rows before one softmax. With that is 256 rows, and the two neighbors compete for the same attention weights.
Each cross-attention reads only the neighbors of one chunk, yet a token can depend on all earlier neighbor sets. In layer 6, the first 63 tokens of attend to . In layer 7, self-attention lets a token of read those activations, which already carry information from , while its own cross-attention in layer 9 reads . The paper notes that this gives the dependence on every with in equation (1) without the cost of attending to all of them. One Cca layer computes attention scores, 2,048 × 256 = 524,288 for a training sequence. Attending instead to every earlier chunk's neighbors would make attending chunk read neighbor sets rather than one, 16 times as many on average over the 31 attending chunks. That count grows with the square of the sequence length, while the cost of Cca grows only linearly with the amount retrieved.
Slide to position 3 and nothing is retrieved: positions 1 to 3 come before the first completed chunk. Position 4 is the first to read , and it predicts token 5. Positions 4 to 7 all read , and at position 8 the direct source switches to . Position 12 reads to predict a token beyond the sequence.
Cross-attention scores also get a relative position term, because a neighbor is expected to line up with the chunk that retrieved it: token of is treated as if it sat at the same offset as token of . Token of the attending chunk is at absolute position , and token of the neighbor is aligned with position , so their distance is
with in 1 to and in 1 to (the half of a value continues past the chunk). The paper prints this equation with , its symbol for the number of chunks, in place of ; the Provenance panel records the slip. In the encoder's cross-attention, where neighbor token attends to token of itself (no shift), the distance is , equation (7). The positional logits are a learned linear map of a cosine encoding of these distances, added to the content scores. Without them the ablation model ends at 0.826 bits per byte instead of 0.822.
The paper's own implementation of Cca (Listing 1, JAX) is short. Rewritten with the 172M model's sizes, and with the multi-head and positional terms left out as in the original:
# Listing 1 of the paper (Appendix B.1.3), one head, no positional logits.
# Sizes of the 172M model: n = 2048, m = 64, l = 32, k = 2, r = 128, d = 896.
def cross_attention(chunk, neighbours): # [m, d], [k, r, d]
q = chunk @ Q # [m, d]
keys = neighbours @ K # [k, r, d]
logits = einsum("md,krd->mkr", q, keys) # [64, 2, 128]
logits = logits.reshape(m, k * r) # one softmax over 256 tokens
values = (neighbours @ V).reshape(k * r, d) # [256, d]
return softmax(logits) @ values # [64, d]
def chunked_cross_attention(H, E): # H [n, d], E [l, k, r, d]
# attending chunk u (1-indexed) = positions u*m .. u*m + m - 1:
# the last token of C_u and the first m - 1 tokens of C_{u+1}
att = pad(H[m - 1:], ((0, m - 1), (0, 0))).reshape(l, m, d)
out = stack([cross_attention(att[u], E[u]) for u in range(l)])
# shift back: positions 1 .. m - 1 get 0, so the residual add leaves them as is
return pad(out.reshape(n, d), ((m - 1, 0), (0, 0)))[:n]The padding does the alignment. Dropping the first rows and reshaping makes row 0 of attending chunk the last token of ; after the per-chunk attention, the output is shifted back down by rows with zeros in front, and the zeros become the identity once the residual is added.
Generating text with RETRO
Sampling interleaves generation and lookup. The prompt is the first chunk. When a chunk is complete, its 64 tokens are embedded with BERT, ScaNN returns the neighbors, the encoder encodes them (conditioned on the chunk's activations), and the next 64 tokens are sampled one at a time while their cross-attention reads that encoding. The cost of sampling stays quadratic in the length of the text, as for any transformer, and the retrieval adds a cost linear in the number of chunks, which the paper calls negligible next to sampling the tokens.
The paper's samples show what retrieval changes. Given the opening of Hamlet, the 7.5B model with retrieval continues with the actual text of the play, robust to formatting differences between the prompt and the retrieved copies, while the same model with retrieval switched off writes plausible but invented dialogue and ends by repeating one character's line (Table 7). Prompted with the first digits of π, the model with retrieval copies a long run of correct digits from its neighbors, and without retrieval it diverges two digits after the prompt (Table 21).
Training setup and ablations
Every model trains for 419,430,400,000 tokens, which is 204.8 million sequences of 2,048 tokens, with a batch of 256 sequences (1,024 for the largest model), so 800,000 steps for the three smaller models and 200,000 for the largest. The optimizer is AdamW (the decoupled weight decay variant of Adam) with weight decay 0.1; the learning rate warms up linearly from over 750 steps to a maximum of ( for the largest model), then follows a cosine decay to a tenth of the maximum. Optimizer state is sharded with ZeRO. Retrieval during training uses neighbors from the 600B-token training database. The tokenizer is SentencePiece with a 128,000-token vocabulary. One training step for the 172M model, in pseudocode:
# one training sequence for the 172M RETRO (Sections 2.2 to 2.5, Algorithm 1)
X = sample_sequence() # [2048] token ids, vocabulary 128,000
nbrs = neighbours[X] # [32, 2, 128] token ids, computed OFFLINE:
# chunk u -> 2 nearest [N, F] by frozen BERT,
# same-document neighbours removed
H = emb(X) # [2048, 896]
for p in range(1, 13): # 12 decoder layers; each op adds its residual
H = attn(H) # causal self-attention
if p == 6: # min(P): the encoder runs once, here
E = encoder(nbrs, H.reshape(32, 64, 896))
# [32, 2, 128, 896]: 2 bidirectional layers,
# d' = 896, cross-attends to its chunk's H_u
if p in (6, 9, 12): # P: the Retro blocks
H = chunked_cross_attention(H, E)
H = ffw(H)
logits = read(H) # [2048, 128000]
loss = cross_entropy(logits[:-1], X[1:])
# outputs at positions 1..63 never saw retrieval; position 64 predicts token 65
# from Ret(C_1). Gradients reach emb, all decoder layers, the encoder and the
# cross-attention weights. BERT and the neighbour indices are constants.The ablations in Appendix D use a model the paper calls 247M, which is the smallest configuration as Table 10 counts it, embedding matrix included. Table 2 excludes embeddings and Table 10 does not: Table 10's sizes minus Table 2's baseline sizes (247 − 132 = 115M, then 196M, 265M and 523M) each match one 128,000-row embedding at the model's width. The ablations train on a compressed 157-billion-token schedule and report C4 bits per byte. The default RETRO ends at 0.822 and the same model with no retrieval at 0.987, a gap of 0.165:
- Neighbor or continuation. Training and evaluating with only the neighbor chunks gives 0.950, 22% of the gap; with only the continuations , 0.895, 56% of the gap. The continuation carries more than twice as much, consistent with it being a guess at the next chunk, and both together give the full 0.822.
- Query conditioning. Removing the encoder's cross-attention to the retrieving chunk gives 0.829. Removing the relative positions in cross-attention gives 0.826.
- Encoder size. A 6-layer encoder gives 0.821, a 0.15% gain in loss for 20% more training time. Sharing the token embedding between encoder and decoder gives 0.823; the default keeps them separate so the encoder can stay at width 896 while the decoder widens to 4,096.
- Training neighbors. One neighbor gives 0.858, four give 0.847. The default two is best in the table, and evaluation can use more neighbors than training did.
- Where the cross-attention goes. One cross-attention at layer 1 gives 0.827, at layer 6 gives 0.823, at layer 12 gives 0.831; every third layer from layer 1 gives 0.823; cross-attention in every layer gives 0.860, the worst setting. The paper keeps every third layer as a trade-off between loss and run time.
Results: how RETRO scales with model and database size
The main comparison trains four pairs of models: a baseline and a RETRO model at each of four sizes, on the same data and schedule. Table 14 of the paper gives C4 bits per byte for each, and also for RETRO evaluated with retrieval switched off, RETRO[OFF]. Retrieval improves the four sizes by 0.16, 0.15, 0.13 and 0.12 bits per byte. In relative terms the gain is almost flat, 16.3% at the two smaller sizes and 15.4% at 7.5B. With retrieval switched off RETRO matches the baseline exactly on C4 at every size, so training with retrieval did not cost the model its ability to predict without it.
The same table sweeps the evaluation database from 2 billion to 900 billion tokens, and the number of neighbors from 1 to 100, with the models trained as before. Figure 3 plots all three sweeps. In the model-size view, look at the vertical distance between the curves; in the database view, at how much of the gain arrives only at the largest sizes; in the neighbors view, at where each model's curve turns back up.
Three things show up. First, the 172M RETRO at 0.82 sits below the 1.5B baseline at 0.84, which is the basis of the paper's claim that retrieval is worth about a tenfold increase in parameters on C4. Second, the database curve is steepest at the large end: shrinking the database from 1.75T to 900B tokens costs the 172M model 0.06 bits per byte, while going from 2B to 90B gains it only 0.02. C4 is also one of the evaluation sets with chunks that overlap the training data, which the leakage section returns to. Third, the models were trained with 2 neighbors and still improve with more at evaluation: the 172M model is flat from 3 to 30 neighbors and degrades beyond, from 0.82 to 0.85 at 100, while the 7.5B model keeps improving to 0.65 at 30 and 40 neighbors.
On the other evaluation sets the gain varies with how much of the set resembles the database. On the September 2021 Wikipedia set, 23 articles added or heavily edited after the database was collected, the 7.5B model goes from 0.65 to 0.61 bits per byte. On Curation Corpus, whose summaries are not in the database, the gain is 0.02. On LAMBADA, last-word prediction in novels, accuracy goes from 69% to 73%. On Wikitext103, Wikipedia articles that overlap the database, perplexity drops from 10.65 to 2.22.
The Pile comparison is against other people's models: Jurassic-1 (178B parameters) and Gopher (280B), trained on different data. Figure 4 shows bits per byte on each of the 16 subsets used (the paper excludes Enron Emails and YouTube Subtitles), relative to the 7B baseline.
RETRO is ahead of its baseline on all 16 subsets, by 1.1% on dm_mathematics and 1.8% on ubuntu_irc at the low end and by 52.6% on GitHub and 50.2% on Project Gutenberg books at the high end. It beats Jurassic-1 on 10 subsets and Gopher on 9, and the unweighted mean over the 16 is 0.670 for RETRO, 0.694 for Gopher and 0.705 for Jurassic-1 (our average of Table 15). The two largest gains are on code and public-domain books, and the database contains a GitHub subset and a books subset. On dm_mathematics and ubuntu_irc the paper suspects the nearest-neighbor search returns chunks that do not help.
Wikitext103 gives a controlled comparison with kNN-LM, which interpolates the language model's next-token distribution with a distribution built from the nearest stored tokens:
The paper trains a baseline on the Wikitext103 training set (validation perplexity 21.53), reimplements kNN-LM on it with λ = 0.118, and retrofits the same baseline into RETRO. Retrieving from the 4-billion-token Wikipedia, the two are close: 18.52 for kNN-LM and 18.46 for RETRO on validation, 19.54 and 18.97 on test. RETRO can also swap databases at evaluation, which kNN-LM at this scale cannot afford: with C4 it reaches 12.87 validation perplexity, with 10% of MassiveText 13.54, and with all of it 3.21 on validation and 3.92 on test. The paper attributes the last two numbers partly to Wikitext103 text that survived deduplication.
On question answering, the 7.5B RETRO is fine-tuned on Natural Questions with the 20 passages that DPR retrieves for each question, fed through the encoder, and the input left-padded so that "answer:" ends exactly at the end of the first 64-token chunk, the point where the decoder starts reading neighbors. It reaches 45.5% exact match, against 30.4% for the 7B baseline answering from its weights alone, 44.5% for RAG and 51.4% for FiD, a T5-based model that reads 100 retrieved passages with its encoder. More than 20 passages did not help RETRO. The paper's hypothesis is that T5's encoder-decoder pre-training makes FiD rely more on its encoder.
Retrofitting a trained transformer with retrieval
A baseline that was trained without retrieval can be turned into a RETRO model afterwards. The paper freezes all of the pretrained weights, adds a freshly initialized encoder and cross-attention layers (every third layer from layer 6), and trains only those on 6 million sequences, 3% of the 204.8 million used in pre-training. For the 7.5B model the new weights are 550M of 7,532M, 7.3%. The learning rate decays from to (half of each for the 7.5B model) at a batch of 256.
Retrofitted models pass the baseline quickly and end close to RETRO trained from scratch (Figure 5 of the paper). Because the original weights are frozen, the retrofitted model with retrieval switched off is exactly the baseline. The authors also tried unfreezing everything and found it worse: the retrieval-off performance degraded. On Wikitext103 retrofitting is the only option the paper found to work, because a RETRO trained from scratch on that small dataset overfit and, retrieving from Wikipedia, ended below the baseline.
How much of RETRO's gain is copying from leaked text
A retrieval model can do two different things with its neighbors. It can copy: when the test document, or a near-duplicate of it, is in the database, the neighbors contain the answer token for token. Or it can use related text: facts, names and phrasing from other documents on the same subject. Table 19 shows the first case, a Wikitext103 article about Daniel Radcliffe whose second and third chunks are almost entirely present, up to formatting, in the neighbors retrieved from C4. Table 17 shows the second: the name of the Tyrone football team appears in a neighbor of chunk 1 and then in chunk 2 of a September 2021 article that did not exist when the database was built.
Section 2.6 measures overlap chunk by chunk. For an evaluation chunk , the 10 nearest training chunks are retrieved, and is the length of the longest run of consecutive tokens that shares with any of them. The overlap ratio is 0 for a chunk with no shared token run and 1 for a chunk found in full. Given each chunk's log-likelihood in bits and its length in bytes, the filtered bits per byte keeps only the chunks with overlap at most :
At every chunk is kept and is the ordinary score. Lowering removes the chunks with the longest shared runs first. A model that gains mostly by copying loses its advantage as falls; a model whose gain survives at small is helped on chunks it could not have copied. For to give positive bits per byte it has to be read as the negative log-likelihood here, the opposite sign of in equation (1).
The reordered example in Figure 5 shares 11 of its 16 words with the neighbor and still has an overlap ratio of 3/16, because counts only consecutive tokens. The measure catches copying and near-copying. It does not catch a chunk that states the same facts in a different order, and a model can still gain on such a chunk from its neighbors.
Figure 6 of the paper plots for C4, Curation Corpus and Wikitext103. On C4 and Wikitext103 both baseline and RETRO improve as grows, since even the baseline has memorized some leaked text, and RETRO's curve falls more steeply, so RETRO exploits leakage more. On Curation Corpus, which by design does not overlap the training data, RETRO is better by a constant offset. At , which keeps only chunks that share fewer than 8 consecutive tokens with any neighbor, RETRO is still better than the baseline on all three datasets. The paper reads this as RETRO generalizing from its neighbors and concludes that only a modest fraction of its gains come from leakage; it gives the curves but no per-dataset numbers at 12.5%.
A later study tested that reading. Norlund et al. (2023) reimplemented RETRO and concluded that its gains from retrieval come largely from tokens that overlap between the database and the test data, and that even limited overlap can lower test loss substantially. A chunk with fewer than 8 consecutive shared tokens can still share many shorter runs, as in Figure 5.
Limits and later work
A model that reads its training data at test time scores better on any test text duplicated in that data, deduplication by 13-gram Jaccard similarity catches only near-exact copies, and the strongest headline numbers (Wikitext103 at 3.92 test perplexity when retrieving from all of MassiveText, the largest Pile gains) come from datasets whose content the database plausibly contains. The filtered evaluation of equation (5) is a method any language model can use; the paper cautions about such leakage in large-scale datasets and calls for further work on its role.
Retrieval quality is fixed. The frozen BERT was not trained for this task, nothing in RETRO's loss can improve it, and the weakest Pile subsets (mathematics problems and chat logs) are where the paper suspects it retrieves unhelpful chunks. On Natural Questions RETRO trails FiD by 5.9 points and stops improving at 20 passages.
The paper argues that a database is easier to change than weights: documents can be removed at inference time to stop the model from copying private or toxic text, or added to keep it current, without retraining. It does not run these experiments. It also notes the other side: a model that can copy its training data verbatim makes memorization of private text a more direct risk.
The main replication at scale is Wang et al. (2023), who rebuilt RETRO in NVIDIA's Megatron-LM up to 9.5B parameters with a 330-billion-token database. They found that RETRO generated text with less repetition and somewhat higher factual accuracy than a GPT model of the same size, scored much better on knowledge-intensive tasks and about the same on others, and that a variant they call RETRO++ scored 8.6 exact-match points above RETRO on Natural Questions.
Questions you might still have
How is RETRO different from retrieval-augmented generation (RAG)?
RAG retrieves once per input, from 21 million Wikipedia passages, with a retriever fine-tuned for the task, and it was built for question answering. RETRO retrieves again for every 64-token chunk of the text it reads or writes, from a database of up to 28 billion chunks, with a frozen BERT, and the retrieval pathway is in place for the whole of pre-training rather than added for one downstream task. Both read retrieved text with cross-attention from an encoder. The RAG explainer on this site covers its marginalization over passages, which RETRO does not do: RETRO gives all neighbors to one cross-attention softmax at once.
Why is the retriever frozen? Would a trained one be better?
A trained retriever changes its embeddings as it learns, so every key in the database goes stale and has to be recomputed. REALM does this periodically for 13 million Wikipedia passages; RETRO has 28 billion chunks, and recomputing them means running BERT over 1.75 trillion tokens. A frozen BERT also lets the paper compute every training chunk's neighbors once, before training, because retrieving on the fly could not keep up with the accelerators. The paper does not test a trained retriever at this scale, so whether one would help is open. Its weakest Pile subsets, dm_mathematics and ubuntu_irc, are where it suspects the nearest-neighbor search returns unhelpful chunks.
Can RETRO answer questions about events after its training data?
Only if the new text is in the database, and only as well as cross-attention can use it. The paper does not add new documents and measure the effect. Its September 2021 Wikipedia set tests text written after the database was collected: the 7.5B model gains 0.04 bits per byte there (0.65 to 0.61), against 0.12 on C4. In Table 18 the retrieved neighbors for the 2020 Summer Paralympics were written before the games were postponed, and they carry the original dates.
Does RETRO need the database at inference time?
No. With retrieval switched off (RETRO[OFF]) the 7.5B model scores 0.78 bits per byte on C4, the same as the baseline trained without retrieval. On the other datasets in Table 14 it is better than the baseline on LAMBADA and at most 2.5% worse elsewhere. With retrieval on it scores 0.66. A retrofitted model keeps its original weights frozen, so its retrieval-off performance is exactly the baseline's.
Why does a chunk use the neighbors of the previous chunk and not its own?
Because its own neighbors are looked up from its own 64 tokens. While the model is predicting token 30 of a chunk, tokens 31 to 64 do not exist yet, so a lookup that depends on them would leak the future during training and could not be computed during sampling. The neighbors of the previous chunk are ready as soon as that chunk is complete, which is why the attending chunks start one token early: the last token of chunk u is the first position that has read all of chunk u.
How much does retrieval cost at inference?
The paper reports 10 milliseconds per query on the 2-trillion-token database with the ScaNN library, one query per 64 tokens, which is about 0.16 milliseconds per generated token. Each query also requires a BERT pass over 64 tokens and the 2-layer encoder over k neighbors of 128 tokens each. The paper calls the retrieval cost negligible next to sampling, but it does not report end-to-end latency, and the database is 93 terabytes, so it has to live on some storage system next to the accelerators.
Why not paste the retrieved text into the prompt instead of using an encoder?
For a 2,048-token sequence with 2 neighbors of 128 tokens per chunk, the retrieved text is 32 x 2 x 128 = 8,192 tokens, four times the sequence. Prepended to the prompt, self-attention would run over about 10,000 tokens, roughly 25 times the attention cost of 2,048 tokens (our arithmetic), and the neighbors for later chunks would have to be in the prompt before the chunks that retrieve them exist. RETRO encodes each neighbor separately with a 2-layer encoder and lets each query attend only 256 retrieved tokens, so the cost grows linearly with the amount retrieved.
Is "25x fewer parameters" a fair summary?
The 7.5B RETRO against the 178B Jurassic-1 is a factor of 23.7, against the 175B GPT-3 23.3, and against the 280B Gopher 37. On the Pile, RETRO has lower bits per byte than Jurassic-1 on 10 of 16 subsets and than Gopher on 9. The comparison counts parameters only: RETRO also reads a 1.75-trillion-token database at evaluation, and its largest wins are on GitHub code and Project Gutenberg books, two kinds of text the database contains.
Footnotes & further reading
- The paper: Borgeaud, Mensch, Hoffmann, Cai, Rutherford, Millican, van den Driessche, Lespiau, Damoc, Clark, de Las Casas, Guy, Menick, Ring, Hennigan, Huang, Maggiore, Jones, Cassirer, Brock, Paganini, Irving, Vinyals, Osindero, Simonyan, Rae, Elsen and Sifre, Improving language models by retrieving from trillions of tokens (ICML 2022; this page follows arXiv v3). The authors released no repository; Listing 1 in Appendix B.1.3 is their implementation of chunked cross-attention.
- The corpus and the Gopher comparison: Rae et al., Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021). The Pile: Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling (2020). GPT-3, whose Pile scores appear in Table 15, is explained at GPT-3.
- Token-level retrieval: Khandelwal, Levy, Jurafsky, Zettlemoyer and Lewis, Generalization through Memorization: Nearest Neighbor Language Models (ICLR 2020).
- Prompt-level retrieval: Guu et al., REALM: Retrieval-Augmented Language Model Pre-Training (2020); Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020), explained at RAG; Izacard and Grave, Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (FiD, EACL 2021).
- The retriever and the search: Devlin et al., BERT (2019), explained at BERT; Guo et al., Accelerating Large-Scale Inference with Anisotropic Vector Quantization (ScaNN, ICML 2020).
- Later work: Norlund, Doostmohammadi, Johansson and Kuhlmann, On the Generalization Ability of Retrieval-Enhanced Transformers (2023); Wang et al., Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023).
How could this explainer be improved? Found an error, or something unclear? I read every message.