RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Let a language model look up Wikipedia passages before it answers.

RAG connects a neural retriever over 21 million Wikipedia passages to a BART generator. It adds up the answer's probability over the passages it retrieved, and it trains both parts from question-answer pairs alone.

Nobody tells the retriever which passage was right. It learns from how much each passage raised the probability of the correct answer, and the facts stay in an index that can be swapped without retraining.

A language model trained on text stores facts in its weights. Ask a model with no access to anything else who wrote The Old Man and the Sea and the answer has to come out of those weights. There is no way to see which part of the network supplied it, no way to correct it short of more training, and when the fact was never learned well the model still produces a fluent answer, sometimes an invented one.

Retrieval-Augmented Generation (RAG), from Facebook AI Research, UCL and NYU in 2020, gives the model a second store of facts that it reads at answer time. Before writing, a retriever searches 21 million Wikipedia passages and returns the few most relevant. A generator then writes the answer once per retrieved passage, and the model adds those answers up, weighted by how relevant the retriever judged each passage. Both parts are fine-tuned together, and the training data holds only questions and answers.

The page goes through the pieces in order: the two memories, the pipeline and its shapes, how retrieval is scored, why the model sums over passages instead of picking one, the two ways to place that sum (RAG-Sequence and RAG-Token), how the retriever learns without being told which passage was right, decoding, the results, and the experiment that swaps the index. One running example carries through: the question "Who wrote The Old Man and the Sea?" with five retrieved passages.

Two kinds of memory

The paper calls knowledge held in a network's weights parametric memory. Large pre-trained models hold a lot of it. In the closed-book setting, a model answers questions from its weights alone, and the largest one the paper compares against is T5-11B, with 11 billion parameters. Parametric memory has three problems the paper names in its introduction: it cannot easily be expanded or revised, it gives no insight into which knowledge produced a prediction, and it can produce hallucinations, fluent statements that are false.

Non-parametric memory is knowledge held as data outside the weights, here a collection of text passages the model searches. Facts in it can be read, cited, added and deleted like rows in a database. The paper's target is knowledge-intensive tasks, which it defines as tasks a human could not reasonably be expected to do without access to an external knowledge source: open-domain question answering, fact verification, trivia-style question writing.

Retrieval-plus-reading systems existed before RAG. REALM and ORQA had combined a masked language model with a differentiable retriever, but only for extractive question answering, where the answer must be a span of characters copied from a retrieved passage. RAG attaches the retriever to a sequence-to-sequence generator, a model that reads an input text and writes an output text token by token. A generator can write an answer that appears in no passage, write a whole sentence, or output a class label, so one recipe covers question answering, question generation and fact verification.

Retriever, index, generator

Three pieces make up the system. The question $x$ goes into the retriever, which scores passages $z$ from the document index and returns the top $K$. The generator reads the question together with one retrieved passage at a time and scores or writes the output $y$. The paper writes the two learned parts as probability distributions: the retriever is $p_\eta(z \mid x)$, the probability that passage $z$ is the one to read, with parameters $\eta$; the generator is $p_\theta(y_i \mid x, z, y_{1:i-1})$, the probability of the next output token given the question, the passage, and the tokens written so far, with parameters $\theta$.

The passage $z$ never appears in the training data. Training pairs are (question, answer), and nothing records which passage the answer came from. A quantity the model uses but never observes is a latent variable. RAG handles it the standard way for latent variables: instead of guessing a single value, it sums over the values it considers, weighted by their probability.

The components, with the sizes from the paper's Appendix G:

• Query encoder $\mathrm{BERT}_q$, a BERT-base network (110M parameters) that maps the question to a 768-number vector. Fine-tuned.
• Document encoder $\mathrm{BERT}_d$, a second BERT-base network (110M) that maps each passage to a 768-number vector. Run once over Wikipedia to build the index, then frozen.
• Document index: 21 million passage vectors (DPR counts 21,015,324), each a 100-word chunk of the December 2018 English Wikipedia with its article title prepended. Not trainable.
• Generator: BART-large, a pre-trained encoder-decoder Transformer with 406M parameters. Fine-tuned.

The paper counts 626M trainable parameters (110M + 110M + 406M, including the document encoder it does not train). T5-11B has about 18 times as many.

To condition BART on a passage, the paper says only that it concatenates the passage and the question. The authors' HuggingFace port spells out the string: passage title, " / ", passage text, " // ", question, truncated to 300 tokens. For our question, the generator input built from the second passage is one string:

# one generator input (a single string, wrapped here to fit)
"The Old Man and the Sea / The Old Man and the Sea is a novella written by
the American author Ernest Hemingway in 1951 ... // who wrote the old man
and the sea"

BART runs on all five strings as a batch, so one question costs five generator passes. BART was pre-trained as a denoising autoencoder, reconstructing text that had been corrupted by masking, deletion and shuffling, which gives it a general ability to write fluent text conditioned on an input.

Retrieval as an inner product

The retriever has to rank 21 million passages for every question, fast, and the question and the right passage may share few words ("who wrote" against "a novella written by the American author"). RAG uses Dense Passage Retrieval (DPR), which maps text to embeddings, vectors placed so that texts with related meaning get vectors with a large inner product. DPR is a bi-encoder: one encoder for questions, $\mathbf{q}(x)$, and a separate one for passages, $\mathbf{d}(z)$, each the 768-dimensional output at BERT's [CLS] position. The score of a passage is the inner product of the two vectors, and the retriever's distribution is a softmax of those scores:

The more expressive alternative is a cross-encoder, a single network that reads the question and passage together and outputs a score (the DPR question-answering system uses one to re-rank its top passages). A cross-encoder has to be run on every (question, passage) pair at query time, which means 21 million BERT passes per question. With a bi-encoder the passage side does not depend on the question, so all 21 million $\mathbf{d}(z)$ are computed once, stored, and each question costs one BERT pass plus a search. The DPR paper chose the inner product for exactly this reason: the similarity function has to decompose so that passage representations can be precomputed.

Finding the passages with the largest $\mathbf{d}(z)^\top \mathbf{q}(x)$ is a Maximum Inner Product Search (MIPS). Scoring every stored vector exactly is 21M × 768 ≈ 16.1 billion multiply-adds per question. RAG instead builds a FAISS index with a Hierarchical Navigable Small World (HNSW) graph, which links each vector to near neighbours and answers a query by walking the graph, visiting a small fraction of the vectors. The search is approximate, and its cost grows sub-linearly with the number of passages. The paper keeps the index on CPU; it needs about 100 GB of RAM for all of Wikipedia, or 36 GB after FAISS compression (the LEANN explainer covers shrinking such an index further). In Figure 2, pick a question and a $K$ and watch which passages are retrieved and how the weight splits among them.

The $\propto$ in (1) hides a normalizing constant, and the sum in that constant runs over the top $K$ retrieved passages only, never over all 21 million:

In the code this is a log_softmax over the $K$ retrieved scores. At retrieval time DPR ranks passages by the raw inner product. RAG adds the softmax over the retrieved set because the sum over passages in the next section needs weights that add up to one. Moving $K$ in Figure 2 renormalizes the weights over whichever passages are included.

The retriever does not start from scratch. RAG initializes both encoders from a trained DPR model and builds the index with its document encoder. That DPR model was trained with retrieval supervision: for Natural Questions and TriviaQA questions it saw passages known to contain the answer, and it learned to score them above a small set of negatives (the other questions' passages in the batch plus one high-scoring BM25 passage that lacks the answer). So the retriever starts out good at finding answer-bearing Wikipedia passages, and RAG's fine-tuning adapts it to each task with no further retrieval labels.

Summing over passages

Suppose the retriever returned five passages for "Who wrote The Old Man and the Sea?". Use these illustrative numbers for the rest of the page: rank 1 is the article on the 1958 film adaptation, which names Spencer Tracy but not the author; rank 2 is the novella's own article, which names Ernest Hemingway; rank 3 is a passage of Hemingway's biography mentioning the 1953 Pulitzer Prize for the book; ranks 4 and 5 are about fishing. The retrieval scores after the softmax are

and running BART once per passage gives the gold answer "Ernest Hemingway" these probabilities:

The obvious design reads only the top passage. Here that gives the gold answer probability 0.08, because the highest-scoring passage is the near-miss. RAG instead treats the passage as a latent variable and sums it out: it adds up each passage's generator probability for the answer, weighted by that passage's retriever probability. This is marginalizing over $z$, and in the simplest variant, RAG-Sequence, one passage is assumed to be behind the whole answer:

The $\approx$ is the top-$K$ truncation: the exact marginal would sum over every passage in the index. The right-hand form writes $p_\theta(y \mid x, z)$ as the product of the per-token probabilities, which is how a sequence model scores a whole output. With the numbers above:

That is four times the top-1 probability, and most of it, $0.323 \times 0.74 = 0.239$, comes from the novella's article at rank 2. The sum also matters for training: every retriever weight appears in it, so the probability of the gold answer depends smoothly on every score, and a gradient can flow back into the retriever. A hard top-1 choice has no such gradient. Figure 3 shows the sum as a weighted vote over candidate answers, for a question whose gold answer is also Hemingway.

The sum runs over $K$ passages, and $K$ is a choice. With $K=1$ the model is the top-1 reader again. Each added passage pulls the marginal toward its own generator probability, by an amount set by its softmax weight after renormalization. A passage that contains the answer pulls it up. An off-topic passage pulls it down, and more strongly when the retriever's scores are close together, because then the softmax gives the off-topic passage a larger share. In Figure 4, with ten ranked passages, the gold passage at rank 2 and a near-miss at rank 1, slide $K$ and then change the score gap between ranks.

The paper trains with $K \in \{5, 10\}$ and reports no significant difference between the two. At test time $K$ is set on development data. For open-domain QA the reported test numbers use 15 passages for RAG-Token and 50 for RAG-Sequence; on Natural Questions, accuracy rises monotonically with more passages for RAG-Sequence and peaks at 10 for RAG-Token (the paper's Figure 3). The paper does not analyse why the two differ. One reading consistent with Figure 4: a trained retriever gives low-ranked passages small weights, so each extra passage dilutes the marginal only a little while adding a chance that it carries the answer.

RAG-Sequence and RAG-Token

Equation (2) sums over passages once, for the whole answer. Some outputs combine facts from different passages. A Jeopardy clue for Hemingway might name two of his novels, and no single retrieved passage need mention both. The second model, RAG-Token, lets a different passage account for each token by moving the sum over passages inside the product over tokens:

The retriever weights $p_\eta(z \mid x)$ are the same in (2) and (3): the question is encoded once and the passages are retrieved once. Only the position of the sum differs. In (2), each passage scores the whole answer and the scores are averaged. In (3), at each position the next-token probabilities of all passages are averaged into one mixture, the token is scored under that mixture, and the per-token scores are multiplied.

Take two passages and a two-token answer. Doc 1 supports the second token (probability 0.9, and 0.1 for the first); doc 2 supports the first token (0.9, and 0.1 for the second); the retriever weights both 0.5. RAG-Sequence gives $0.5(0.1)(0.9) + 0.5(0.9)(0.1) = 0.09$, because under each passage one of the two tokens is unlikely. RAG-Token gives $(0.05 + 0.45)(0.45 + 0.05) = 0.25$, because each token is scored under the mixture, where one passage supports it. If instead doc 1 supports both tokens and doc 2 neither, RAG-Sequence gives 0.41 and RAG-Token 0.25: summing whole-answer scores favours an answer one passage explains end to end. Figure 5 has these cases as presets.

Both forms reduce to the same thing when the output is one token. With $N = 1$ the product has a single factor and the sum can go on either side of it, so the paper treats classification as generating a one-token answer: for FEVER, the labels supports, refutes and not-enough-info are each mapped to a single output token, and the two models are identical. In code, the only difference is the order of a sum over tokens and a logsumexp over passages:

# one question, K = 5 passages (shapes as in the HuggingFace port)
q      = query_encoder(x)                  # (768,)  BERT-base [CLS] vector
ids, D = index.search(q, k=5)              # D: (5, 768) stored doc vectors
s      = D @ q                             # (5,)  inner products d(z).q(x)
log_pe = log_softmax(s)                    # (5,)  over the 5 retrieved only
ctx    = [title[i] + " / " + text[i] + " // " + x for i in ids]
logits = bart(ctx, y)                      # (5, N, vocab): one pass per passage
log_pt = gather(log_softmax(logits), y)    # (5, N): log p_theta(y_i | x, z, y_<i)

# RAG-Sequence: product over tokens first, then the sum over passages
log_p_seq = logsumexp(log_pe + log_pt.sum(1))
# RAG-Token: sum over passages at every token, then the product over tokens
log_p_tok = logsumexp(log_pe[:, None] + log_pt, dim=0).sum()

The paper shows RAG-Token combining passages on a real generation. Given the answer entity "Hemingway" with five retrieved passages, the model wrote the Jeopardy clue "The Sun Also Rises" is a novel by this author of "A Farewell to Arms". For each token, the paper plots which passage most likely produced it, the posterior over the passage given that token:

The prior $p_\eta(z \mid x)$ is fixed for the whole clue; only the second factor changes from token to token. So the posterior moves away from the prior exactly where the passages disagree about the next token. The paper reports that passage 2, which mentions The Sun Also Rises, dominates the posterior when "Sun" is generated, and passage 1 dominates when A Farewell to Arms is generated. After the first token of each title the posterior flattens. Figure 6 rebuilds this with an illustrative table of per-token probabilities; the paper publishes the plot but not its numbers.

The flattening is the paper's evidence that the two memories divide the work. Once "The Sun" is written, every passage gives "Also" about the same high probability, because the title is stored in BART's weights. The authors checked this with a BART model trained without retrieval: fed the partial clue "The Sun, it completed "The Sun Also Rises" is a novel by this author of "The Sun Also Rises", and fed the clue up to "A it completed "A Farewell to Arms". The paper reads this as the retrieved passage guiding the generation and drawing out knowledge already stored in the generator's parameters. In the RAG-Sequence mode of Figure 6, doc 2 takes 0.75 of the single posterior because it explains the whole clue best, and the whole clue is 1.75 times more probable under RAG-Token than under RAG-Sequence.

Training without retrieval labels

The training set is pairs $(x_j, y_j)$ of inputs and target outputs. RAG minimizes the negative log of the marginal probability of each target, summed over the pairs, with Adam:

where $p(y \mid x)$ is (2) or (3). For the running example the loss is $-\log 0.319 = 1.14$ nats (natural-log units); a top-1 model would have $-\log 0.08 = 2.53$. Nothing in (4) mentions which passage was correct, yet minimizing it changes the retriever in a specific direction. To see which, write the retrieval scores as $s_z = \mathbf{d}(z)^\top \mathbf{q}(x)$ and put the softmax of (1) into (2). The log marginal is then a difference of two log-sums:

The derivative of a log-sum-of-exponentials with respect to one score is that term's share of the sum. The first term's shares are the retriever weights reweighted by how well each passage predicts $y$, which is the posterior over passages given the answer, $p(z \mid x, y)$; the second term's shares are the retriever weights themselves:

Gradient ascent on the log-likelihood raises the score of a passage whose posterior exceeds its prior, which happens exactly when $p_\theta(y \mid x, z) > p(y \mid x)$: the passage predicted the gold answer better than the current marginal does. For the running example the posterior is $(0.121, 0.749, 0.125, 0.004, 0.001)$, so the gradients on the five scores are:

The film article (answer probability 0.08, below the marginal 0.319) is pushed down, the novella article pushed up. The biography passage, at 0.41, is just above the marginal and gets a small push up. The gradients sum to zero because the posterior and the prior each sum to one. For RAG-Token the log-likelihood is a sum over tokens, so the gradient on $s_z$ is the sum over tokens of the per-token posterior from the Hemingway example minus the prior. Figure 7 runs these gradient steps on five free scores.

In the real model the scores are not free parameters. Each is an inner product with the same query vector, and the document vectors are fixed, because RAG freezes the document encoder and the index. The gradient reaches the query encoder through $\mathbf{q}(x)$, and by the chain rule ($\partial s_z / \partial \mathbf{q} = \mathbf{d}(z)$) it is

The query moves toward the posterior-weighted average of the retrieved document vectors and away from the prior-weighted average. Two consequences follow from the scores sharing one vector. Moving the query toward a helpful passage also raises the score of every passage whose vector points the same way, so a neighbour can enter the top $K$. And a passage outside the top $K$ gets no gradient at all, however useful it would have been, because it is not in the sum. Figure 8 shows both.

The document encoder is frozen to save computation. If $\mathrm{BERT}_d$ were trained, every update would make the stored vectors stale, and keeping the index valid would mean re-encoding 21 million passages and rebuilding the index periodically, which is what REALM does during its pre-training. The paper reports that this step is not necessary for strong performance and trains only $\mathrm{BERT}_q$ and BART. In code, a training step continues the forward pass shown earlier:

# one training step on a (question, answer) pair; nothing says which passage
loss = -log_p_seq             # or -log_p_tok: one term of equation (4)
loss.backward()
#   BART's 406M weights get a gradient through log_pt
#   the query encoder gets one through s = D @ q
#   D was read out of the index as plain numbers: the document encoder gets none
opt.step()                    # Adam on the query encoder and BART

The paper checks that the learned retrieval matters by freezing the query encoder too. On the Natural Questions development set, RAG-Sequence drops from 44.0 to 41.2 exact match and RAG-Token from 43.5 to 37.8. Replacing the dense retriever with BM25, a classic word-overlap ranking function whose scores the paper uses as the retrieval logits, drops them to 31.8 and 29.7. Training ran in Fairseq with mixed precision on eight 32 GB V100 GPUs.

The label-free signal can also fail. In preliminary experiments on story generation, the retriever "collapsed" to returning the same passages for every input, after which the generator learned to ignore them and the model performed like plain BART (Appendix H). The authors suggest two causes: story generation needs less factual knowledge, and long targets could give less informative gradients to the retriever. When no retrieved passage predicts the target better than the others, the posterior in (5) stays equal to the prior and the scores receive no consistent push.

Decoding

At test time the model has to produce an output, approximately $\arg\max_y p(y \mid x)$, and the two variants need different procedures. RAG-Token fits a standard decoder: its per-token mixture is an ordinary next-token distribution,

so it plugs into beam search unchanged. RAG-Sequence's probability does not factor into per-token terms, because the sum over passages sits outside the product. The paper runs a separate beam search for each passage, collects every hypothesis from every beam into a candidate set, and then needs each candidate's probability under every passage. Thorough Decoding runs an extra forward pass for each (candidate, passage) pair whose candidate did not appear in that passage's beam. Fast Decoding skips those passes and treats $p_\theta(y \mid x, z_i)$ as 0 when $y$ was not in passage $z_i$'s beam. The paper used Thorough Decoding with greedy decoding for open-domain QA, where answers are short, and Fast Decoding with beam size 4 for MS-MARCO and Jeopardy, where Thorough Decoding did not improve the scores.

Results

Open-domain QA gives the model only a question, and the score is exact match (EM): the percentage of questions whose generated answer string equals a gold answer. The paper's Table 1, with the closed-book models (parametric memory only) and the retrieve-and-extract systems:

RAG sets the state of the art on Natural Questions (44.5 for RAG-Sequence against DPR's 41.5), WebQuestions (45.5 for RAG-Token) and CuratedTrec (52.2 for RAG-Sequence). TriviaQA has two test sets. On the TQA-Wiki set used by T5, RAG-Sequence scores 68.0 against 60.5 for T5-11B+SSM. On the standard open-domain test set, DPR's 57.9 beats RAG-Sequence's 56.8. The DPR system compared here uses a BERT cross-encoder to re-rank passages and an extractive reader; RAG uses neither. T5-large, the T5 model closest to RAG's size at 770M parameters, scores 28.9 on NQ.

Because RAG generates the answer, a passage can help without containing the answer string, and the model can be right when no passage contains it. On Natural Questions, in cases where the correct answer appears in none of the retrieved passages, RAG is still right 11.8% of the time. An extractive reader, which can only copy a span, scores 0% on those cases by construction.

On generation tasks the baseline is BART fine-tuned without retrieval. For abstractive question answering on MS-MARCO (the NLG v2.1 task, used here without its supplied gold passages), RAG-Sequence beats BART by 2.6 points on both BLEU-1 (44.2 against 41.6) and ROUGE-L (40.8 against 38.2), metrics that count word and subsequence overlap with the reference answer. For Jeopardy question generation, where the input is an answer entity and the output a clue, RAG-Token scores 22.2 on Q-BLEU-1 (a BLEU variant that up-weights matching entities), RAG-Sequence 21.4 and BART 19.7. Human raters compared 452 pairs of BART and RAG-Token clues: they judged RAG more factual in 42.7% of pairs and BART in 7.1%, and RAG more specific in 37.4% against 16.8%. RAG's clues are also more varied: 53.8% of RAG-Sequence's trigrams are distinct and 46.8% of RAG-Token's, against 32.4% of BART's, with no diversity-promoting decoding.

FEVER asks whether Wikipedia supports or refutes a claim, or has not enough information. RAG reaches 72.5% label accuracy on the three-way task, 4.3 points below the state of the art (76.8), and 89.5% on the two-way task, 2.7 below 92.2. Those systems are pipelines trained with supervision on which evidence to retrieve, and the two-way one is given the gold evidence sentence; RAG sees only the claim and retrieves its own evidence. Its top retrieved passage comes from a gold evidence article 71% of the time, and a gold article is in the top 10 in 90% of cases.

Swapping the index, and limits

Facts in the index can be changed without training. The paper built a second index from the December 2016 Wikipedia dump (the one used by DrQA), listed 82 world leaders who changed between December 2016 and December 2018, and asked the Natural Questions RAG model "Who is {position}?" (for example, "Who is the President of Peru?") with each index. In Figure 9, pick a position and an index; the table under the flow has the paper's measured accuracies.

If the model answered from its weights, swapping the index would leave the answers unchanged, and the 2016-index model would do as well on 2018 leaders as the 2018-index model. Accuracy instead falls from 68% to 4%, so the answer comes from the retrieved passage. A purely parametric model such as T5 or BART needs further training to change these answers. The same property makes RAG's answers checkable: the retrieved passages can be shown next to the output.

The paper's results come with limits:

• RAG hallucinates less than BART, not never. The evidence is qualitative examples (Table 3) and the Jeopardy human evaluation. A wrong or biased passage in the index can produce a wrong answer that looks grounded.
• "No retrieval labels" describes RAG's fine-tuning. The retriever it starts from was trained with retrieval supervision on Natural Questions and TriviaQA.
• Dense retrieval is not always the better retriever. On FEVER, BM25 scores 75.1 and 91.6 (three-way and two-way, dev set) against 74.5 and 90.6 for the learned retriever; the paper suggests FEVER's entity-heavy claims suit word-overlap matching.
• Retrieval can collapse on tasks with weak factual signal, as described in the training section.
• RAG has no way to decline retrieval. The authors tried a REALM-style "null document" with a learned logit and found it did not help; on MS-MARCO the model learned to retrieve a fixed set of passages for questions that benefit little from retrieval (Appendix F).

The paper's proposed next step is to pre-train the retriever and generator jointly from scratch, for example with a denoising objective like BART's. The recipe it did establish has four parts: a bi-encoder retriever over a fixed dense index, a pre-trained seq2seq generator run once per retrieved passage, a sum over the top $K$ passages either per answer or per token, and training on (input, output) pairs alone, where the gradient (5) moves the query encoder toward passages that raised the probability of the target.