T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Every NLP task becomes the same task: text in, text out.

A Google team wrote translation, classification, question answering and summarization in one string-to-string format, used it to compare pre-training objectives, architectures and data sets one factor at a time, and then trained the winning recipe at up to 11 billion parameters.

The authors propose no new method. They contribute a controlled comparison of training choices, a 750 GB cleaned web corpus called C4, and five models, the largest of which set the state of the art on 18 of the 24 tasks they measured.

By 2019 the standard recipe for language tasks was transfer learning: pre-train a large network on unlabeled text, then fine-tune it on a small labeled data set for the task you care about. BERT pre-trained an encoder by filling in masked words and attached a small classifier for each task. GPT-2 trained a decoder to predict the next word. XLNet, RoBERTa, UniLM, MASS and SpanBERT each changed the architecture, the pre-training objective, the corpus and the amount of training, usually several at once. When one of them beat the last, nobody could say which change had produced the gain.

The T5 paper sets out to answer that question by experiment. It fixes one baseline model and one training procedure, changes a single factor at a time (the architecture, the objective, the corpus, the multi-task mixture, the use of extra compute), and scores every variant on the same benchmarks. That only works if every task shares the same model, loss and decoding, so the paper first rewrites every task as text-to-text. At the end the best choices are combined and trained at five sizes, and those models are T5, the Text-to-Text Transfer Transformer.

The page follows the paper's order: the text-to-text format; the baseline Transformer and its three small changes; the comparison of attention masks; the pre-training objectives and the span-corruption objective T5 adopted; the C4 corpus; multi-task training; how to spend four times the compute; and the final models and where they fell short.

Every task is text in, text out

A sentiment classifier, a translator and a question-answering system normally have different output layers, different losses and different decoding procedures. T5 gives every task the same interface. The input is a string, the target is a string, and the model is trained to produce the target given the input.

A task prefix at the start of the input says which task is meant. The prefix is ordinary text, tokenized like the rest of the input, with no special machinery behind it. In training, every input that begins "translate English to German:" is paired with a German target, so the model learns that those words predict German output, the same way any context shapes what a language model writes next. The two examples at the top of the listing are verbatim from the paper; the rest follow the same format:

# every task is one (input string -> target string) pair
"translate English to German: That is good."   ->  "Das ist gut."
"mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons
    are filled with animosity."                ->  "entailment"
# illustrative, same format:
"cola sentence: The professor talked us."       ->  "unacceptable"
"stsb sentence1: ... sentence2: ..."            ->  "3.8"
"summarize: <a news article>"                   ->  "<a short summary>"

For a classification task the target is the label written as a word. MNLI gives a premise and a hypothesis and asks whether the premise entails the hypothesis, contradicts it, or neither, so the target is one of entailment, contradiction or neutral. Regression needs one more step. STS-B scores the similarity of two sentences from 1 to 5; T5 rounds each score to the nearest 0.2, so a human rating of 3.84 becomes the four-character target 3.8. That leaves 21 possible strings (1.0, 1.2, and so on up to 5.0), which turns the regression into a 21-class problem. An output that is not a valid label or number is scored as wrong; the authors report they never saw one. The Winograd pronoun tasks are recast so the model writes out the noun that a starred pronoun refers to: for "The city councilmen refused the demonstrators a permit because *they* feared violence", the target is "The city councilmen".

Because every target is a token sequence, one loss covers every task, in pre-training and in fine-tuning. For an input $x$ and a target $y = (y_1, \dots, y_T)$, the model is trained by maximum likelihood with teacher forcing: at step $t$ the decoder is fed the true previous target tokens, not its own guesses, and the loss adds up the cross-entropy of each true next token:

At test time the baseline decodes greedily, taking the most likely token at each step. For MNLI the target is one or two tokens, so "classification" is a single step of the same decoder that translates.

The obvious alternative is BERT's: share a pre-trained body and give each task its own output head and loss. Then a comparison of two pre-training objectives also compares whatever heads and losses were tuned for each. With one format, one loss and one decoder, the pre-training objective, architecture or corpus can change while everything around it stays fixed, and the rest of the paper uses that to run controlled comparisons.

The baseline Transformer

The model under the format is the encoder-decoder Transformer from Attention Is All You Need, kept close to its original form. The encoder reads the input tokens, and each position may look at every other position. The decoder writes the target one token at a time; it looks back at the target tokens produced so far and, through cross-attention, at the encoder's output. Both stacks are built from blocks of self-attention followed by a small feed-forward network. Self-attention replaces each position with a weighted average of the positions it may attend to:

In (2), $x_j$ is the vector for position $j$ (after a learned value projection), $q_i$ and $k_j$ are learned query and key projections, and $w_{ij}$ is how much position $i$ takes from position $j$. The softmax makes the weights non-negative and sum to one over $j$. Each layer runs several of these heads in parallel on smaller projections and concatenates their outputs. Cross-attention is the same operation with the queries from the decoder and the keys and values from the encoder output. The original Transformer divides the logit by $\sqrt{d_{kv}}$ to keep its size independent of the head width; the T5 code applies no such factor and instead initializes the query projection with a standard deviation of $(d_{\text{model}} d_{kv})^{-1/2}$, which has the same effect at the start of training.

The baseline makes each stack the size of BERT-Base: 12 blocks, a model width of $d_{\text{model}} = 768$ (the length of every token vector), 12 heads of width $d_{kv} = 64$, and feed-forward layers of width $d_{\text{ff}} = 3072$. With two stacks the model has about 220 million parameters, roughly twice BERT-Base. In shapes: a 512-token input becomes a $512 \times 768$ matrix; each head projects it to $512 \times 64$ queries and keys and forms a $512 \times 512$ matrix of logits, the scores that go into the softmax; the 12 heads' $512 \times 64$ outputs concatenate back to $512 \times 768$. The vocabulary is one SentencePiece model of 32,000 wordpieces (subword units, so a rare word splits into several pieces), trained on English C4 mixed 10 to 1 with each of German, French and Romanian so the same tokens cover the translation tasks.

T5 changes the original design in three places. The paper calls the changes orthogonal to its study and leaves ablating them to future work, so the comparisons below say nothing about whether they help.

The first change simplifies layer normalization. Standard layer norm subtracts each vector's mean, divides by its standard deviation, then multiplies by a learned gain $\gamma$ and adds a learned bias. T5 keeps only the rescaling:

For $x = (2, 4, 4, 6)$, the mean of the squares is $(4 + 16 + 16 + 36)/4 = 18$, so (3) divides by $\sqrt{18} \approx 4.24$ and returns $(0.47, 0.94, 0.94, 1.41)$ before the gain. Standard layer norm would first subtract the mean 4 and return $(-1.41, 0, 0, 1.41)$. Both outputs have a root-mean-square of 1, which bounds the scale of what enters the next layer; only standard layer norm also removes the offset. The paper describes the change as rescaling only, with no additive bias; the missing mean subtraction is visible in the code. The same operation appeared a week earlier, in October 2019, under the name RMSNorm, in an independent paper that T5 does not cite, and LLaMA and many later models use it.

The second change moves the normalization to the input of each sublayer. Each attention or feed-forward sublayer $F$ computes

so the residual path from block to block carries $x$ without passing through any normalization. The original Transformer normalized after the addition, $\mathrm{LN}(x + F(x))$. This "pre-norm" placement became common in later models; T5 itself gives no reason for it and does not test it.

The third change replaces absolute position embeddings with a relative position bias. Self-attention on its own treats its input as an unordered set, so the model needs some position signal. T5 adds one learned scalar to each attention logit, chosen by the offset between query and key:

Offsets are sorted into 32 buckets, 16 for each sign of $i - j$. Within a sign, offsets 0 through 7 get a bucket each; after that the bands widen logarithmically (8–11, 12–15, 16–22, 23–31, 32–45, 46–63, 64–90), and a final bucket takes every offset of 91 or more. The paper describes the log ranges as reaching 128; rounding down in the code starts the last bucket at 91, and offsets from 91 to 128 and beyond all share its scalar. Each head $h$ has its own 32 scalars, and the same table is used by every layer, so the baseline encoder's position scheme has $32 \times 12 = 384$ numbers in total. A single layer cannot tell an offset of 100 from one of 400, but stacked layers can combine nearby offsets into longer ones. This is a reduced form of the learned relative embeddings of Shaw et al., which added a learned vector per offset instead of a scalar.

Drag the offset below and read off its bucket. An offset of 5 gets a bucket to itself, 90 shares a band 27 offsets wide, and everything from 91 up lands in the last one.

The baseline is pre-trained for $2^{19} = 524{,}288$ steps on batches of 128 sequences of 512 tokens, about $2^{16} = 65{,}536$ tokens per batch, which comes to $2^{35} \approx 34$ billion tokens. That is about a quarter of BERT's pre-training, and small enough that the baseline never sees any C4 text twice. The optimizer is AdaFactor, a memory-saving relative of Adam, with a learning rate of 0.01 for the first $10^4$ steps that then falls as $1/\sqrt{n}$ in the step number $n$. Fine-tuning runs $2^{18}$ more steps at a constant 0.001.

Every experiment reports the same benchmarks. GLUE averages nine English understanding tasks (grammaticality, sentiment, paraphrase, entailment and others) into one score; SuperGLUE is a harder eight-task successor built to stay difficult for pre-trained models; SQuAD is reading comprehension scored by exact match with a reference answer; CNN/Daily Mail is news summarization scored by ROUGE-2 (word-pair overlap with a reference summary); and three WMT translation tasks, English to German, French and Romanian, are scored by BLEU (n-gram overlap with reference translations). The baseline scores 83.28 on GLUE and 71.36 on SuperGLUE, and those two numbers recur below as the reference point. The same network fine-tuned with no pre-training scores 66.22 and 53.04.

Three attention masks

The text-to-text format does not require an encoder and a decoder. A single decoder-only stack, the shape of GPT-2, can do it too: glue the input and target into one sequence and train on next-token prediction. The paper gives the translation example as the single sequence "translate English to German: That is good. target: Das ist gut." To predict, the model is fed everything up to "target:" and generates the rest. The architectures differ mainly in their attention mask, the rule for which positions may attend to which.

A mask sets some weights $w_{ij}$ in (2) to zero. A fully visible mask zeroes none, so every position sees every other; the encoder uses it. A causal mask zeroes $w_{ij}$ for every $j > i$, so position $i$ sees only itself and earlier positions; the decoder uses it so that training cannot peek at the tokens it must predict. A causal mask with a prefix is fully visible over the first part of the sequence and causal after it.

The three masks give three architectures. The encoder-decoder has a fully visible encoder and a causal decoder joined by cross-attention. The language model is one causal stack over input plus target. The prefix LM is one stack whose mask is fully visible over the input and causal over the target. In the figure, look at the input block in the upper left. The language model's is a triangle, so the vector for the third input word is built without the fourth and fifth, even though all of the input is known before any output is written; the other two fill the block.

Comparing the architectures fairly needs a common budget, and the paper notes that an encoder-decoder and a single stack cannot be matched on parameters and compute at the same time. Let $P$ be the parameters of one BERT-Base-sized stack of $L = 12$ layers, about 110 million. An encoder-decoder with $L$ layers in each stack has about $2P$, the same as a 24-layer language model. Its compute per example, though, matches a 12-layer language model, because each token passes through only one of its stacks: input tokens through the encoder, target tokens through the decoder.

A forward pass costs roughly 2 FLOPs per parameter per token it passes through (a common rule of thumb that ignores the attention terms that grow with sequence length). Take an example with 400 input tokens and 100 target tokens. The encoder-decoder costs about $400 \times 2 \times P + 100 \times 2 \times P = 1000P$, about $1.1 \times 10^{11}$ FLOPs. A 12-layer language model runs all 500 tokens through its $P$ parameters: also $1000P$. A 24-layer language model with $2P$ parameters runs all 500 tokens through all of them: $2000P$. The paper calls this common cost $M$ and confirms it empirically: the step times of $L$-layer language models and $L+L$ encoder-decoders were nearly identical. Cross-attention adds about 10% to the encoder-decoder's parameters, which the paper's $2P$ ignores.

With the denoising objective (Table 2), the encoder-decoder ($2P$, $M$) scored 83.28 on GLUE and was the best configuration on every benchmark. Sharing weights between its encoder and decoder ($P$, $M$) cost little, 82.81. Halving each stack to 6 layers ($P$, $M/2$) cost more, 80.88. The prefix LM ($P$, $M$) scored 81.82 and the plain language model 74.70. The shared encoder-decoder and the prefix LM have the same parameters and compute, so the paper reads the 1-point gap between them as a benefit of explicit cross-attention. The prefix LM's 7-point lead over the language model comes from the one difference in their masks, the visibility of the input.

The same table repeats every architecture with a plain language-modeling objective. On GLUE, denoising beat it in every architecture: 83.28 against 79.56 for the encoder-decoder, 81.82 against 79.68 for the prefix LM, 74.70 against 73.78 for the language model. Per benchmark there are exceptions, such as the language model scoring 56.51 on SuperGLUE with its own objective and 55.02 with denoising.

Seven pre-training objectives

A pre-training objective turns unlabeled text into (input, target) pairs, so the model can train with loss (1) on text nobody labeled. The older choice was language modeling: split the text and predict the continuation. By 2019 most strong models used a denoising objective instead: corrupt the input and train the model to repair it, as in BERT's masked language modeling.

The paper's Table 3 shows each objective it tested on one sentence, "Thank you for inviting me to your party last week." The figure reproduces the table. For each objective, compare two things: what the encoder still sees, and how many tokens the decoder has to write. Decoder tokens cost training time, because every target token is one more step of the decoder and one more term in (1).

The objectives fall into three groups. The prefix LM objective splits the text at a random point, gives the first part to the encoder and trains the decoder to write the second. Deshuffling shuffles the tokens and asks for the original order. The BERT-style objective corrupts 15% of the tokens, replacing 90% of those with a mask token and 10% with a random token (the sentence shows "last" turned into "apple"), and the decoder writes out the entire original sentence.

T5's BERT-style objective differs from BERT in two ways. BERT chooses 15% of positions and turns 80% of them into [MASK], 10% into a random token, and leaves 10% unchanged. BERT is an encoder only, so it predicts just the chosen positions, each from the encoder's output vector at that position. T5 has a decoder, so its version asks the decoder for the entire uncorrupted sequence. The remaining rows vary the corruption. MASS-style drops the random swaps and keeps only mask tokens. Replace spans collapses each run of corrupted tokens into one sentinel token and has the decoder write only the corrupted tokens, each run after its sentinel. Drop tokens deletes the corrupted tokens with no marker and asks for them in order. Random spans is replace spans with the corruption chosen as contiguous spans rather than as independent tokens. The last three write 3 to 9 tokens for this sentence where the full-reconstruction objectives write 11.

Span corruption

The objective T5 ends up with is span corruption. Pick 15% of the tokens to corrupt, grouped into contiguous spans with a mean length of 3. Replace each span in the input with a sentinel, a special token that appears once in the example (the paper writes them <X>, <Y>, <Z>; the released vocabulary calls them <extra_id_0>, <extra_id_1> and so on). The target lists each span after its sentinel and ends with one more sentinel; the tokens left alone get no prediction.

The paper's own worked example uses a 500-token sequence. Corrupting 15% gives $500 \times 0.15 = 75$ tokens; asking for 25 spans makes their mean length $75 / 25 = 3$. Counting sentinels, the encoder input has $500 - 75 + 25 = 450$ tokens and the target has $75 + 25 + 1 = 101$. An objective that rebuilds the whole text would need a 500-token target, five times as long. The released code sets the counts from the rate and the mean span length and then places the spans at random:

# span corruption on one training sequence (rate 0.15, mean span 3)
n        = len(tokens)                       # 512 in pre-training
n_noise  = round(n * 0.15)                   # 77 tokens to corrupt
n_spans  = max(1, round(n_noise / 3))        # 26 spans
mask     = random_spans(n, n_noise, n_spans) # spans at random positions
src, tgt, k = [], [], 0
for run in runs(tokens, mask):               # alternating kept / corrupted runs
    if run.corrupted:
        src += [sentinel(k)]                 # the whole span becomes one token
        tgt += [sentinel(k)] + run.tokens    # the target spells the span out
        k += 1
    else:
        src += run.tokens                    # kept tokens stay in place
tgt += [sentinel(k)]                         # a final sentinel ends the target
# len(src) = 512 - 77 + 26 = 461, len(tgt) = 77 + 26 + 1 = 104
# loss: cross-entropy on tgt only, teacher-forced through the decoder

In the figure, the sliders set the same two numbers the code takes. The passage shows one sampled mask; the bars underneath apply the same counting to a real 512-token sequence. Raise the mean span and the number of sentinels falls, so both the input and the target get shorter. Raise the rate and the target grows: at 50% with spans of 3, a 512-token example has a 342-token target and a 341-token input, and the target is no longer the short side. At 0% nothing is corrupted and the target is a lone sentinel with nothing to predict.

The baseline used for the paper's comparisons corrupts independent tokens ("i.i.d." in the tables), which is the replace-spans row of Figure 4: 15% of tokens chosen one at a time, with any that happen to be adjacent merged under one sentinel. At a 15% rate, independent choices produce mostly one-token runs, so the baseline needs more sentinels than a mean span of 3 does. The final models switch to span corruption with mean length 3, for a small score gain and slightly shorter targets.

The objective search

The paper chose the objective in four stages, fixing each decision before moving to the next (its Figure 5): the high-level approach, then how to corrupt, then the corruption rate, then the span length. Each variant is pre-trained and fine-tuned like the baseline, so the scores differ only through the objective. To tell a real difference from run-to-run noise, the authors trained the baseline 10 times from scratch: its GLUE score has a standard deviation of 0.235 and its SuperGLUE score 0.416, and the tables bold every score within two standard deviations of the best. In the figure, each stage has a fixed axis for its metric so the spreads can be compared across stages; switch to SuperGLUE to see where it disagrees with GLUE.

Stage one compares the three families. BERT-style denoising scored 82.96 on GLUE, the prefix LM 80.69, and deshuffling 73.17. The stage spans almost ten points, but most of that is deshuffling's deficit; denoising led prefix language modeling by 2.27 GLUE points and 4.58 SuperGLUE points (69.85 against 65.27).

Stage two varies the corruption scheme. On GLUE the four variants span 2.1 points: MASS-style 82.32, BERT-style 82.96, replace spans 83.28 and drop tokens 84.44. Dropping tokens owes its GLUE lead to CoLA, the grammaticality task (60.04 against the baseline's 53.84), which the authors suggest is related to noticing that words are missing; on SuperGLUE it falls to 68.67 against replace spans' 71.36. The paper calls the variants similar and continues with replace spans: it and drop tokens both avoid predicting the full sequence, so they train faster, and replace spans held up better on SuperGLUE.

Stage three tries corruption rates of 10%, 15%, 25% and 50%. GLUE reads 82.82, 83.28, 83.00 and 81.27: little change until 50%, which also lowers SQuAD from 80.88 to 79.80 and lengthens the targets. T5 keeps 15%, which is also BERT's rate. Stage four replaces independent corruption with spans of mean length 2, 3, 5 and 10. The GLUE scores all sit within 0.7 of each other, about three baseline standard deviations, but span length 3 is best on SuperGLUE (72.53 against the i.i.d. 71.36) and on CNN/Daily Mail, and the paper calls its gain over i.i.d. small but significant.

Among denoising variants, then, no choice moved GLUE by more than about two points, and the paper picks among them partly on cost. The earlier decision, denoising instead of predicting a continuation, was worth 2 to 4 GLUE points: 2.27 in stage one, and 3.72 for the encoder-decoder in the architecture comparison (83.28 against 79.56).

C4, the cleaned web crawl

The corpus is its own variable, and the paper built one it could vary. C4, the Colossal Clean Crawled Corpus, starts from Common Crawl, a public archive that scrapes the web and publishes about 20 TB of extracted text each month. Most of that text is menus, error messages, boilerplate and duplicates. The authors took the April 2019 crawl and filtered it with a list of simple rules. At the line level: keep only lines that end in a period, exclamation mark, question mark or closing quotation mark; keep only lines with at least 5 words; drop any line containing the word "Javascript", since many pages carried warnings that it should be enabled; drop lines with cookie and policy notices ("terms of use", "privacy policy", "uses cookies" and similar); and delete Wikipedia-style citation markers such as [1]. At the page level: discard any page with fewer than 3 sentences, any page containing a curly bracket (a sign of source code), any page containing "lorem ipsum", and any page with a word from a public list of obscene and offensive words. Then deduplicate across the corpus, keeping one copy of any three-sentence span that occurs more than once, and keep only pages that the langdetect tool classifies as English with probability at least 0.99.

What survives is about 750 GB of English text, orders of magnitude more than most pre-training corpora then in use, and the authors released it through TensorFlow Datasets. The filters change results. Pre-training the baseline on the same crawl without them (6.1 TB) lowered GLUE from 83.28 to 81.46 and SuperGLUE from 71.36 to 68.04, and the unfiltered version scored worst of the corpora on every task.

Smaller corpora from one domain sometimes did better (Table 8). A 20 GB mix of Wikipedia and the Toronto Books Corpus raised SuperGLUE to 73.24, and almost all of that came from one task: MultiRC, a reading-comprehension set whose largest source is fiction, rose from 25.78 to 50.93 exact match. A 35 GB news-only corpus raised the news reading task ReCoRD from 68.16 to 73.72.

A single domain usually means a small corpus, and small corpora have a cost in long training runs. Table 9 truncates C4 and pre-trains for the same $2^{35}$ tokens, so the data repeats. With $2^{29}$ tokens, each token is seen 64 times and GLUE barely moves (82.87). With $2^{25}$ tokens, about 34 million, each is seen 1,024 times and GLUE falls to 79.55; at $2^{23}$, 4,096 repeats, it falls to 76.34 and SuperGLUE to 59.29. The training loss drops lower as the corpus shrinks, which the paper reads as the model starting to memorize its data. The final T5 runs see about a trillion tokens, which would repeat the 20 GB Wikipedia-and-books corpus hundreds of times, so they stay on C4.

Training on many tasks

In the text-to-text format, training on several tasks at once means mixing their (input, target) pairs into one stream, with the unsupervised objective as one more task. The open question is the mixing rate. Sampling in proportion to data set size would give almost every batch to the unlabeled corpus, which is orders of magnitude larger than any labeled set. The paper's examples-proportional mixing caps every data set at an artificial size $K$ before computing the proportions. With $e_n$ examples in task $n$, task $m$ is sampled with probability

With $K = 2^{19}$, a task with a million examples and a task with two million get equal shares, and a task with 100,000 examples gets about a fifth of their share each. Trained on such a mixture for the same total number of steps as the baseline's pre-training plus fine-tuning, a multi-task model scored 81.42 on GLUE and 67.30 on SuperGLUE, below the baseline's 83.28 and 71.36 (Table 12). Pre-training on the multi-task mixture and then fine-tuning on each task recovered the difference: 83.11 and 71.03. Leaving the target task out of the mixture and fine-tuning on it afterwards scored 81.98 and 71.68. Pre-training on the supervised tasks alone, with no unsupervised objective, scored 79.93 and 65.36 and was worse everywhere except translation.

The final models use multi-task pre-training followed by fine-tuning. It matched unsupervised pre-training in Table 12, and it lets the authors watch downstream scores throughout pre-training.

Spending 4× compute

Everything above holds the compute fixed. The paper then asks a direct question: you were just given 4× more compute; how should you use it? The premise is Rich Sutton's "bitter lesson", that general methods which use more computation have tended to win over hand-built ones, and the experiment compares ways of using it. The bigger models follow BERT-Large's shape ($d_{\text{model}} = 1024$, 16 heads) with 16 or 32 layers per stack, for about 2× and 4× the baseline's parameters and compute.

The options are: train 4× as many steps, train with a 4× batch, train a 2× model for 2× the steps, train a 4× model for the baseline number of steps, ensemble four separately trained baselines, or, as a cheaper variant that does not spend the full 4×, pre-train once and ensemble four models fine-tuned from that single checkpoint. On SuperGLUE the baseline scores 71.36; 4× steps gives 74.72 and 4× batch 74.64, with no clear winner between them; the 2× model for 2× steps gives 77.18 and the 4× model 78.04. On GLUE the order of the last two flips (86.18 against 85.91). The paper found no large difference between them on any task and concludes that training longer and training bigger are complementary, and that a bigger model gives an extra gain beyond more steps or a bigger batch.

Ensembling is a separate axis. Four independently trained models, averaged, were the best of all the options on CNN/Daily Mail, English-to-German and English-to-Romanian, but neither ensembling variant significantly improved SuperGLUE (71.74 against 71.36). The section is a table of measured tradeoffs, not a fitted law: the power-law fits of Kaplan et al. (January 2020) and Chinchilla (2022) came after it.

The final models

The final recipe combines what the study found:

• span corruption with mean span length 3 and a 15% corruption rate;
• pre-training for 1 million steps on batches of $2^{11}$ sequences of 512 tokens, about 1 trillion tokens, 32 times the baseline, all from C4;
• multi-task pre-training on a mixture of the unsupervised objective and all downstream tasks, then fine-tuning on each GLUE and SuperGLUE task separately with a batch of 8 sequences, so small tasks do not repeat inside a batch;
• beam search (keeping the 4 best partial outputs at each step instead of one, with a length penalty of 0.6) for the long outputs of WMT translation and CNN/Daily Mail, and greedy decoding everywhere else;
• five model sizes.

From Large up, the width $d_{\text{model}}$ stays at 1,024. The 3B and 11B models grow mainly by widening the feed-forward layers, to 16,384 and 65,536, with 128-wide heads ($d_{kv} = 128$). The paper chose $d_{\text{ff}}$ because TPUs are most efficient on large dense matrix multiplications, and those are what the feed-forward layers compute.

On SuperGLUE, T5-11B scored 88.9, up from the previous best of 84.6 and 0.9 below the human baseline of 89.8, and it was best on the SuperGLUE average and on each of its eight tasks. On GLUE it averaged 90.3 against a previous best of 89.4, though it trailed earlier systems on three GLUE tasks: MRPC (paraphrase detection), QNLI (question answering posed as entailment; 96.9 against 99.2) and QQP (duplicate Quora questions) accuracy. On SQuAD it reached 91.26 exact match and 96.22 F1. Human exact match on SQuAD is estimated at 82.30, so the paper doubts further gains there mean much. It also set the record on CNN/Daily Mail. Counting those misses and the three translation tasks, T5 was state of the art on 18 of the 24 tasks. The paper names the move to 11 billion parameters as the most important ingredient, and a separate check (Table 15) found T5-Base beat the baseline trained on the same trillion tokens on every task, so the other recipe changes also counted.

T5 did not reach the state of the art on any WMT translation task. In the figure, English-to-German goes from 26.7 (Small) to 32.0 (Large) and then stops, 31.8 at 3B and 32.1 at 11B, against a previous best of 33.8. The paper offers hedged reasons: the English-only pre-training corpus "may be in part" responsible; most of the best systems used backtranslation, which creates extra training pairs by translating target-language text back into the source language; the best English-to-Romanian result used cross-lingual pre-training; and the best English-to-German result trained on the much larger WMT 2018 data. The paper does not isolate which of these matters, and does not blame the text-to-text format itself.

Much of what came after reused the paper's pieces directly. Switch Transformers built its mixture-of-experts models on the T5 architecture and trained them on C4, the text-to-text format became a common way to train one model on many instructions, and C4 became a standard pre-training corpus.