Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ArXiv: 1910.10683

🎯 Pitch

You don't actually need separate architectures or objectives for different NLP tasks—casting everything from translation to classification into a simple text-to-text format with task prefixes works state-of-the-art. A massive empirical study reveals that a denoising objective that randomly replaces token spans with a single sentinel outperforms all prior pre-training strategies, including BERT-style masking. When scaled to 11 billion parameters and trained on a 750 GB cleaned Common Crawl corpus, this approach hits near-human performance on SuperGLUE and sets new records on 18 of 24 benchmarks.

1. Executive Summary

This paper studies how to systematically unify and scale transfer learning for NLP by introducing the Text-to-Text Transfer Transformer (T5), a framework that casts every text-based language problem into a text-to-text format—feeding the model a task-specific prefix and training it to generate the target text (e.g., feeding “translate English to German: That is good.” to produce “Das ist gut.”). Through an extensive empirical survey of pre-training objectives, architectures, unlabeled data sets, and transfer approaches on a suite of benchmarks spanning summarization, question answering, and classification, the paper identifies a span-corruption denoising objective (replacing contiguous spans of corrupted tokens with sentinel markers) as the most effective pre-training strategy and demonstrates that the original encoder-decoder Transformer architecture outperforms decoder-only variants. By combining these insights with scale—training models up to 11 billion parameters on the new “Colossal Clean Crawled Corpus” (C4), a 750 GB heuristically filtered Common Crawl derivative—T5 achieves state-of-the-art results on 18 of 24 tasks, including a SuperGLUE score of 88.9 (nearly matching human performance of 89.8), a SQuAD exact match of 91.26, and a GLUE average of 90.3, establishing that a unified text-to-text approach combined with aggressive scaling can push transfer learning to near-human levels on many benchmarks, though English-only pre-training proves insufficient to surpass state-of-the-art translation systems.

2. Context and Motivation

The Core Problem: Fragmentation in Transfer Learning for NLP

By early 2019, transfer learning had emerged as the dominant paradigm in NLP. The basic recipe—pre-train a model on a data-rich task, then fine-tune it on downstream tasks of interest—had produced breakthrough results across the board. But the field was in a state of rapid, unstructured proliferation. Different research groups were introducing different pre-training objectives (masked language modeling for BERT, permutation language modeling for XLNet, autoregressive language modeling for GPT), different model architectures (encoder-only for BERT, decoder-only for GPT, encoder-decoder for MASS), different unlabeled pre-training corpora (BooksCorpus + Wikipedia, OpenWebText, Common Crawl derivatives), and different fine-tuning strategies (adapter layers, gradual unfreezing, multi-task learning).

The fundamental problem this paper addresses is not that any particular technique was failing—quite the opposite, each was succeeding on its own benchmarks. The problem was that the field lacked a systematic understanding of which design choices mattered and why. As the authors put it in Section 1:

"The rapid rate of progress and diversity of techniques in this burgeoning field can make it difficult to compare different algorithms, tease apart the effects of new contributions, and understand the space of existing methods for transfer learning."

This is a meta-scientific problem: when every paper introduces a new model, a new pre-training objective, a new dataset, and a new fine-tuning protocol simultaneously, it becomes impossible to attribute performance gains to any specific factor. Did BERT outperform GPT because of bidirectional context, or because of the masked language modeling objective, or because of the larger pre-training corpus, or because of the encoder-only architecture? The published literature could not answer these questions definitively because no two approaches were compared under controlled conditions.

Why This Matters: Beyond Academic Confusion

The fragmentation was not merely an academic inconvenience. It had concrete consequences for practitioners and the broader trajectory of NLP research:

Barriers to entry. The proliferation of bespoke architectures and training protocols meant that applying transfer learning to a new task required deep expertise and significant engineering effort. Each model family (BERT, GPT, XLNet, MASS) had its own codebase, its own input format, its own fine-tuning procedure. A practitioner wanting to compare these approaches empirically would need to reproduce multiple research codebases and resolve subtle differences in tokenization, sequence length handling, and optimization. This was a practical obstacle to adoption.

Inefficient resource allocation. Without a systematic understanding of which factors drive performance, the research community risked over-investing in individually minor innovations packaged as major breakthroughs. If a new model achieves +1% on GLUE by changing the pre-training objective, but the gain actually came from training on more data (a factor that was not controlled), then the apparent scientific advance is misleading. This phenomenon—where apparent progress stems from uncontrolled factors rather than genuine methodological improvements—distorts the research landscape and wastes computational resources.

The scaling hypothesis was untested in a unified setting. A growing body of evidence across deep learning (Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016) suggested that simply scaling up models and data could yield improvements that outstrip algorithmic innovations. Sutton's "bitter lesson" argued that general methods leveraging computation ultimately win out over hand-crafted approaches. But for NLP transfer learning, this hypothesis had not been tested systematically because every large-scale effort (GPT-2, XLNet, RoBERTa) used a different architecture, objective, and corpus. It was impossible to disentangle the benefits of scale from the benefits of specific design choices. A controlled scaling study—where the same model, objective, and data pipeline are used, varying only model size and training duration—was needed to assess whether scale alone could push performance toward human baselines.

The field was converging toward text-to-text, but not explicitly. The authors observe in Section 1 that prior work had already moved toward unifying NLP tasks:

"This approach is inspired by previous unifying frameworks for NLP tasks, including casting all text problems as question answering (McCann et al., 2018), language modeling (Radford et al., 2019), or span extraction Keskar et al. (2019b) tasks."

McCann et al.'s "Natural Language Decathlon" required all tasks to be formulated as question answering. Radford et al.'s GPT-2 showed that a single language model could perform translation, summarization, and question answering by conditioning on a textual prefix ("TL;DR:" for summarization). Keskar et al. recast many tasks as span extraction from an input text. Each of these frameworks demonstrated that unifying task formats was possible, but none combined the unification with a controlled empirical study of pre-training methodology. T5's text-to-text framework was a natural synthesis: take the unification insight from these prior works, apply it to a standard encoder-decoder architecture (already used for sequence-to-sequence tasks), and use the resulting consistency to perform the controlled ablations the field lacked.

Where Prior Approaches Fall Short

The paper identifies specific limitations in the existing landscape, organized along four axes:

1. Architectural Fragmentation

By 2019, three distinct Transformer-based architectural paradigms were in widespread use:

Encoder-only models (BERT, RoBERTa, ALBERT): These use a stack of Transformer layers with bidirectional self-attention. A special [CLS] token representation is fed to a classification head for sentence-level tasks, or each token representation is used for token-level tasks (like span prediction in SQuAD). This architecture is natural for classification and span extraction, but cannot perform generative tasks like translation or abstractive summarization without architectural modifications. BERT could not translate English to German because it has no autoregressive decoder.
Decoder-only models (GPT, GPT-2): These use causal (left-to-right) self-attention and are trained as language models. They can generate text autoregressively, making them applicable to both classification (by generating the label word) and generation tasks. However, causal masking means that each token's representation can only attend to previous tokens. For tasks where the model receives a context (premise, passage, source sentence) and must produce an answer, the representation of early tokens in the context is unnecessarily impoverished—it cannot incorporate information from later tokens. The paper explains this limitation in Section 3.2.1:

"With fully causal masking, the model's representation of a prefix state can only depend on prior entries of the prefix. So, when predicting an entry of the output, the model will attend to a representation of the prefix that is unnecessarily limited."

Encoder-decoder models (original Transformer, MASS): These use a bidirectional encoder to process the input and an autoregressive decoder to generate the output, with cross-attention from decoder to encoder. This design is natural for sequence-to-sequence tasks and handles both understanding (encoder) and generation (decoder). But at the time, no major transfer learning work had adopted this architecture as its primary framework. BERT proved that bidirectional context was valuable. MASS (Song et al., 2019) showed encoder-decoder pre-training was viable but was specifically designed for language generation tasks, not the full spectrum of NLP benchmarks.

The field lacked a systematic comparison of these architectures under otherwise identical conditions. This is what Section 3.2 provides, comparing five model variants at matched computational cost.

2. Proliferation of Incomparable Pre-Training Objectives

The pre-training objective is the mechanism through which the model acquires general-purpose knowledge from unlabeled text. By 2019, the landscape included:

Causal language modeling (GPT, GPT-2): Predict the next token given all previous tokens. Effective for generation tasks, but the unidirectional context limits representation quality for understanding tasks.
Masked language modeling (BERT): Corrupt 15% of input tokens (replace 80% with [MASK], 10% with a random token, 10% unchanged) and train the model to predict the original tokens at the masked positions. This objective requires the model to use bidirectional context and was shown to outperform language modeling pre-training on GLUE benchmarks. However, it was designed for encoder-only architectures—predicting tokens at the output positions of the encoder rather than generating text.
Permutation language modeling (XLNet): Train on all possible permutations of the factorization order, which provides bidirectional context while maintaining an autoregressive formulation. This was motivated as an improvement over BERT's independence assumption (BERT assumes each masked token is independent given the others, which is false), but added significant complexity.
Denoising autoencoding (MASS, BART): The encoder receives a corrupted version of the input and the decoder is trained to reconstruct the original sequence. MASS masked the last portion of the encoder input and had the decoder predict it. This objective was specifically designed for sequence-to-sequence pre-training.

Each of these objectives was evaluated under different conditions—different architectures, different corpora, different fine-tuning protocols. The question of which pre-training objective is genuinely most effective, controlling for all other factors, was unanswered.

Furthermore, the specific design of BERT's masking procedure (random token replacement, masking individual tokens) had never been ablated in a controlled way. Was the random token replacement important? Was masking individual tokens better than masking spans? Was predicting only the masked tokens better than reconstructing the entire sequence? These micro-design choices had become de facto standards through BERT's influence, but their individual contributions were unknown.

3. Pre-Training Data Sets: An Underexplored Factor

The NLP community had accumulated substantial evidence that pre-training data influences downstream performance, but data set choices were rarely studied as an independent variable:

BERT used BooksCorpus (800M words) + English Wikipedia (2,500M words)—roughly 16 GB of text.
GPT-2 used WebText, a 40 GB corpus of Reddit-linked web pages filtered for quality by upvote score.
XLNet used BooksCorpus + Wikipedia + ClueWeb 2012-B + Common Crawl, totaling roughly 160 GB.
RoBERTa used BooksCorpus + Wikipedia + CC-News + OpenWebText + Stories, totaling roughly 160 GB.

The community was converging on larger, web-sourced corpora, but the specific filtering approaches varied dramatically. Should you filter by Reddit score? By news domain? By Wikipedia-like clean text? The paper's introduction of C4 (Section 2.2) was motivated partly by the observation that existing large corpora were either not publicly available, used inconsistent filtering, or were domain-limited:

"We opted to create a new data set because prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope (e.g. are limited to News data... comprise only Creative Commons content... or are focused on parallel training data for machine translation)."

Moreover, the question of data set size vs. repetition had not been systematically studied. If a corpus is small enough that pre-training requires repeating examples many times, does that hurt performance? BERT was trained on roughly 137B tokens, but its corpus was only about 3.3B words, meaning each example was seen approximately 40 times. Was that repetition harmful, or was it mitigated by the stochasticity of the masking objective? The paper's Section 3.4.2 addresses this directly.

4. Inconsistent Fine-Tuning and Multi-Task Protocols

Even when pre-training was held constant, fine-tuning methodology varied across papers:

Some works (BERT) fine-tuned all model parameters.
Others (ULMFiT's gradual unfreezing) argued that updating all parameters on low-resource tasks caused catastrophic forgetting and proposed incrementally unfreezing layers.
Adapter layers (Houlsby et al., 2019) proposed adding small trainable modules between Transformer layers while keeping pre-trained weights frozen, trading parameter efficiency for some performance.
Multi-task learning (MT-DNN) proposed fine-tuning on multiple tasks simultaneously rather than individually, with the hypothesis that joint training acts as a regularizer.

No controlled comparison existed of these strategies. Did gradual unfreezing really help, or was the effect specific to the LSTM architecture used in ULMFiT (which differs from Transformers in how information flows across layers)? Did adapter layers work equally well for high-resource and low-resource tasks? Could multi-task pre-training match or exceed the standard pre-train-then-fine-tune pipeline? Section 3.5 systematically addresses these questions.

How This Paper Positions Itself

The paper's positioning is distinctive: it does not claim to introduce novel pre-training objectives, architectures, or fine-tuning methods. Instead, it positions itself as a large-scale empirical survey and controlled ablation study wrapped in a unifying framework that makes such comparisons possible for the first time.

The text-to-text framework (Section 2.4) is the enabler. By recasting every task—classification, regression, generation—as producing text output given text input, the paper creates a consistent testbed where the model architecture, loss function, training procedure, and decoding process are identical across all tasks. This means that when the paper compares, say, a BERT-style denoising objective to a language modeling objective, the only difference is the pre-training task—the model architecture, the corpus, the optimizer, the sequence length, and the fine-tuning protocol are all held constant. Prior work could not make this claim.

The approach is explicitly self-aware about its goals. From Section 1:

"We emphasize that our goal is not to propose new methods but instead to provide a comprehensive perspective on where the field stands."

And from the introduction to Section 3:

"This 'coordinate ascent' approach might miss second-order effects (for example, some particular unsupervised objective may work best on a model larger than our baseline setting), but performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive."

This is an honest acknowledgment of the study's limitations: by changing one factor at a time from a fixed baseline, the paper might miss interactions (e.g., the optimal pre-training objective might depend on model size). But the alternative—exhaustively exploring all combinations—is computationally infeasible. The coordinate ascent design is a pragmatic choice that enables broad coverage at the cost of potentially missing interaction effects.

The paper also positions itself relative to the "scaling hypothesis" in a nuanced way. Section 3.6 frames the scaling experiments with a specific premise:

"'You were just given 4× more compute. How should you use it?'"

This question operationalizes the bitter lesson: if you have additional computational resources, should you train a bigger model, train for longer, increase batch size, or ensemble? The paper systematically compares these options rather than assuming a single scaling strategy is optimal.

Finally, the paper's release strategy—open-sourcing the code, the C4 corpus, and pre-trained model weights for all model sizes (Small through 11B)—reflects a commitment to reproducibility and enabling the kind of controlled comparisons the field had been missing:

"Recognizing that the main utility of transfer learning is the possibility of leveraging pre-trained models in data-scarce settings, we release our code, data sets, and pre-trained models."

This positioned T5 not just as a research contribution but as infrastructure for the field—a unified baseline against which future innovations could be meaningfully compared.

3. Technical Approach

3.1 Reader Orientation

The Text-to-Text Transfer Transformer (T5) is a unified framework that treats every language problem—whether classification, regression, generation, or translation—as feeding text into a Transformer model and training it to produce text as output. It solves the fragmentation problem in transfer learning by providing a single, consistent experimental testbed where the model architecture, loss function, and training procedure are identical across all tasks, enabling controlled comparisons of pre-training objectives, architectures, data sets, and training strategies that were impossible when every prior approach used different setups.

3.2 Big-Picture Architecture (Diagram in Words)

The T5 system has four major components that form a complete transfer learning pipeline:

Text-to-Text Task Formatter: Converts any NLP task into a pair of text strings—an input sequence (prefixed with a task identifier like "translate English to German:" or "cola sentence:") and a target sequence (the answer, label, or generated text). This component ensures every task uses the same interface, allowing a single model to handle everything from sentiment analysis to summarization without task-specific heads or loss functions.
Transformer Model (Encoder-Decoder): A standard sequence-to-sequence Transformer that reads the formatted input through its bidirectional encoder and generates the target text autoregressively through its decoder. The encoder builds rich contextualized representations of the entire input (free from the causal masking limitations of decoder-only models), while the decoder uses these representations via cross-attention to produce outputs token by token. The same model processes all tasks—no architectural modifications between pre-training and fine-tuning.
Unsupervised Pre-Training Pipeline: Before seeing any labeled data, the model is trained on the Colossal Clean Crawled Corpus (C4) using a span-corruption denoising objective. The text-to-text format intrinsically supports this: unlabeled text spans are corrupted (contiguous token spans replaced by sentinel markers) to form the input, and the original spans form the target. This teaches the model general language understanding through reconstruction, with the same maximum-likelihood training used for all downstream tasks.
Fine-Tuning and Evaluation Harness: After pre-training, the model is fine-tuned separately on each downstream task (or on a multi-task mixture) by simply providing task-formatted examples. The model is trained with teacher forcing to predict the target text given the input, using a cross-entropy loss. At test time, greedy or beam decoding produces predictions, which are compared to ground-truth labels using task-appropriate metrics.

Information flows through these components sequentially: raw task data enters the formatter → becomes (text input, text target) pairs → the Transformer is pre-trained on C4 corruption pairs → then fine-tuned on supervised task pairs → predictions are decoded and evaluated against ground truth. This is a pre-train then fine-tune pipeline, though the paper also explores multi-task variants where pre-training and supervised data are mixed.

3.3 Roadmap for the Deep Dive

First, the text-to-text conversion mechanism, because it is the foundational abstraction that makes every other component work. Without understanding how tasks are unified, the subsequent architectural and objective comparisons would be unmotivated.
Second, the Transformer model architecture and its variants (encoder-decoder, language model, prefix LM), since the architecture determines what kinds of representations the model can build and constrains which objectives are applicable. I'll explain the specific architectural choices (relative position embeddings, simplified layer norm, parameter sharing) and why the encoder-decoder design was chosen over decoder-only alternatives.
Third, the pre-training objectives, starting with the baseline span-corruption denoising objective and systematically explaining each variant tested (BERT-style, MASS-style, i.i.d. corruption, span corruption, language modeling). This is the most extensively ablated component—I'll walk through the design space explored in Sections 3.3.1 through 3.3.4, including corruption rates, span lengths, and the decision to predict only corrupted tokens versus the entire sequence.
Fourth, the C4 data set creation pipeline, including the heuristic filtering steps, deduplication strategy, and the comparison data sets (RealNews-like, WebText-like, Wikipedia + TBC). The data set is both a contribution in itself and a controlled variable in the ablation studies of Section 3.4.
Fifth, the training and fine-tuning strategies (Section 3.5), covering adapter layers, gradual unfreezing, multi-task mixing strategies (examples-proportional, temperature-scaled, equal mixing), and the hybrid approach of multi-task pre-training followed by individual fine-tuning. This section explains how the paper operationalizes "transfer" beyond the standard fine-tuning paradigm.
Sixth, the scaling methodology (Section 3.6), which asks how to allocate additional compute among model size, training steps, batch size, and ensembling. I'll explain the FLOPs-matched comparison framework and the specific model size variants (Small, Base, Large, 3B, 11B).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis and survey paper whose core idea is that casting all NLP tasks into a text-to-text format enables the first controlled, large-scale ablation study of transfer learning design choices. The paper does not propose fundamentally new pre-training objectives or architectures; instead, it systematically compares existing techniques under identical conditions and then pushes the most promising combinations to unprecedented scale.

The Text-to-Text Conversion Mechanism

The text-to-text framework is simultaneously the paper's simplest and most consequential design choice. Every task is formulated as providing a text prefix (specifying the task) concatenated with the task input, and training the model to generate the target text. This means there is no task-specific output layer, no task-specific loss function, and no task-specific decoding procedure. A single model processes everything.

Input construction. For a given task, the raw input is converted to a string by prepending a task-specific prefix. The authors state that "the choice of text prefix used for a given task is essentially a hyperparameter; we found that changing the exact wording of the prefix had limited impact" (Section 2.4). The prefixes used in the paper include:

"translate English to German: That is good." for English-to-German translation
"cola sentence: The course is jumping well." for the CoLA acceptability judgment task
"mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity." for MNLI natural language inference
"summarize: state authorities dispatched emergency crews..." for CNN/Daily Mail summarization
"stsb sentence1: The rhino grazed on the grass. sentence2: A rhino is grazing in a field." for STS-B semantic textual similarity
"question: What does increased oxygen concentrations displace? context: Hyperbaric..." for SQuAD question answering

The prefix serves as a task descriptor that tells the model which behavior to execute. During pre-training, the model learns to condition its generation on these prefixes. During fine-tuning, the prefixes for the target task are used consistently, and during multi-task training, the model sees many different prefixes mixed together, learning to use them as conditional signals.

Target construction. The target is always a text string. For classification tasks, this is simply the label word or phrase (e.g., "entailment", "not_duplicate", "acceptable"). For generation tasks, it is the full target sequence (e.g., the German translation, the summary). For regression tasks like STS-B (which predicts similarity scores between 1 and 5), the floating-point score is rounded to the nearest 0.2 increment and converted to a string (e.g., 2.57 becomes "2.6"), effectively recasting regression as a 21-class classification problem. The authors note:

"We found that most of these scores were annotated in increments of 0.2, so we simply rounded any score to the nearest increment of 0.2 and converted the result to a literal string representation of the number" (Section 2.4).

An important edge case: what if the model outputs text that doesn't correspond to any valid label for a classification task? For example, if for an MNLI example the model outputs "hamburger" instead of "entailment", "neutral", or "contradiction". The paper handles this by always counting such outputs as incorrect (Section 2.4). Notably, the authors report that "we never observed this behavior in any of our trained models"—the task prefix and fine-tuning are sufficient to constrain the output space.

Special handling for Winograd tasks. The Winograd Schema Challenge tasks (WNLI, WSC, DPR) required additional preprocessing because they involve ambiguous pronoun resolution. The paper converts these to a "referent noun prediction" format: the ambiguous pronoun in the passage is highlighted with asterisks (e.g., "The city councilmen refused the demonstrators a permit because *they* feared violence.") and the model is trained to predict the referent noun ("The city councilmen").

For WSC specifically, examples come with a passage, an ambiguous pronoun, a candidate noun, and a True/False label. The paper only trains on True-labeled examples because the correct referent is unknown for False examples. At evaluation, the model's predicted noun text is compared to the candidate noun: if either is a subset of the other (ignoring case and articles), the prediction is True; otherwise False (Section 2.4). The WNLI training and validation sets overlap significantly with WSC, so the paper never trains on WNLI data to avoid leakage and never reports WNLI validation results (following standard practice since Devlin et al., 2018). For the final test set evaluation, a special preprocessing pipeline (described in Appendix B) converts WNLI hypotheses to the referent noun prediction format by matching pronoun-containing spans between premise and hypothesis.

Teacher forcing and maximum likelihood. Across all tasks and all training phases (pre-training and fine-tuning), the model is trained with teacher forcing (Williams and Zipser, 1989): at each decoder timestep, the model receives the ground-truth previous token (not its own previous prediction) and is trained to maximize the probability of the next target token. The loss function is always standard cross-entropy between the model's predicted token distribution and the ground-truth token:

$\mathcal{L} = -\sum_{t=1}^{T} \log p_\theta(y_t | y_{1:t-1}, x)$

where $T$ is the number of target tokens, $y_t$ is the $t$ -th target token, $x$ is the input sequence, and $p_\theta(y_t | y_{1:t-1}, x)$ is the probability the model assigns to token $y_t$ given the input and all previous target tokens.

What it computes: for each target sequence, the model processes the input through the encoder to produce a contextualized representation, then generates the target autoregressively through the decoder. At each generation step, the decoder's self-attention sees all previously generated tokens (via causal masking), and the encoder-decoder cross-attention attends to the full encoder output. The loss sums the negative log-probability of each target token, penalizing the model for assigning low probability to the correct next token. The total loss across a batch is the average of these per-sequence sums.

Why this form: teacher forcing with cross-entropy is the standard maximum-likelihood objective for autoregressive sequence models. It has the property that the model's predicted distribution over the full sequence factorizes as a product of conditional per-token distributions, which makes training efficient (no need to sample from the model during training) and optimization well-behaved (the loss surface is convex in the model's output logits). Alternative approaches like reinforcement learning (used in some summarization work to directly optimize ROUGE) or minimum risk training would introduce high-variance gradient estimates and training instability. The cross-entropy objective is consistent across all task types because the text-to-text format makes every task a sequence generation problem—there is no distinction between "classification loss" and "generation loss."

Decoding at inference. At test time, the model generates predictions autoregressively. For most experiments, the paper uses greedy decoding: at each timestep, select the token with the highest predicted probability. For the final results (Section 3.7), beam search with a beam width of 4 and a length penalty of $\alpha = 0.6$ (Wu et al., 2016) is used for translation and summarization tasks. The length penalty adjusts the beam search scores to favor longer or shorter outputs: a score $S(Y)$ for a candidate sequence $Y$ is computed as:

$S(Y) = \frac{\log p(Y|x)}{|Y|^\alpha}$

where $|Y|$ is the sequence length and $\alpha$ is the length penalty. When $\alpha < 1$ , shorter sequences are penalized less heavily; when $\alpha > 1$ , shorter sequences are penalized more heavily. The value $\alpha = 0.6$ encourages the model to generate longer outputs, which is appropriate for summarization and translation where output length can vary substantially.

The Transformer Model Architecture and Variants

The paper uses the Transformer as its backbone, but explores multiple structural variants. The baseline is a standard encoder-decoder Transformer as proposed by Vaswani et al. (2017), with specific modifications to layer normalization and position embeddings.

Baseline encoder-decoder model (Section 3.1.1). The baseline model is sized so that both the encoder and decoder match a BERT_BASE stack in depth and width:

Encoder: 12 blocks. Each block has self-attention (12 heads, $d_{kv} = 64$ ) followed by a feed-forward network (two dense layers with $d_{ff} = 3072$ and a ReLU activation between them).
Decoder: 12 blocks. Each block has causal self-attention (12 heads, $d_{kv} = 64$ ), cross-attention to the encoder output (12 heads, $d_{kv} = 64$ ), and the same feed-forward network as the encoder.
Embeddings: $d_{model} = 768$ . The input embedding matrix is shared with the output softmax layer weight matrix.
Total parameters: approximately 220 million, which is about twice BERT_BASE's 110M parameters because the baseline has two 12-layer stacks rather than one.
Regularization: dropout probability of 0.1 applied in the feed-forward networks, on the residual skip connections, on the attention weights, and at the input and output of the entire stack.

A key detail: the encoder uses fully-visible (bidirectional) self-attention. Every token in the encoder can attend to every other token. The decoder uses causal (unidirectional) self-attention, where token $i$ can only attend to tokens $1$ through $i$ . This is enforced through an attention mask that sets pre-softmax attention logits to $-\infty$ (effectively zeroing the post-softmax weight) for prohibited positions. Figure 3 in the paper visualizes these masking patterns as binary matrices.

Simplified layer normalization. The paper uses a variant of layer normalization (Ba et al., 2016) where activations are rescaled but no additive bias is applied. Additionally, layer normalization is placed outside the residual path—specifically, it is applied to the input of each subcomponent (self-attention, cross-attention, feed-forward) rather than to the output after the residual addition. The original Transformer applied layer norm after the residual addition (post-norm). The paper's approach (pre-norm) has been shown in subsequent work to improve training stability for deep Transformers, but the authors note that "these architectural changes are orthogonal to the experimental factors we consider in our empirical survey" and leave ablation to future work.

Relative position embeddings. Instead of the original Transformer's sinusoidal or learned absolute position embeddings, the paper uses a simplified form of relative position embeddings (Shaw et al., 2018; Huang et al., 2018a). In standard self-attention, the attention weight between query $q_i$ at position $i$ and key $k_j$ at position $j$ is computed as:

$w_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_{kv}}}$

With relative position embeddings, a learned scalar $b_{i-j}$ is added to this logit:

$w_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_{kv}}} + b_{i-j}$

What it computes: the attention weight now depends not just on the content of the query and key vectors, but also on their relative offset $i-j$ . The scalar $b_{i-j}$ is a learned parameter that captures patterns like "the token immediately to the left is typically more relevant than the token five positions away." The model learns a different $b$ value for each possible relative offset, up to a maximum of 128 positions. Offsets beyond 128 are all assigned to the same embedding.

Why this form: relative position embeddings provide the model with explicit knowledge of token proximity without requiring absolute position encoding. This is particularly useful in the text-to-text setting where input and output sequences can vary dramatically in length (a 512-token news article followed by a 50-token summary). Absolute position embeddings would require the model to learn that position 456 in the encoder corresponds to position 12 in the decoder, which is a difficult generalization. Relative position embeddings naturally capture the idea that "two tokens that are close together in the source text should interact similarly regardless of where they appear in the sequence." The simplification to a scalar (rather than a full vector as in Shaw et al., 2018) reduces the parameter count while retaining the key inductive bias.

The paper uses 32 learned embeddings for relative position offsets. The offsets are binned so that embeddings are shared across ranges, with the bin sizes increasing logarithmically. This means small offsets (1, 2, 3) each get their own embedding, while larger offsets (100–128) share an embedding. The position embedding parameters are shared across all layers (every layer uses the same set of 32 scalars), but within each layer, each attention head uses a different learned embedding. This means if there are 12 heads, each head has its own 32 learned position scalars, totaling $12 \times 32 = 384$ position parameters per layer.

Vocabulary. Text is tokenized using SentencePiece (Kudo and Richardson, 2018) into WordPiece subword units (Sennrich et al., 2015; Kudo, 2018) with a vocabulary of 32,000 tokens. The vocabulary is shared between input and output. To handle the non-English translation tasks (German, French, Romanian), the SentencePiece model was trained on a mixture of 10 parts English C4 data to 1 part each of German, French, and Romanian Common Crawl data classified by language detector. Special sentinel tokens (used for the pre-training objective, described below) are added to the vocabulary and do not correspond to any wordpiece.

Model parallelism and training infrastructure. The largest models (3B and 11B parameters) cannot fit on a single accelerator. The paper uses the Mesh TensorFlow library (Shazeer et al., 2018) to implement a combination of data parallelism (different batches processed on different devices) and model parallelism (different parts of the model placed on different devices). Training runs on Cloud TPU v3 Pods, which contain 1,024 TPU v3 chips connected via a high-speed 2D mesh interconnect. The authors note that TPUs are most efficient for large dense matrix multiplications, which motivated scaling $d_{ff}$ (the feed-forward dimension) specifically in the largest models rather than scaling all dimensions equally.

Architectural variants explored (Section 3.2). The paper compares five architectural variants under matched computational cost. To understand the comparison, define:

$L = 12$ : the number of layers in a BERT_BASE-sized stack
$P$ : the number of parameters in one BERT_BASE stack (approximately 110M)
$M$ : the FLOPs required for an $L+L$ encoder-decoder model to process a given input-target pair

The five variants are:

Encoder-decoder ( $2P$ params, $M$ FLOPs): The baseline. $L$ encoder layers, $L$ decoder layers. Used for all experiments in Sections 3.3–3.7.
Encoder-decoder with shared parameters ( $P$ params, $M$ FLOPs): The encoder and decoder share the same weight matrices. This halves the parameter count but keeps the same computational cost because the encoder and decoder must still be run separately (the encoder over the input, the decoder over the output). Results in Table 2 show this achieves nearly identical performance to the unshared baseline (e.g., GLUE 82.81 vs. 83.28, SQuAD 80.63 vs. 80.88).
Encoder-decoder with 6 layers ( $P$ params, $M/2$ FLOPs): Halving the number of layers in both encoder and decoder reduces both parameter count and computation. Performance degrades significantly—GLUE drops from 83.28 to 80.88, SQuAD from 80.88 to 77.59—showing that depth matters more than parameter count for this architecture.
Decoder-only language model ( $P$ params, $M$ FLOPs): A single Transformer stack with $L$ layers and causal masking throughout. To use this for text-to-text tasks, the input and target are concatenated into a single sequence (e.g., "translate English to German: That is good. target: Das ist gut.") and the model is trained as a standard autoregressive language model. At inference, the model is fed the prefix and generates the remainder. This architecture sees a fully causal mask—the input portion cannot attend to later input tokens—which limits representation quality. Performance is substantially worse across all tasks (e.g., GLUE 73.78 vs. 83.28, SQuAD 53.81 vs. 80.88).
Decoder-only prefix LM ( $P$ params, $M$ FLOPs): Same as the language model, but the input prefix uses fully-visible (bidirectional) masking rather than causal masking. The target portion still uses causal masking. This addresses the main limitation of the language model architecture by allowing the input representation to incorporate bidirectional context. Performance improves substantially over the language model (GLUE 81.82 vs. 73.78, SQuAD 78.94 vs. 53.81) but still slightly underperforms the encoder-decoder baseline (GLUE 81.82 vs. 83.28).

Why encoder-decoder wins. The paper attributes the encoder-decoder's advantage to two factors visible in the results. First, the shared-parameter encoder-decoder (variant 2) outperforms the prefix LM (variant 5) despite both having $P$ parameters and $M$ FLOPs, suggesting that the explicit encoder-decoder cross-attention mechanism is beneficial beyond what parameter sharing provides. In the prefix LM, the decoder attends to the input tokens directly through self-attention (which must serve double duty as both self-attention over the output and cross-attention to the input). In the encoder-decoder, there is a dedicated cross-attention sub-layer that specifically learns input-output alignment patterns.

Second, the decoder-only variants (language model and prefix LM) apply their $L$ layers to both the input and output, whereas the encoder-decoder applies $L$ layers to the input (encoder) and $L$ separate layers to the output (decoder), giving the model specialized processing for encoding and decoding at the same total FLOP cost. The paper notes:

"the same L + L encoder-decoder model will have approximately the same computational cost as a language model with only L layers. This is a consequence of the fact that the L layers in the language model must be applied to both the input and output sequence, while the encoder is only applied to the input sequence and the decoder is only applied to the output sequence." (Section 3.2.2)

Unsupervised Pre-Training Objectives

The pre-training objective is how the model learns general language knowledge from unlabeled text. The text-to-text framework allows many different objectives to be expressed in the same format: each objective defines a way to corrupt an input text span and requires the model to reconstruct the original or the dropped information. The paper systematically explores this design space.

Baseline objective: i.i.d. span corruption with sentinel tokens (Section 3.1.4). The baseline pre-training objective works as follows:

Start with a span of unlabeled text (e.g., "Thank you for inviting me to your party last week.").
Randomly select 15% of the tokens for corruption. Each token is independently selected with probability 0.15 (i.i.d. selection).
Identify consecutive spans of corrupted tokens. If multiple corrupted tokens appear consecutively, they form a single span. For example, if "for" and "inviting" are both selected, they form one span.
Replace each corrupted span with a unique sentinel token. Sentinel tokens have IDs that are unique to this example and are added to the vocabulary. The first span gets sentinel <X>, the second gets <Y>, and so on.
The target is the concatenation of all the dropped-out spans, each delimited by the same sentinel token that replaced it, with a final sentinel <Z> at the end.

For the sentence above with "for inviting" and "last" as corrupted spans, the input becomes "Thank you <X> me to your party <Y> week." and the target becomes "<X> for inviting <Y> last <Z>". This is illustrated in Figure 2 of the paper.

Why predict only corrupted spans rather than the full sequence. A key design choice is that the target contains only the dropped spans, not the full uncorrupted text. This makes pre-training more computationally efficient because the target sequence is much shorter—only 15% of the original tokens on average. In the standard BERT objective (where the model outputs a representation for every input token and must predict the original token at each masked position), the decoder would need to process the full sequence length during pre-training. By predicting only the dropped spans, the decoder's sequence length is reduced by approximately 85%, which substantially speeds up training. The tradeoff is that the model has less practice generating long coherent text during pre-training, but the paper found this did not hurt downstream performance (Table 5 shows similar results across variants).

Comparison of high-level objective types (Section 3.3.1). Before settling on the denoising approach, the paper compares three fundamentally different pre-training strategies:

Prefix language modeling: Split a text span into two parts at a random position. The first part becomes the input and the second part becomes the target. The model is trained as a standard autoregressive language model on the target given the prefix. For example, input = "Thank you for inviting", target = "me to your party last week.". This objective is essentially "learn to continue a given text."
BERT-style masking: Corrupt 15% of tokens: 90% of corrupted tokens are replaced with a special [MASK] token, 10% are replaced with a random token (to simulate the possibility that a token in the input is not actually masked but is just some other word). The target is the full original uncorrupted text. For example, input = "Thank you [MASK] [MASK] me to your party apple week." (where "apple" is the random replacement), target = "Thank you for inviting me to your party last week.". This objective was designed for encoder-only models, where the output at each masked position is used to predict the original token.
Deshuffling: Randomly permute the tokens in the input and train the model to reconstruct the original order. For example, input = "party me for your to . last fun you inviting week Thank", target = "Thank you for inviting me to your party last week .". This objective forces the model to learn word order and syntactic structure without any explicit linguistic supervision.

Results (Table 4) show that the BERT-style objective (GLUE 82.96, SQuAD 80.65) substantially outperforms both prefix language modeling (GLUE 80.69, SQuAD 77.99) and deshuffling (GLUE 73.17, SQuAD 67.61). The prefix language model does better on translation tasks (EnDe 26.86 vs. 26.78, EnFr 39.73 vs. 40.03), likely because translation requires autoregressive generation and the prefix LM objective provides practice at exactly this skill.

Simplifying the BERT objective (Section 3.3.2). Starting from the BERT-style objective as the best performer, the paper explores simplifications that reduce target sequence length:

MASS-style: Drop the random token replacement step. Simply replace 15% of tokens with a mask token and train the model to reconstruct the full original text. (Song et al., 2019 used this for sequence-to-sequence pre-training but masked only contiguous suffixes, not random spans.) Performance: GLUE 82.32, SQuAD 80.10—nearly identical to BERT-style (82.96, 80.65).
Replace corrupted spans (baseline): Instead of replacing each individual corrupted token with a mask token, replace each contiguous span of corrupted tokens with a single unique sentinel token. Train the model to predict only the dropped spans (not the full sequence). Performance: GLUE 83.28, SQuAD 80.88—slightly better than both previous variants, and the target sequences are much shorter.
Drop corrupted tokens: Simply remove the corrupted tokens from the input entirely (no sentinel tokens). Train the model to predict the dropped tokens in order. For example, input = "Thank you me to your party week.", target = "for inviting last". Performance: GLUE 84.44, SQuAD 80.52. The GLUE improvement is driven largely by a jump on CoLA (60.04 vs. 53.84), which the authors hypothesize is because "being able to determine when tokens are missing is closely related to detecting acceptability" (Section 3.3.2). However, this variant performed slightly worse on SuperGLUE and is harder to interpret (the model must figure out where the missing tokens belong without explicit position markers).

Varying the corruption rate (Section 3.3.3). Using the baseline replace-corrupted-spans objective, the paper tests corruption rates of 10%, 15%, 25%, and 50% (Table 6). The corruption rate controls what fraction of the input tokens are dropped out. At 10%, the target is very short (only 10% of tokens), and the model sees more of the original context, which might make the reconstruction task too easy. At 50%, half the input is missing, making reconstruction harder but also making the target twice as long as at 25%, which slows training. Results show that 15%, 25%, and to some extent 10% produce similar performance, while 50% shows degradation on GLUE and SQuAD. The paper sticks with 15%, following BERT's precedent and because it provides a good balance of task difficulty and target length.

Span corruption instead of i.i.d. corruption (Section 3.3.4). The i.i.d. corruption strategy selects tokens independently, so the number and length of corrupted spans is random and often results in many single-token spans. Since each span gets a unique sentinel token, having many single-token spans means many sentinel tokens in both the input and target, which increases sequence length without adding much learning signal. The paper introduces span corruption: instead of selecting tokens independently, the objective specifies the total number of tokens to corrupt and the number of spans, and then selects span lengths randomly to meet these constraints. Specifically, the objective is parameterized by:

Corruption rate $c$ : fraction of tokens to corrupt (kept at $c = 0.15$ )
Mean span length $\ell$ : average number of consecutive tokens per corrupted span

Given a sequence of $N$ tokens, the total number of corrupted tokens is $cN$ , and the number of spans is $cN / \ell$ . For example, with $c = 0.15$ , $\ell = 3$ , and $N = 500$ , the objective would select approximately 75 tokens to corrupt, grouped into approximately 25 spans, with the spans ranging from 1 to perhaps 8 tokens to achieve the correct mean.

The paper compares mean span lengths of 2, 3, 5, and 10 against the i.i.d. baseline (Table 7). Results show that $\ell = 3$ slightly but significantly outperforms the i.i.d. baseline on most tasks (GLUE 83.49 vs. 83.28, SQuAD 82.09 vs. 80.88, SGLUE 72.53 vs. 71.36). The $\ell = 10$ variant slightly underperforms on some tasks (GLUE 82.85, SGLUE 70.44). The span-corruption objective also produces shorter sequences on average than i.i.d. corruption because forcing tokens into contiguous spans reduces the number of sentinel tokens needed.

The paper's final pre-training objective (used for T5 in Section 3.7) is span corruption with mean span length 3 and 15% corruption rate. This choice is motivated by the small performance advantage and the computational efficiency gain from shorter target sequences.

Discussion of objective exploration (Section 3.3.5). Figure 5 summarizes the exploration path as a flow chart. The headline finding is that all denoising objectives perform similarly—the choice among them matters far less than the choice to use a denoising objective rather than language modeling or deshuffling. The authors conclude:

"We did not observe a remarkable difference across the many variants of the denoising objectives we explored. However, different objectives (or parameterizations of objectives) can lead to different sequence lengths and thus different training speeds. This implies that choosing among the denoising objectives we considered here should mainly be done according to their computational cost." (Section 3.3.5)

This is a significant negative result: the explosion of pre-training objective variants in the literature (BERT's random token replacement, XLNet's permutation LM, SpanBERT's span masking) may be contributing less than assumed, at least within the encoder-decoder text-to-text framework. The real gains come from scale and data quality.

The C4 Data Set and Pre-Training Data Comparisons

The Colossal Clean Crawled Corpus (C4) is introduced as a large, diverse, clean English text corpus derived from Common Crawl. The paper describes both the creation pipeline and a set of controlled comparisons against alternative pre-training data sets.

C4 creation pipeline (Section 2.2). The pipeline starts with the "web extracted text" from the April 2019 Common Crawl dump, which has already had HTML markup and non-text content removed. This raw text is then filtered using the following heuristics:

Sentence-level filtering: Only retain lines ending in a terminal punctuation mark (period, exclamation mark, question mark, or end quotation mark). This removes fragments, lists, code, and other non-sentence text.
Page-level filtering: Discard pages with fewer than 3 sentences and only retain lines with at least 5 words. This removes very short pages (error messages, navigation elements) and very short lines (headers, menu items).
Bad word filtering: Remove any page containing a word from the "List of Dirty, Naughty, Obscene or Otherwise Bad Words." This is a blunt content filter.
Javascript filtering: Remove any line containing the word "Javascript." Many pages contain "Javascript must be enabled" warnings.
Lorem ipsum filtering: Remove any page containing the phrase "lorem ipsum," which is placeholder text.
Code filtering: Remove any page containing a curly bracket {, which is common in many programming languages but rare in natural text.
Citation marker removal: Remove citation markers like [1] or [citation needed] that are common on Wikipedia-derived pages.
Policy notice filtering: Remove any lines containing "terms of use," "privacy policy," "cookie policy," "uses cookies," "use of cookies," or "use cookies."
Deduplication: Discard all but one instance of any three-sentence span that appears more than once in the data set. This is a locality-sensitive hashing approach: rather than deduplicating full documents, it identifies repeated boilerplate text that might appear across many pages.

After applying these filters, langdetect is used to filter for English text: any page not classified as English with probability at least 0.99 is discarded. The resulting data set is approximately 750 GB of reasonably clean, deduplicated English text.

Why these specific heuristics. The filters are inspired by prior work on using Common Crawl for NLP (Grave et al., 2018; Smith et al., 2013), but the paper creates a new data set because prior efforts used fewer filters, were not publicly available, or were limited to specific domains (news, Creative Commons, parallel translation data). The heuristics are designed to be simple, fast to apply, and broadly effective at removing the most common types of non-natural-language content from web text.

Alternative data sets for comparison (Section 3.4.1). To measure the effect of data quality, domain, and size on pre-training, the paper compares C4 against several variants and existing corpora:

Unfiltered C4 (6.1 TB): C4 without the heuristic filtering (but still English-filtered with langdetect). This is much larger (6.1 TB vs. 750 GB) but contains substantial noise. Performance is uniformly worse than filtered C4 across all tasks (e.g., GLUE 81.46 vs. 83.28, SQuAD 78.78 vs. 80.88), showing that the filtering heuristics provide a genuine quality improvement despite reducing the data set size by 8×.
RealNews-like (35 GB): C4 further filtered to only include content from domains used in the RealNews data set (Zellers et al., 2019), which consists of news articles. This is a domain-limited subset but contains high-quality journalistic text. Performance is mixed: improves SGLUE (72.38 vs. 71.36, driven by a large ReCoRD gain from 68.16 to 73.72, since ReCoRD is a reading comprehension task based on news articles) but similar on other tasks.
WebText-like (17 GB): C4 filtered to only include pages submitted to Reddit with a score of at least 3, following the WebText approach (Radford et al., 2019). To get enough data, 12 months of Common Crawl were used instead of one. This produces a 17 GB data set (comparable to the original 40 GB WebText). Performance is slightly better than C4 on some tasks (GLUE 84.03 vs. 83.28, SQuAD 81.42 vs. 80.88), suggesting Reddit voting provides a useful quality signal.
Wikipedia (16 GB): English Wikipedia from TensorFlow Datasets, with markup and references removed. Performance is worse than C4 on GLUE (81.85 vs. 83.28) but better on SQuAD (81.29 vs. 80.88), which is expected because SQuAD passages are sourced from Wikipedia.
Wikipedia + Toronto Books Corpus (20 GB): The combination used by BERT. This domain-diverse corpus (encyclopedia + fiction) performs notably better on SuperGLUE (73.24 vs. 71.36), driven almost entirely by a massive improvement on MultiRC (Exact Match 50.93 vs. 25.78). MultiRC's largest data source is fiction books, so the TBC provides in-domain pre-training.

The in-domain effect. A clear pattern emerges: pre-training on data from the same domain as the downstream task improves performance. The paper documents three cases:

"using the Wikipedia + TBC corpus produced a SuperGLUE score of 73.24, beating our baseline's score (using C4) of 71.36. This is almost entirely attributable to a boost in performance from 25.78 (baseline, C4) to 50.93 (Wikipedia + TBC) on the Exact Match score for MultiRC... Similarly, using the RealNews-like data set for pre-training conferred an increase from 68.16 to 73.72 on the Exact Match score for ReCoRD... As a final example, using data from Wikipedia produced significant (but less dramatic) gains on SQuAD" (Section 3.4.1).

This has both a positive and a negative implication. Positive: if you know your target domain, you can improve performance by including in-domain unlabeled data. Negative: a general-purpose pre-trained model cannot be simultaneously optimal for all domains. The paper opts for the diverse C4 as a compromise that performs reasonably well across all tasks.

Data set size and repetition (Section 3.4.2). The baseline pre-training uses $2^{35} \approx 34$ billion tokens, which is only a fraction of C4 (750 GB ≈ roughly 150B tokens, though the exact token count depends on the text density after filtering). The paper tests what happens when the pre-training data set is artificially truncated: using only the first $2^{29}$ , $2^{27}$ , $2^{25}$ , and $2^{23}$ tokens from C4, which corresponds to repeating the data set 64, 256, 1024, and 4096 times respectively over the course of pre-training.

Results (Table 9) show that performance degrades as the data set size shrinks and repetition increases. With 64× repetition (229 tokens), performance is nearly identical to the full data set (GLUE 82.87 vs. 83.28). With 256× repetition, degradation is noticeable but small (GLUE 82.62, SQuAD 79.78). With 1024× and 4096× repetition, performance drops sharply (GLUE 79.55 and 76.34, SQuAD 76.27 and 70.92). The pre-training loss curves (Figure 6) confirm that the model achieves much lower training loss with smaller data sets, suggesting memorization rather than generalization. The paper concludes:

"This suggests that some amount of repetition of pre-training data might not be harmful. However, given that additional pre-training can be beneficial (as we will show in Section 3.6) and that obtaining additional unlabeled data is cheap and easy, we suggest using large pre-training data sets whenever possible." (Section 3.4.2)

This finding is practically important because it means pre-training on 1 trillion tokens (as done for the final T5 models) requires a data set large enough to avoid excessive repetition. C4, at 750 GB, is large enough for this scale; the domain-limited alternatives (Wikipedia + TBC at 20 GB, WebText-like at 17 GB) would be repeated hundreds or thousands of times, which the experiments show would hurt performance.

Training and Fine-Tuning Strategies

Beyond the standard pre-train-then-fine-tune approach, the paper explores alternative strategies for adapting pre-trained models to downstream tasks.

Fine-tuning methods (Section 3.5.1). Three approaches are compared:

All parameters (baseline): Fine-tune every parameter of the model. The model is initialized from pre-trained weights and trained on the downstream task for $2^{18} = 262,144$ steps with a constant learning rate of 0.001. A checkpoint is saved every 5,000 steps and the best validation checkpoint is used for evaluation. This is the simplest approach but requires storing and updating all parameters.
Adapter layers: Insert small trainable modules between the existing Transformer layers while keeping the pre-trained weights frozen. Each adapter is a bottleneck feed-forward network: a dense layer projecting from $d_{model}$ down to a smaller inner dimensionality $d$ , followed by a ReLU activation, followed by a dense layer projecting back up to $d_{model}$ . The adapter output is added to the original layer output via a residual connection. During fine-tuning, only the adapter parameters and layer normalization parameters are updated.

The inner dimensionality $d$ is the key hyperparameter, controlling how many new parameters are added. The paper tests $d = 32, 128, 512, 2048$ . Results (Table 10) show that adapter performance depends strongly on task size: for SQuAD (a relatively large QA dataset), even $d = 32$ achieves reasonable performance (79.32 vs. 80.88 for full fine-tuning). But for translation tasks, performance collapses with small $d$ (e.g., EnDe BLEU drops from 26.98 to 13.84 with $d = 32$ ). With $d = 2048$ , adapter performance approaches full fine-tuning on most tasks, but at that point the adapters add nearly as many parameters as the original model.

"Pursuant with past results (Houlsby et al., 2019; Bapna et al., 2019) we find that lower-resource tasks like SQuAD work well with a small value of d whereas higher resource tasks require a large dimensionality to achieve reasonable performance." (Section 3.5.1)

Gradual unfreezing: Start fine-tuning with only the final layer's parameters trainable, then incrementally "unfreeze" earlier layers over time. Specifically, with 12 layers in each of encoder and decoder and 262,144 fine-tuning steps, the process is divided into 12 episodes of 21,845 steps each. In episode $n$ (where $n$ goes from 1 to 12), layers $12-n+1$ through 12 are trainable. The input/output embedding matrix (shared) is trainable throughout.

Results show gradual unfreezing causes minor degradation (GLUE 82.50 vs. 83.28, SQuAD 79.17 vs. 80.88) while providing some training speedup. The authors note that "better results may be attainable by more carefully tuning the unfreezing schedule" (Section 3.5.1).

Multi-task learning (Section 3.5.2). In the text-to-text framework, multi-task learning is remarkably simple: just mix examples from different tasks into the same training batches. There is no need for task-specific heads, loss balancing, or gradient manipulation. However, the mixing proportions—how many examples from each task appear in a batch—become a critical hyperparameter.

The paper defines three mixing strategies, all starting from the dataset sizes $e_n$ for each of $N$ tasks:

Examples-proportional mixing. For each task $m$ , compute a capped size $r_m = \min(e_m, K)$ where $K$ is an artificial limit. The sampling probability for task $m$ is then $r_m / \sum_n r_n$ . This means tasks smaller than $K$ are sampled in proportion to their actual size, while tasks larger than $K$ are capped at $K$ . The cap $K$ is necessary because the unsupervised task (C4) is orders of magnitude larger than supervised tasks and would otherwise dominate every batch.

The paper sweeps $K$ from $2^{16}$ to $2^{21}$ . At $K = 2^{16}$ (small cap), all tasks are capped to the same size, so the mixture is nearly uniform. At $K = 2^{21}$ (large cap), large tasks dominate.

Temperature-scaled mixing. Instead of hard capping, adjust the proportions by raising each to the power $1/T$ and renormalizing:

$p_m = \frac{r_m^{1/T}}{\sum_n r_n^{1/T}}$

where $r_m = \min(e_m, 2^{21})$ and $T$ is the temperature. When $T = 1$ , this is equivalent to examples-proportional mixing. When $T \to \infty$ , all $p_m$ become equal (uniform mixing). Intermediate $T$ values provide a smooth interpolation between these extremes. The paper tests $T = 2, 4, 8$ .

Equal mixing. Simply sample each task with probability $1/N$ . This means every batch contains approximately the same number of examples from each task, regardless of dataset size.

What it computes for a specific configuration: consider an examples-proportional mixture with $K = 2^{19} = 524,288$ . Each task's capped size $r_m$ is the minimum of its actual size and 524,288. The total weight $\sum_n r_n$ sums these capped sizes across all tasks (including unsupervised C4). The probability of sampling task $m$ is $r_m / \sum_n r_n$ . For a task with 10,000 examples (below the cap), $r_m = 10,000$ . For C4 with billions of tokens, $r_{unsup} = 524,288$ . The C4 unsupervised task therefore receives weight proportional to 524,288 rather than billions, preventing it from dominating.

Why this form: the capping mechanism addresses the fundamental challenge of multi-task learning with imbalanced datasets: if tasks are sampled proportionally to their true sizes, the largest datasets drown out the smallest ones, and the model never learns the small tasks. If tasks are sampled uniformly, the model overfits on small tasks (seeing the same few examples repeatedly) and underfits on large tasks (never seeing most of the data). The cap $K$ provides a tunable compromise.

Multi-task results (Table 11). No multi-task strategy matches the baseline pre-train-then-fine-tune approach. The best configuration (examples-proportional with $K = 2^{19}$ ) achieves GLUE 81.42 vs. 83.28 for the baseline. The gap is larger on translation tasks (EnDe 25.21 vs. 26.98, EnFr 36.30 vs. 39.82) and SuperGLUE (67.30 vs. 71.36). Equal mixing performs substantially worse (GLUE 76.13), likely because the model both overfits on small tasks and doesn't see enough unlabeled data to learn general language capabilities.

Combining multi-task pre-training with fine-tuning (Section 3.5.3). An important variant: pre-train on a multi-task mixture, then fine-tune on each individual task. This is the approach used by MT-DNN (Liu et al., 2019b) and is adopted for the final T5 models. Specifically:

Pre-train the model for $2^{19}$ steps on an examples-proportional mixture of unsupervised C4 data and all supervised tasks, using a cap $K = 2^{19}$ (for the baseline model size; the cap varies for larger models as described in Section 3.7).
After this multi-task pre-training, fine-tune separately on each individual task for $2^{18}$ steps, as in the standard approach.

Results (Table 12) show this approach achieves comparable performance to the standard pre-train-then-fine-tune baseline (GLUE 83.11 vs. 83.28, SQuAD 80.26 vs. 80.88). This is significant because it shows that the benefits of unsupervised pre-training are not diluted by mixing in supervised data during pre-training.

The paper also tests a leave-one-out variant: pre-train on the mixture but omit one downstream task, then fine-tune on that held-out task. Performance is only slightly worse (e.g., GLUE 81.98 vs. 83.11 for full multi-task pre-training), suggesting that "a model that was trained on a variety of tasks can still adapt to new tasks" and that "multi-task pre-training might not result in a dramatic task interference" (Section 3.5.3).

A purely supervised multi-task pre-training (no unsupervised data, only the supervised tasks mixed together) underperforms significantly (GLUE 79.93, SGLUE 65.36), except for translation tasks where it slightly helps. This reinforces the importance of unsupervised pre-training for general language understanding.

Scaling Methodology

Section 3.6 asks a specific question: "You were just given 4× more compute. How should you use it?" This operationalizes the scaling hypothesis in a controlled way.

Model size variants. Starting from the 220M-parameter baseline, the paper defines two larger variants:

2× size (2× params, 2× FLOPs): Following BERT_LARGE: $d_{ff} = 4096$ , $d_{model} = 1024$ , $d_{kv} = 64$ , 16 heads, 16 layers each in encoder and decoder. Approximately 440M parameters.
4× size (4× params, 4× FLOPs): Same width as 2× but with 32 layers each in encoder and decoder. Approximately 880M parameters.

Scaling strategies compared. Using these variants, the paper compares four ways to use 4× the baseline compute:

1× size, 4× training steps: Keep the 220M model but pre-train for $4 \times 2^{19} = 2^{21}$ steps and fine-tune for $4 \times 2^{18} = 2^{20}$ steps.
1× size, 4× batch size: Keep the 220M model and the same number of steps, but use a batch size of 512 sequences instead of 128. This processes 4× as many tokens in the same number of steps.
2× size, 2× training steps: Use the 440M model (2× compute per step) and train for 2× as many steps, for a total of 4× compute.
4× size, 1× training steps: Use the 880M model (4× compute per step) with the baseline number of steps, for a total of 4× compute.

Ensembling comparisons. Two ensembling strategies provide an orthogonal way to use additional compute:

4× ensembled: Train 4 completely separate models (separate pre-training and fine-tuning runs), then average their output logits before softmax.
4× ensembled, fine-tune only: Pre-train 1 model, then fine-tune 4 separate copies on the downstream task, and ensemble their predictions. This uses less than 4× total compute but provides a cheaper ensembling baseline.

Results (Table 13). The headline finding is that all scaling strategies improve over the baseline, but no single strategy dominates:

1× size, 4× training steps: GLUE 85.33, SQuAD 82.45, SGLUE 74.72
1× size, 4× batch size: GLUE 84.60, SQuAD 82.52, SGLUE 74.64
2× size, 2× training steps: GLUE 86.18, SQuAD 84.18, SGLUE 77.18
4× size, 1× training steps: GLUE 85.91, SQuAD 83.86, SGLUE 78.04

The 2× size + 2× training steps configuration slightly edges out the 4× size configuration on most metrics, suggesting that increasing both model size and training duration is more effective than increasing either alone. For translation tasks, ensembling dominates: 4× ensembled achieves the best EnDe (28.05 vs. 27.08–27.52) and EnRo (28.57 vs. 27.84–28.19) BLEU scores.

Why the paper uses all scaling strategies in the final T5. The final T5 models (Section 3.7) combine insights: they use larger model sizes (up to 11B parameters), much longer training (1 trillion tokens, roughly 32× the baseline), and multi-task pre-training. The scaling experiments in Section 3.6 justify this approach by showing that size, training duration, and ensembling are complementary rather than redundant.

Final T5 model configurations (Section 3.7). The paper trains five model sizes for the final results:

T5-Small: $d_{model} = 512$ , $d_{ff} = 2048$ , 8 heads, 6 layers each in encoder and decoder. 60M parameters.
T5-Base: Same as the baseline: $d_{model} = 768$ , $d_{ff} = 3072$ , 12 heads, 12 layers. 220M parameters.
T5-Large: BERT_LARGE-sized: $d_{model} = 1024$ , $d_{ff} = 4096$ , 16 heads, 24 layers. 770M parameters.
T5-3B: $d_{model} = 1024$ , $d_{ff} = 16384$ , 32 heads, 24 layers. 2.8B parameters. The $d_{ff}$ scaling specifically targets efficient TPU utilization for large matrix multiplications.
T5-11B: $d_{model} = 1024$ , $d_{ff} = 65536$ , 128 heads, 24 layers. 11B parameters. Same motivation for $d_{ff}$ scaling.

All models use $d_{kv} = 64$ (Small, Base, Large) or $d_{kv} = 128$ (3B, 11B). The 3B and 11B models use $d_{model} = 1024$ rather than scaling the embedding dimension proportionally, meaning the feed-forward networks dominate the parameter count—the 11B model's $d_{ff} = 65536$ means each feed-forward block has approximately $1024 \times 65536 \times 4$ parameters (two dense layers with ReLU, plus bias terms), dwarfing the attention and embedding parameters.

Pre-training for the final models. The final T5 models use:

1 million pre-training steps with batch size $2^{11} = 2048$ sequences of length 512, totaling approximately 1 trillion tokens.
The span-corruption objective with mean span length 3 and 15% corruption rate.
Multi-task pre-training with examples-proportional mixing, using artificial dataset size limits that scale with model size: 710,000 for Small, 2,620,000 for Base, 8,660,000 for Large, 33,500,000 for 3B, and 133,000,000 for 11B.
WMT English-to-French and English-to-German datasets capped at 1 million examples during pre-training.
The inverse square root learning rate schedule with 10,000 warm-up steps.
AdaFactor optimizer.
For the final GLUE and SuperGLUE results, fine-tuning is done on individual tasks rather than the concatenated benchmark, using a smaller batch size of 8 length-512 sequences and saving checkpoints every 1,000 steps to capture performance before overfitting on low-resource tasks.
Beam search (width 4, length penalty $\alpha = 0.6$ ) for translation and summarization; greedy decoding for other tasks.

The multi-task pre-training caps are chosen so that larger models (which are more prone to overfitting on small datasets) see a higher proportion of unsupervised data. The cap for T5-11B (133M) is approximately 50× the cap for T5-Small (710K), meaning the largest model sees dramatically more unlabeled data relative to supervised data during pre-training.

4. Key Insights and Innovations

Innovation 1: Unification as a Prerequisite for Controlled Science, Not Just Engineering Convenience

The text-to-text framework is often described as an engineering contribution—a way to train one model on many tasks. But the paper's deeper intellectual move is to treat unification as the precondition for rigorous empirical science in transfer learning. Before T5, the field's fragmentation was not merely inconvenient; it made it impossible to answer the most basic question: which design choices actually matter? When every paper introduces a new architecture, a new objective, and a new dataset simultaneously, a +1% GLUE gain could be attributed to any of these factors, or to their interaction, or to unreported differences in hyperparameter tuning.

The text-to-text framework solves this by creating a fixed coordinate system for experimentation. Because every task—classification, regression, generation—uses the same loss function (cross-entropy), the same decoding procedure (autoregressive sampling or beam search), and the same model interface (text in, text out), the paper can change exactly one variable at a time and measure its effect. This is what enables the "coordinate ascent" approach in Section 3: vary the architecture while holding the objective and data constant (Section 3.2), vary the objective while holding the architecture and data constant (Section 3.3), vary the data while holding architecture and objective constant (Section 3.4). Prior work could not do this because each model family (BERT, GPT, XLNet) was architecturally incompatible with others' objectives and tasks.

This is a methodological innovation masquerading as a modeling one. The paper's most cited contribution—the text-to-text format—is not primarily beneficial because it simplifies engineering (though it does). It is beneficial because it enables the systematic ablation study that occupies the majority of the paper. Without it, Sections 3.2–3.6 would be a series of non-comparable papers rather than a single controlled experiment.

The evidence that this framing matters, beyond the convenience argument, is in the negative results it enables. The finding that all denoising objectives perform similarly (Section 3.3.5: "We did not observe a remarkable difference across the many variants of the denoising objectives we explored") is only credible because the comparison was conducted under identical conditions. If each objective variant had been tested in its own paper with its own model and data, small differences could have been attributed to the objective when they were actually due to uncontrolled factors. The text-to-text framework converts the field's fragmentation from a source of confusion into a source of experimental control.

Compare this to the dominant approach at the time: each model was its own ecosystem. BERT used an encoder-only architecture with a masked LM objective and a specific fine-tuning protocol involving a [CLS] token classifier. GPT-2 used a decoder-only language model with a different tokenizer and zero-shot prompting. XLNet used a permutation LM objective requiring a specialized architecture. The DecaNLP benchmark (McCann et al., 2018) proposed a unified QA format but required multi-task training and task-specific answer modules. T5's insight was that unification at the model level is more scientifically valuable than unification at the task level, because it is the model-level unification that enables controlled comparisons, not the task-level unification.

Innovation 2: The Denoising Objective Landscape Is Surprisingly Flat—and That's a Finding, Not a Failure

The paper's exploration of unsupervised objectives (Section 3.3) could have been a disappointment: after testing BERT-style masking, MASS-style masking, i.i.d. span corruption, contiguous span corruption with various mean lengths, dropping tokens entirely, and corruption rates from 10% to 50%, the conclusion was that none of these choices matters very much. The paper states plainly that "choosing among the denoising objectives we considered here should mainly be done according to their computational cost" (Section 3.3.5).

This is a genuinely important negative result that has been underappreciated because negative results are harder to celebrate than positive ones. At the time, the NLP community was investing substantial effort in designing increasingly sophisticated pre-training objectives. BERT's masking involved careful random-token-replacement to handle the train-test mismatch from the [MASK] token. XLNet introduced permutation language modeling to address BERT's independence assumption. SpanBERT argued for masking contiguous spans rather than individual tokens. Each of these was motivated by a specific linguistic or probabilistic argument and was presented as an improvement.

T5's null result recasts this landscape. The finding that even the simplest denoising variant (drop corrupted tokens entirely, with no sentinel markers, and predict them in order) performs competitively suggests that the denoising objective's function is not the specific corruption pattern but rather the generic requirement to model the distribution of missing text given context. The model learns general language understanding not because of the clever masking strategy but because the task of filling in gaps forces it to build rich contextualized representations of the surrounding text. This is a conceptual simplification: the community's focus on objective design was largely optimizing a relatively flat surface.

The paper does find one meaningful distinction: denoising objectives (all variants) substantially outperform language modeling and deshuffling (Table 4: BERT-style GLUE 82.96 vs. prefix LM 80.69 vs. deshuffling 73.17). So the type of objective (denoising vs. autoregressive generation vs. ordering) matters, but the specific instantiation of denoising does not. This is a fundamental finding that redirects research attention from objective engineering to other factors—specifically scale and data quality, which the paper shows matter substantially more.

The practical implication, which the paper acts on in Section 3.7, is that objective choice should optimize for computational efficiency rather than linguistic cleverness. The span-corruption objective with mean span length 3 is chosen for the final T5 models because it produces the shortest target sequences (faster training) while performing marginally better than alternatives—but the margin is small enough that if a different variant were even faster, it would be the right choice.

Innovation 3: Pre-Training on In-Domain Data Provides a Substantial Signal—General-Purpose Models Face a Fundamental Tradeoff

Section 3.4.1 documents a pattern that is individually unsurprising but collectively profound: pre-training on text from the same domain as a downstream task improves performance on that task, sometimes dramatically. The paper quantifies this with specific examples: pre-training on Wikipedia + Toronto Books Corpus improves MultiRC exact match from 25.78 to 50.93 (a near-doubling) because MultiRC's passages come from fiction books. Pre-training on RealNews-like data improves ReCoRD exact match from 68.16 to 73.72 because ReCoRD uses news articles. Pre-training on Wikipedia improves SQuAD because SQuAD passages are from Wikipedia.

The conceptual innovation is not the observation itself—prior work had noted domain matching benefits—but the quantification of the tradeoff in the context of a general-purpose model. C4 is deliberately diverse, and the paper shows it performs reasonably well across all tasks. But it is never the best pre-training corpus for any single task except those without strong domain characteristics. The domain-limited corpora (Wikipedia + TBC at 20 GB, WebText-like at 17 GB) outperform C4 on specific benchmarks while being 40× smaller.

This creates a fundamental tension that the paper makes explicit but does not resolve: a general-purpose pre-trained model cannot be simultaneously optimal for all domains. The choice between C4 (broad but never best) and a domain-specific corpus (best in domain, worse elsewhere) depends on the intended use case. This is a diagnostic finding that reframes the problem: rather than searching for a universally optimal pre-training corpus, the field should develop methods for efficiently adapting general-purpose models to specific domains, or for combining diverse corpora in ways that preserve domain-specific performance without sacrificing breadth.

The scaling analysis in Section 3.4.2 adds another dimension: domain-specific corpora are typically small, and small corpora cause performance degradation when the model is pre-trained for long enough to repeat them many times (Table 9: repeating the dataset 1024× drops GLUE from 83.28 to 79.55). C4's size (750 GB) is necessary to support the 1-trillion-token pre-training used in Section 3.7 without harmful repetition. So the tradeoff is not just domain match vs. diversity—it is also domain match vs. scale. A 20 GB domain-specific corpus might outperform C4 at 34B tokens of pre-training, but it would likely underperform C4 at 1 trillion tokens because of overfitting from repetition. The paper's final choice of C4 is driven by this scale constraint as much as by diversity considerations.

Innovation 4: The Bitter Lesson Applied to Transfer Learning—Scale Is a First-Class Research Variable, Not Just an Engineering Detail

The "bitter lesson" (Sutton, 2019) argues that general methods leveraging computation ultimately outperform hand-crafted approaches. The paper operationalizes this in the specific context of NLP transfer learning through a systematic scaling study (Section 3.6) that asks a concrete, actionable question: given 4× more compute, should you train a bigger model, train for longer, increase the batch size, or ensemble? The answer—that all approaches help and are complementary—is less important than the framing of scale as an experimental variable to be optimized, not an afterthought.

Before T5, the scaling of NLP transfer learning models was largely ad hoc. BERT_LARGE was defined as "wider and deeper" than BERT_BASE. GPT-2 scaled up by increasing layers and parameters. RoBERTa trained for much longer than BERT but used the same architecture. Each paper reported results at one or two sizes, making it impossible to determine whether performance improvements came from the method or from the increased scale. The field lacked a scaling methodology—a systematic way to compare the efficiency of different approaches to using additional compute.

Section 3.6 provides exactly this. By comparing model size scaling, training duration scaling, batch size scaling, and ensembling under a fixed 4× compute budget, the paper generates a scaling landscape that allows practitioners to make principled resource allocation decisions. The finding that increasing both model size and training duration (2× size + 2× training) is more effective than increasing either alone (4× size or 4× training) is a concrete result with direct implications: if you have a fixed compute budget for a new model, you should allocate it across both dimensions rather than maximizing one.

The conceptual contribution extends beyond the specific results. The paper demonstrates that scaling studies can and should be part of the core scientific contribution of an NLP paper, not relegated to an ablation table at the end. This has influenced subsequent work (e.g., scaling laws papers, Chinchilla, PaLM) that treat the relationship between compute, model size, and data as a first-class research question.

The paper also shows that scale amplifies other design choices rather than overriding them. Table 15 demonstrates this concretely: T5-Base (which incorporates the non-scaling insights from Sections 3.2–3.5) substantially outperforms a baseline model trained on the same 1 trillion tokens (GLUE 85.97 vs. 84.80, SQuAD 85.44 vs. 83.01). The non-scaling improvements (architectural choice, objective selection, data filtering) are worth roughly 1–2 percentage points across benchmarks, while scaling from 34B to 1T tokens adds another 1–2 percentage points. The innovations are additive: better design and more compute both matter, and their combination produces the best results. This is a more nuanced position than the pure bitter lesson, which argues that compute eventually dominates. T5's results suggest that at the scales studied (up to 11B parameters and 1T tokens), design choices and scale are roughly equally important, and the field should invest in both.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a diverse set of English-language NLP benchmarks. These include the GLUE meta-benchmark (Wang et al., 2018) with 8 constituent tasks (CoLA acceptability, SST-2 sentiment, MRPC paraphrase, STS-B similarity, QQP paraphrase, MNLI inference, QNLI inference, RTE inference); the SuperGLUE meta-benchmark (Wang et al., 2019b) with 10 constituent tasks (BoolQ QA, CB inference, COPA completion, MultiRC QA, ReCoRD QA, RTE inference, WiC disambiguation, WSC coreference, plus DPR coreference added by the authors); CNN/Daily Mail abstractive summarization (Hermann et al., 2015; Nallapati et al., 2016) using the non-anonymized version from See et al. (2017); SQuAD v1.1 question answering (Rajpurkar et al., 2016); and WMT English to German (newstest2013 validation, newstest2014 test), English to French (newstest2014 validation, newstest2015 test), and English to Romanian (WMT 2016 splits) translation. All data is sourced from TensorFlow Datasets. For baseline and exploratory experiments (Sections 3.2–3.6), all reported results are on validation sets to avoid test set contamination during model selection. Final results (Section 3.7) use official test sets: the GLUE and SuperGLUE evaluation servers, standard WMT newstest sets, and the CNN/DM standard test split. SQuAD test evaluation was infeasible for the largest models due to benchmark server computational limits, so SQuAD results in Section 3.7 are reported on the validation set (the previous state-of-the-art also reported validation results for comparison).
Base model. All experiments use the Transformer architecture (Vaswani et al., 2017) as their foundation. The baseline for Sections 3.2–3.6 is an encoder-decoder model with 12 layers each in encoder and decoder, $d_{model} = 768$ , $d_{ff} = 3072$ , $d_{kv} = 64$ , 12 attention heads, and relative position embeddings, totaling approximately 220 million parameters. This size was chosen so that the encoder and decoder are each comparable to a BERT_BASE stack (Devlin et al., 2018). The final T5 models (Section 3.7) span five scales: Small (60M parameters, 6 layers, $d_{model} = 512$ , $d_{ff} = 2048$ , 8 heads), Base (same as baseline, 220M), Large (770M, 24 layers, $d_{model} = 1024$ , $d_{ff} = 4096$ , 16 heads), 3B (2.8B, 24 layers, $d_{model} = 1024$ , $d_{ff} = 16384$ , 32 heads, $d_{kv} = 128$ ), and 11B (11B, 24 layers, $d_{model} = 1024$ , $d_{ff} = 65536$ , 128 heads, $d_{kv} = 128$ ). The 3B and 11B models scale $d_{ff}$ disproportionately because TPUs are most efficient for large dense matrix multiplications. The choice of the Transformer encoder-decoder as the base architecture was itself an experimental outcome: Section 3.2 demonstrated that this architecture, using a denoising objective, outperformed decoder-only language model and prefix LM variants at matched computational cost.
Metrics. Each benchmark uses its standard metric, and where a benchmark comprises multiple tasks, the paper reports the official aggregate. For GLUE, this is the average score across all 8 subtasks (Matthew's correlation for CoLA, accuracy for SST-2/MNLI/QNLI/RTE/WNLI, F1/accuracy average for MRPC/QQP, and Pearson/Spearman average for STS-B). Note that WNLI is excluded from the average in all validation-set tables because the paper never trains on WNLI data (its training set overlaps with WSC) and treats WNLI validation as uninformative due to its adversarial construction. For SuperGLUE, the average is across BoolQ (accuracy), CB (F1/accuracy average), COPA (accuracy), MultiRC (F1a/EM average), ReCoRD (F1/accuracy average), RTE (accuracy), WiC (accuracy), and WSC (accuracy). For translation, BLEU is reported via SacreBLEU v1.3.0 (Post, 2018) with "exp" smoothing and "intl" tokenization. For CNN/Daily Mail, the paper finds ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F to be highly correlated, so it reports ROUGE-2-F as the representative metric. For SQuAD, exact match and F1 are highly correlated, so exact match is reported. All validation-set experiments use greedy decoding (selecting the highest-probability token at each timestep). The final test-set results in Section 3.7 use beam search with width 4 and length penalty $\alpha = 0.6$ for translation and summarization tasks.
Baselines. The primary comparisons throughout Sections 3.2–3.6 are against the paper's own internal baseline (the 220M encoder-decoder trained with the i.i.d. span-corruption objective on C4, pre-trained for $2^{19}$ steps and fine-tuned for $2^{18}$ steps), marked with a star (⋆) in all tables. This self-baseline approach is essential because it enables controlled comparison: any changed factor is isolated against an otherwise identical setup. Additional baselines include: "no pre-training" — training the same model architecture from scratch on each downstream task for the same number of fine-tuning steps (Table 1), which measures the absolute benefit of pre-training; majority voting, which selects the most common prediction among multiple samples and appears implicitly in some multi-task analyses; and comparison to published state-of-the-art results on each benchmark, cited with superscripts in Table 14 (e.g., ALBERT (Lan et al., 2019) for GLUE and SQuAD, RoBERTa (Liu et al., 2019c) for SuperGLUE, Edunov et al. (2018) for WMT translation, Dong et al. (2019) for CNN/DM). The "no pre-training" baseline reveals that pre-training is essential for low-resource tasks: for example, SQuAD exact match drops from 80.88 to 50.31, and SuperGLUE average drops from 71.36 to 53.04 without pre-training, while high-resource translation tasks show marginal gains (EnFr 39.82 vs. 39.77).
Generation budget / compute accounting. Compute is measured in three ways depending on context. For architectural comparisons (Section 3.2), models are compared at matched FLOPs for processing a given input-target pair, with the paper defining $M$ as the FLOPs for an $L+L$ -layer encoder-decoder model and calibrating other architectures to have roughly equivalent computational cost. For scaling experiments (Section 3.6), compute is measured as total training FLOPs (pre-training plus fine-tuning), with the baseline representing 1× compute and variants representing 2× or 4× through changes in model size (larger models require more FLOPs per step), training steps (more steps at the same per-step cost), or batch size (more tokens processed per step). For final model training, pre-training volume is measured in tokens processed: the baseline uses $2^{35} \approx 34$ B tokens, while the final T5 models use approximately 1 trillion tokens (1M steps × batch size 2048 × sequence length 512). This token-based accounting matters because Section 3.4.2 shows that repeating a small dataset harms performance, so compute measurement must account for whether the model sees fresh or repeated data.
Cross-validation / statistical protocol. The paper acknowledges that full statistical rigor (multiple runs with confidence intervals) is prohibitively expensive given the scope of the study. As a practical alternative, it trains the baseline model 10 times from scratch with different random seeds and data orderings, and reports the standard deviation across these runs (Table 1, and the full per-task standard deviations in Table 16). The paper then assumes that this variance applies approximately to experimental variants, using it to flag scores that are within two standard deviations of the best in each experiment (boldfaced in tables) and to assess the significance of differences. The standard deviation analysis reveals that most tasks have low variance (<1% of the baseline score), but low-resource tasks like CoLA, CB, and COPA have much higher variance (e.g., CB F1 standard deviation of 3.237 on a baseline of 91.22), which complicates comparisons on GLUE and SuperGLUE aggregates. For model selection during fine-tuning, the paper saves checkpoints every 5,000 steps (every 1,000 steps for the final individual-task fine-tuning in Section 3.7) and selects the checkpoint with the best validation performance independently for each task. For the final test-set submissions (Section 3.7), results are from the GLUE and SuperGLUE evaluation servers, which enforce single-submission constraints, preventing test-set overfitting.

Main Quantitative Results

The experimental results are organized around the paper's systematic ablation structure, where each section varies one component of the transfer learning pipeline while holding all others constant. I present results following this structure.

Architecture Comparison (Section 3.2, Table 2)

The encoder-decoder architecture with a denoising objective outperforms all other architectural variants at matched computational cost (line marked with ⋆ in Table 2). It achieves GLUE 83.28, SQuAD 80.88, SGLUE 71.36, EnDe 26.98, EnFr 39.82, EnRo 27.65, and CNNDM ROUGE-2 19.24. Compare this to the decoder-only language model with the same denoising objective (2× fewer parameters but same FLOPs): GLUE drops to 74.70, SQuAD collapses to 61.14, SGLUE to 55.02. The prefix LM (which adds fully-visible masking over the input portion) recovers most of this gap (GLUE 81.82, SQuAD 78.94, SGLUE 68.11) but still underperforms the encoder-decoder by 1–2 points across the board.

One non-obvious finding: sharing parameters between encoder and decoder (Enc-dec, shared row: $P$ parameters, $M$ FLOPs) performs nearly as well as the unshared encoder-decoder (GLUE 82.81 vs. 83.28, SQuAD 80.63 vs. 80.88) while having half the parameters. The paper notes that this shared variant also outperforms the prefix LM (82.81 vs. 81.82 GLUE), which has the same parameter count and computational cost, demonstrating that an explicit encoder-decoder attention mechanism is beneficial even when parameters are shared.

Halving the number of layers (Enc-dec, 6 layers: $P$ params, $M/2$ FLOPs) produces a sharp drop (GLUE 80.88, SQuAD 77.59), indicating that depth matters more than parameter count for this architecture.

The language modeling objective (right column of Table 2) consistently underperforms the denoising objective across all architectures and tasks. For the encoder-decoder architecture, LM pre-training drops GLUE from 83.28 to 79.56 and SQuAD from 80.88 to 76.02. The gap is even larger for the decoder-only language model architecture (GLUE 73.78 for LM vs. 74.70 for denoising; SQuAD 53.81 for LM vs. 61.14 for denoising).

The clear conclusion from Table 2: the encoder-decoder architecture with a denoising objective achieves the best performance, justifying its selection as the baseline for all subsequent experiments.

Unsupervised Objective Comparison (Section 3.3, Tables 4–7)

Disparate approaches (Section 3.3.1, Table 4). Comparing three fundamentally different pre-training strategies on the same encoder-decoder architecture, the BERT-style denoising objective (Table 4, row 2: GLUE 82.96, SQuAD 80.65, SGLUE 69.85) substantially outperforms both prefix language modeling (GLUE 80.69, SQuAD 77.99, SGLUE 65.27) and deshuffling (GLUE 73.17, SQuAD 67.61, SGLUE 58.47). The prefix LM does marginally better on translation (EnDe 26.86 vs. 26.78 for BERT-style), but the denoising advantage on understanding tasks is decisive.

Simplifications of the BERT objective (Section 3.3.2, Table 5). Starting from the BERT-style objective (predict the full original text), three simplifications are tested that predict only the corrupted tokens to reduce target sequence length and speed up training. The "replace corrupted spans" variant (the paper's baseline) achieves GLUE 83.28, SQuAD 80.88, and SGLUE 71.36—essentially identical to or slightly better than the full BERT-style objective (GLUE 82.96, SQuAD 80.65, SGLUE 69.85) while being computationally cheaper. The "drop corrupted tokens" variant (no sentinel markers, model must infer where tokens are missing) achieves the highest GLUE (84.44) driven by a large CoLA improvement (60.04 vs. 53.84, see Table 16), but slightly lower SQuAD (80.52) and notably lower SGLUE (68.67). The MASS-style variant (mask tokens but predict full sequence) is intermediate (GLUE 82.32, SQuAD 80.10). All variants outperform the full BERT-style objective on SuperGLUE, with "replace corrupted spans" providing the best balance.

Corruption rate (Section 3.3.3, Table 6). Using the baseline replace-corrupted-spans objective, corruption rates of 10%, 15%, 25%, and 50% are compared. Results show limited sensitivity to this hyperparameter: GLUE ranges from 81.27 (50%) to 83.28 (15%), SQuAD from 79.80 (50%) to 80.96 (25%). The 50% corruption rate shows clear degradation on GLUE and SQuAD (Table 6) but performs slightly better on CNNDM and EnDe. The 15% rate, following BERT precedent, is selected going forward.

Span corruption vs. i.i.d. corruption (Section 3.3.4, Table 7). Deliberately corrupting contiguous spans (rather than independent tokens) with mean span lengths of 2, 3, 5, and 10 is compared against the i.i.d. baseline. The mean span length of 3 achieves the best overall performance: SQuAD 81.84 vs. 80.88 for baseline, SGLUE 72.53 vs. 71.36, CNNDM 19.62 vs. 19.24, with GLUE essentially tied (83.49 vs. 83.28). Mean span length 10 slightly underperforms (GLUE 82.85, SGLUE 70.44). The i.i.d. baseline is roughly equivalent to mean span length ~1.5–2.0 (since 15% corruption with random independent selection produces mostly short spans), so the span-length sweep reveals that moderate span lengths (3–5 tokens) are optimal—long enough to force the model to generate multi-token coherent text (unlike single-token prediction), but not so long that the reconstruction task becomes overwhelmingly difficult.

The key negative result from Sections 3.3.2–3.3.4 is that all denoising variants perform similarly once the basic design is chosen. The paper explicitly states in Section 3.3.5:

"We did not observe a remarkable difference across the many variants of the denoising objectives we explored."

This is a significant finding: the explosion of objective variants in the literature (BERT's random token replacement, SpanBERT's span masking, MASS's suffix masking) may be overengineered. The practical implication is that objective choice should be made on computational efficiency grounds (shorter targets = faster training), not on the expectation of large performance differences.

Pre-training Data Sets (Section 3.4, Tables 8–9)

Data quality filtering (Table 8). C4 (750 GB, filtered) versus Unfiltered C4 (6.1 TB, no heuristic filtering beyond langdetect) shows that filtering provides a consistent improvement across all tasks: GLUE 83.28 vs. 81.46, SQuAD 80.88 vs. 78.78, SGLUE 71.36 vs. 68.04. Unfiltered C4 performs worst on nearly every metric despite being 8× larger, demonstrating that data quality outweighs data quantity when the raw data is noisy web text.

Domain effects (Table 8). Compared to C4 (diverse web text), domain-specific corpora show tradeoffs:

Wikipedia + TBC (20 GB): Best SuperGLUE (73.24 vs. 71.36), driven by a massive MultiRC improvement (EM 50.93 vs. 25.78 from Table 16), because MultiRC's passages come from fiction books (TBC domain). Best SQuAD (82.08 vs. 80.88), because SQuAD passages are from Wikipedia. GLUE is comparable (83.65 vs. 83.28).
RealNews-like (35 GB): Best SGLUE among the smaller corpora (72.38), driven by a ReCoRD improvement (EM 73.72 vs. 68.16 from Table 16), because ReCoRD is a news reading comprehension task.
WebText-like (17 GB): Slightly outperforms C4 on GLUE (84.03 vs. 83.28) and SQuAD (81.42 vs. 80.88), suggesting Reddit upvote filtering provides a useful quality signal.
Wikipedia alone (16 GB): Worse than C4 on GLUE (81.85) and SGLUE (68.01), but better on SQuAD (81.29), again due to SQuAD's Wikipedia sourcing.

The pattern is clear: in-domain unlabeled data provides a meaningful advantage for downstream tasks. However, these domain-specific corpora are 20–40× smaller than C4, which limits their utility for large-scale pre-training (as shown next).

Data set size and repetition (Table 9, Figure 6). When the pre-training corpus is artificially truncated while keeping the pre-training volume fixed at $2^{35}$ tokens:

64× repetition (229 tokens): Nearly identical to full C4 (GLUE 82.87 vs. 83.28, SQuAD 80.97 vs. 80.88).
256× repetition (227 tokens): Small but consistent degradation (GLUE 82.62, SQuAD 79.78).
1024× repetition (225 tokens): Significant degradation (GLUE 79.55, SQuAD 76.27, SGLUE 64.76).
4096× repetition (223 tokens): Severe degradation (GLUE 76.34, SQuAD 70.92, SGLUE 59.29).

The pre-training loss curves in Figure 6 confirm that the model achieves substantially lower training loss with smaller datasets (suggesting memorization rather than generalization). This finding has a critical implication for the final T5 models: training on 1 trillion tokens requires a dataset large enough to avoid excessive repetition. C4 at 750 GB is sufficient; the domain-specific corpora (20–35 GB) would be repeated hundreds of times, which Table 9 shows would significantly harm performance.

Training Strategies (Section 3.5, Tables 10–12)

Fine-tuning methods (Table 10). Full parameter fine-tuning (the baseline) achieves the best performance (GLUE 83.28, SQuAD 80.88, SGLUE 71.36). Adapter layers with small inner dimensionality ( $d = 32$ ) work adequately for SQuAD (79.32 vs. 80.88) but fail catastrophically on high-resource tasks: translation BLEU collapses (EnDe 13.84 vs. 26.98, EnFr 17.88 vs. 39.82). As $d$ increases to 2048, adapter performance approaches full fine-tuning on most tasks (GLUE 82.62, SQuAD 79.40), but the parameter savings diminish. Gradual unfreezing causes a minor degradation across all tasks (GLUE 82.50, SQuAD 79.17, EnDe 26.71) while providing some training speedup.

Multi-task learning (Table 11). No multi-task training strategy matches the pre-train-then-fine-tune baseline. The best multi-task configuration (examples-proportional mixing with $K = 2^{19}$ ) achieves GLUE 81.42 vs. 83.28, SQuAD 79.78 vs. 80.88, SGLUE 67.30 vs. 71.36, and substantially worse translation (EnDe 25.21 vs. 26.98, EnFr 36.30 vs. 39.82). Equal mixing performs dramatically worse (GLUE 76.13), confirming that mixing proportions are a critical hyperparameter. Temperature-scaled mixing with $T = 2$ provides a reasonable compromise (GLUE 81.90) but still trails the baseline. The finding is unambiguous: the standard pre-train-then-fine-tune approach is superior to joint multi-task training across all tasks.

Multi-task pre-training + fine-tuning (Table 12). However, pre-training on a multi-task mixture and then fine-tuning on individual tasks essentially matches the pure unsupervised pre-training baseline: GLUE 83.11 vs. 83.28, SQuAD 80.26 vs. 80.88, SGLUE 71.03 vs. 71.36, EnDe 27.08 vs. 26.98, EnFr 39.80 vs. 39.82. This is a practically important finding: mixing supervised data into pre-training does not hurt, and it provides the practical benefit of monitoring downstream performance throughout pre-training (since supervised tasks are in the mixture). The leave-one-out variant (omitting one task from the pre-training mixture, then fine-tuning on it) performs only slightly worse (GLUE 81.98 vs. 83.11), suggesting multi-task pre-training generalizes reasonably to held-out tasks. Supervised-only multi-task pre-training (no unsupervised data) performs significantly worse (GLUE 79.93, SGLUE 65.36), except for translation tasks where it roughly matches, confirming the essential role of unsupervised pre-training for language understanding.

Scaling (Section 3.6, Table 13)

Under a 4× compute budget increase from the baseline:

1× size, 4× training steps: GLUE 85.33, SQuAD 82.45, SGLUE 74.72, EnDe 27.08.
1× size, 4× batch size: GLUE 84.60, SQuAD 82.52, SGLUE 74.64, EnDe 27.07. Nearly identical to 4× steps, suggesting increasing steps and increasing batch size are roughly interchangeable at this scale.
2× size, 2× training steps: GLUE 86.18, SQuAD 84.18, SGLUE 77.18, EnDe 27.52. Slightly better than 1× size / 4× training on most metrics, and notably better than 4× size / 1× training on GLUE and SQuAD.
4× size, 1× training steps: GLUE 85.91, SQuAD 83.86, SGLUE 78.04. Best SuperGLUE among the non-ensembled approaches, demonstrating that larger models particularly help on SuperGLUE's more complex reasoning tasks.
4× ensembled (separate pre-training + fine-tuning): Achieves the best CNNDM (20.10 vs. 19.24–19.73), EnDe (28.05 vs. 27.08–27.52), and EnRo (28.57 vs. 27.84–28.19). Ensembling is particularly effective for generation tasks.
4× ensembled (fine-tune only, shared pre-training): GLUE 84.05, SQuAD 82.36. Substantially cheaper than full ensembling (shared pre-training) and still outperforms the single-model baseline.

The key scaling insight: increasing both model size and training duration (2× size + 2× training) is more effective than increasing either alone to the 4× level. Additionally, ensembling provides an orthogonal and complementary means of leveraging additional compute, particularly for generation tasks.

Final T5 Results (Section 3.7, Table 14)

Combining the insights from Sections 3.2–3.6 (span-corruption objective, encoder-decoder architecture, C4 data, multi-task pre-training, scaling up model size and training tokens) with beam search and individual-task fine-tuning, T5 achieves state-of-the-art results on 18 of 24 tasks.

GLUE (Table 14, left block). T5-11B achieves an average GLUE score of 90.3, compared to the previous best of 89.4 (ALBERT, Lan et al., 2019). This is driven by particularly strong performance on natural language inference tasks: MNLI-m accuracy 92.2 (vs. 91.3 previous best), MNLI-mm 91.9 (vs. 91.0), RTE 92.8 (vs. 89.2), and WNLI 94.5 (vs. 91.8). RTE accuracy of 92.8 approaches the human performance estimate of 93.6 (Wang et al., 2018). Performance scales monotonically with model size: GLUE improves from 77.4 (Small) to 82.7 (Base) to 86.4 (Large) to 88.5 (3B) to 90.3 (11B).

SQuAD (Table 14, left-middle block). T5-11B achieves Exact Match 91.26 and F1 96.22, compared to the previous best of 90.1 EM and 95.5 F1 (ALBERT). The improvement of over one point on exact match is notable because SQuAD performance had been plateauing, with most recent improvements at fractions of a percentage point. Performance again scales cleanly: 79.10 (Small) → 85.44 (Base) → 86.66 (Large) → 88.53 (3B) → 91.26 (11B).

SuperGLUE (Table 14, right-middle block). T5-11B achieves an average 88.9, compared to the previous best of 84.6 (RoBERTa, Liu et al., 2019c). This nearly matches the human performance baseline of 89.8. Notably, T5 exceeds human performance on MultiRC and ReCoRD: MultiRC F1a 88.1 (human 84.4) and EM 63.3 (human 52.5), ReCoRD F1 94.1 (human 90.6) and EM 93.4 (human 90.0). This suggests that the evaluation metrics may be biased toward machine predictions, or that these tasks' human baselines were set conservatively. On the other hand, T5 significantly underperforms humans on COPA (94.8 vs. 100.0) and WSC (93.8 vs. 100.0), indicating that low-resource commonsense reasoning and coreference resolution remain challenging.

Translation (Table 14, right block). T5 does not achieve state-of-the-art on any WMT translation task: EnDe 32.1 (previous best 33.8; Edunov et al., 2018), EnFr 43.4 (previous best 43.8; Edunov et al., 2018), EnRo 28.1 (previous best 38.5; Lample and Conneau, 2019). The authors attribute this to English-only pre-training lacking the cross-lingual signal that state-of-the-art systems obtain from backtranslation and cross-lingual unsupervised training. Translation performance also scales weakly with model size: EnDe improves from 26.7 (Small) to 32.1 (11B)—a gain of 5.4 BLEU—while GLUE improves by 12.9 points over the same model size range. Translation benefits less from English pre-training scale because the model must generate in a new language it has barely seen during pre-training.

CNN/Daily Mail (Table 14, bottom block). T5-11B achieves ROUGE-1-F 43.52, ROUGE-2-F 21.55, ROUGE-L-F 40.69. This exceeds the previous state-of-the-art on ROUGE-2-F (20.30; Dong et al., 2019) and ROUGE-L-F (40.63), and roughly matches on ROUGE-1-F (43.47). The improvement is concentrated in ROUGE-2 (bigram overlap), suggesting T5 produces summaries with better local coherence.

Non-scaling factors contribution (Table 15). To measure how much the insights from Sections 3.2–3.5 contribute beyond pure scale, the paper compares three Base-sized configurations: the original baseline (34B tokens), a baseline trained on 1T tokens (baseline-1T), and T5-Base (1T tokens + the non-scaling improvements). Baseline-1T improves over the original baseline by 1.5–3 points (GLUE 84.80 vs. 83.28, SQuAD 83.01 vs. 80.88, SGLUE 73.90 vs. 71.36). T5-Base improves over baseline-1T by an additional 1–2.5 points (GLUE 85.97 vs. 84.80, SQuAD 85.44 vs. 83.01, SGLUE 75.64 vs. 73.90). This demonstrates that both scale and methodological improvements matter, with each contributing roughly comparable gains at the Base model size.

Ablation Studies and Robustness Checks

Adapter layer inner dimensionality (Table 10): Low-resource tasks (SQuAD) achieve 79.32 EM with $d = 32$ (only 1.56 points below baseline) while translation tasks require $d \geq 512$ to approach baseline performance (EnDe 23.45 BLEU at $d = 512$ vs. 26.98 baseline). This demonstrates that adapter capacity must be scaled with task size—a single small $d$ cannot serve all tasks. At $d = 2048$ , adapters add parameters comparable to the original model, largely defeating their parameter-efficiency purpose.
Corruption rate robustness (Table 6): Performance is stable across 10–25% corruption (GLUE range 82.82–83.28, SQuAD range 80.38–80.96), but degrades at 50% (GLUE 81.27, SQuAD 79.80). The degradation at 50% is asymmetric: CNNDM and translation tasks are largely unaffected (CNNDM 19.32 vs. 19.24, EnDe 27.01 vs. 26.98), while GLUE and SQuAD suffer. This suggests that understanding tasks are more sensitive to input quality than generation tasks.
Span length robustness (Table 7): Mean span lengths from 2 to 5 produce similar results (GLUE 83.40–83.54, SQuAD 81.84–82.09). Span length 10 slightly degrades GLUE (82.85) and SGLUE (70.44). The i.i.d. baseline is roughly equivalent to the span-based variants, confirming that the specific corruption pattern is not critical.
Data set filtering value (Table 8): Removing C4's heuristic filtering causes uniform degradation (GLUE -1.82, SQuAD -2.10, SGLUE -3.32). Unfiltered C4 is the worst-performing data set on nearly every metric despite being 8× larger than filtered C4, confirming that data quality filtering is essential for web-sourced pre-training data.
Data repetition tolerance (Table 9, Figure 6): Up to 64× repetition causes negligible harm (GLUE 82.87 vs. 83.28). 256× repetition shows small but consistent degradation (GLUE -0.66, SQuAD -1.10). Beyond 1024× repetition, degradation is severe and across the board. This establishes a practical guideline: pre-training data sets should be large enough to limit repetition to well under 100× per example.
Multi-task mixing ratio sensitivity (Table 11): Performance is highly sensitive to the artificial data set size limit $K$ . At $K = 2^{16}$ (near-uniform mixing), GLUE is 80.45; at $K = 2^{19}$ (the optimum), GLUE is 81.42; at $K = 2^{21}$ (large tasks dominate), GLUE drops to 79.83. The optimal $K$ varies by task—translation tasks prefer larger $K$ (more data from those tasks), while understanding tasks prefer intermediate $K$ . This complexity is part of why multi-task training underperforms separate fine-tuning.
Multi-task pre-training + fine-tuning parity (Table 12): Multi-task pre-training followed by individual fine-tuning achieves statistically indistinguishable performance from unsupervised pre-training + fine-tuning across all tasks (GLUE 83.11 vs. 83.28, within baseline standard deviation of 0.235). This robustness check validates the final T5 training procedure and shows that mixing supervised data into pre-training does not cause interference when fine-tuning is performed afterward.
Leave-one-out multi-task pre-training (Table 12): Holding out one downstream task during multi-task pre-training and then fine-tuning on it causes only minor degradation (GLUE 81.98 vs. 83.11 for the full multi-task model). This suggests that multi-task pre-training transfers positively to unseen tasks rather than causing negative transfer.
Ensembling variant comparison (Table 13): Fine-tune-only ensembling (single pre-trained model, 4 separate fine-tunings, ensemble predictions) achieves GLUE 84.05 vs. 84.77 for fully separate ensembles, while being substantially cheaper (shared pre-training, the most expensive phase). The results are notably better on CNNDM (19.57 vs. 20.10) and SQuAD (82.36 vs. 83.09) for full ensembling, but the gap is modest.
Supervised-only multi-task pre-training failure (Table 12): Training solely on supervised tasks without unsupervised data degrades performance on all non-translation tasks (GLUE 79.93 vs. 83.28, SGLUE 65.36 vs. 71.36), confirming that unsupervised pre-training provides knowledge that cannot be replicated by supervised multi-task learning alone.
Scale comparison with non-scaling improvements (Table 15): The decomposition into baseline → baseline-1T (scaling alone) → T5-Base (scaling + non-scaling improvements) shows that the methodological insights from Sections 3.2–3.5 contribute approximately 1–2.5 GLUE points and are roughly comparable in magnitude to the benefit of 30× more pre-training data. This is an unusually clean demonstration that algorithmic and scaling improvements can be additive.

Critical Assessment

The experiments in this paper are among the most comprehensive in the NLP transfer learning literature, and their breadth is genuinely extraordinary—24 tasks, 5 model sizes, 5 architectures, 7+ unsupervised objectives, 6 data sets, 4 corruption rates, 4 span lengths, 3 fine-tuning methods, 7 mixing strategies, 6 scaling strategies. This coverage enables convincing answers to the paper's central questions. Nevertheless, there are important limitations in what the experiments do and do not demonstrate.

Does the text-to-text framework actually improve performance compared to task-specific architectures? The paper demonstrates that a text-to-text encoder-decoder can match or exceed task-specific models at the time (e.g., Table 14 shows T5-11B outperforming BERT-style and ALBERT models on GLUE, SQuAD, and SuperGLUE). However, the paper never runs the crucial experiment: a direct comparison between the text-to-text framework and a task-specific architecture at matched scale with matched pre-training data. T5-11B is pre-trained on 1T tokens of C4, while BERT is pre-trained on ~137B tokens of Wikipedia + BooksCorpus, and ALBERT uses additional ensembling. The performance gap could be due to C4, to the 7× longer pre-training, to the larger model, or to the architecture. Without an experiment where a BERT-style encoder-only model is trained on C4 for 1T tokens with the span-corruption objective (adapted for encoder-only pre-training) and then fine-tuned with a [CLS] classifier, we cannot isolate the contribution of the text-to-text format itself. The paper tacitly acknowledges this gap: when comparing T5-Base to BERT_BASE (Section 3.1.5), the results are "comparable" (SQuAD 80.88 vs. 80.8, MNLI 84.24 vs. 84.4), but T5-Base was pre-trained on 34B tokens while BERT_BASE had 137B—so the text-to-text model achieves similar performance with 4× less pre-training, but this comparison confounds architecture, objective, and data. The paper's architectural comparison in Section 3.2 isolates architecture (comparing encoder-decoder to decoder-only variants), but does not compare to an encoder-only (BERT-style) model because encoder-only architectures cannot handle generation tasks in the unified framework. This is an inherent limitation of the text-to-text premise: by requiring all tasks to be generative, the framework cannot be applied to encoder-only architectures, making a direct controlled comparison impossible.

Does the paper successfully demonstrate that the "coordinate ascent" approach identifies the best combination? The paper acknowledges in Section 3.1 that "this 'coordinate ascent' approach might miss second-order effects" and that "performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive." The concern is real: does the optimal pre-training objective depend on model size? Does the optimal architecture depend on the dataset? The paper provides no evidence either way. The final T5 model uses the span-corruption objective because it performed best at the Base size (Table 7), but there is no experiment showing it remains optimal at 11B parameters. Similarly, the multi-task pre-training caps are scaled with model size using a heuristic (Section 3.7: 710K for Small → 133M for 11B), but the optimal cap was determined at the Base size (Table 11). A single experiment at, say, the Large size sweeping the most important hyperparameters (objective type, cap size, span length) would have gone a long way toward validating the coordinate ascent methodology. The final results (Table 14) show that the chosen configuration works very well, but they do not prove it is optimal.

Is the scaling analysis (Section 3.6) a genuine scaling law, or just a scaling trend? The 4× compute analysis is informative but limited to a single scaling factor—we see what happens when compute is quadrupled from a single baseline point. A proper scaling law (like Hoffmann et al., 2022) varies compute over multiple orders of magnitude and fits parametric relationships. The paper's analysis is better characterized as a scaling comparison: at one specific compute multiplier, it compares different allocation strategies. Extrapolating from this single point to, say, a 100× compute budget is not supported by the data. The final T5 models (up to 11B parameters and 1T tokens) represent a much larger scaling factor, but the systematic comparison from Section 3.6 is not repeated at that scale—we don't know whether the "2× size + 2× training" advantage persists when going from Large to 11B.

How much do the final results depend on the specific test sets used? Table 14 reports on test sets, but the GLUE and SuperGLUE test sets are evaluated via the benchmark servers, which enforce single submissions—this is a strength, preventing test-set overfitting. However, SQuAD results are on the validation set because "the computational resources on this server are insufficient for obtaining predictions from our largest models" (Section 3.7). This is a meaningful gap because SQuAD validation performance is known to be somewhat inflated relative to the test set due to the community's extensive use of the validation set for model development over several years. The SQuAD exact match of 91.26, while ostensibly beating ALBERT's test score of 90.1, is not a clean test-set comparison. The paper partially addresses this by comparing to ALBERT's validation score (which ALBERT also reported), but the state-of-the-art claim should be interpreted cautiously.

Are the translation results genuinely disappointing, or an artifact of the evaluation setup? The paper notes that "most of the best results on these tasks use backtranslation... and additional forms of cross-lingual unsupervised training" (Section 3.7). This is an understatement: the English-to-Romanian state-of-the-art (38.5 BLEU) used cross-lingual masked language modeling pre-training (Lample and Conneau, 2019), which provides a fundamentally different pre-training signal compared to T5's English-only denoising. The comparison is informative (it shows that English-only pre-training is insufficient for translation) but not controlled—a fair comparison would require a T5 variant pre-trained multilingually or with cross-lingual objectives. The paper's vocabulary design (10:1 English-to-other-language mixture for SentencePiece training) gives the model some exposure to German, French, and Romanian subwords, but this is far from the bilingual signal state-of-the-art systems use. The translation result is therefore better framed as a lower bound on what English-only pre-training can achieve, rather than a failure of the T5 approach per se.

What about the claim that T5 "nearly matches" human performance on SuperGLUE? T5-11B achieves 88.9 vs. human 89.8, a difference of 0.9 points. However, disaggregating the tasks reveals that T5 substantially exceeds humans on MultiRC and ReCoRD (where the metrics may be poorly calibrated) while significantly underperforming humans on COPA (94.8 vs. 100.0) and WSC (93.8 vs. 100.0). The aggregate therefore masks a bimodal distribution: T5 is superhuman on reading comprehension tasks and subhuman on low-resource commonsense reasoning tasks. The overall average is "close to human" only because the tasks where T5 wins (MultiRC, ReCoRD) have metrics that allow machine scores to exceed human baselines. A more accurate characterization would be that T5 demonstrates near-human aggregate performance but clearly subhuman performance on tasks requiring reasoning over limited data.

Missing experiments that would have strengthened the paper:

A direct BERT vs. T5 comparison at matched pre-training data and token count. Train BERT_BASE on C4 with the span-corruption objective (adapted for masked token prediction) for 34B tokens and fine-tune normally, then compare to T5-Base. This would isolate the architectural contribution.
A scalability check on the key hyperparameters selected at the Base size. For example, running the span-length comparison at Large scale to verify that mean span length 3 remains optimal.
A confidence interval analysis for the final T5 results. The baseline 10-run experiment in Table 1 provides variance estimates, but no such analysis is performed for the larger models. The variance on low-resource tasks (CoLA, CB, COPA) is high enough that some of the reported improvements might not be statistically significant.
An analysis of catastrophic forgetting in multi-task pre-training. When T5 pre-trains on a mixture of tasks and then fine-tunes on a single task, does performance on other tasks degrade? The leave-one-out experiment (Table 12) partially addresses this, but only for the held-out task, not for the tasks that were in the pre-training mixture.
A breakdown of whether the text-to-text format imposes a cost on classification tasks where a simple [CLS] token classifier might be more parameter-efficient. The paper handles classification by generating label words, which means the decoder must learn to produce exactly one token from a small set. An experiment comparing text-to-text classification to a classification head (on tasks where both are applicable) would clarify whether the generative formulation imposes a penalty.

In summary, the experiments strongly support the paper's central claim that a unified text-to-text framework, combined with a systematic study of transfer learning design choices and aggressive scaling, produces state-of-the-art results across a wide range of NLP tasks. The ablations convincingly demonstrate which factors matter (denoising vs. language modeling, data quality, model scale) and which do not (specific denoising variant, corruption rate, span length within reasonable ranges). The main limitations are: (1) the inability to compare against encoder-only architectures in a controlled way due to the text-to-text constraint, (2) the coordinate ascent methodology's untested assumption that optimal settings are independent, (3) the reliance on a single scaling factor (4×) rather than a scaling law spanning orders of magnitude, and (4) the aggregate metrics masking important per-task variations in performance relative to human baselines.

6. Limitations and Trade-offs

The Inability to Compare Against Encoder-Only Architectures in a Controlled Way

The assumption or constraint. The text-to-text framework—the paper's central methodological contribution—fundamentally requires all tasks to be formulated as generating text autoregressively from a decoder. This means that encoder-only architectures like BERT, which produce representations at each input position and use a separate classification head, cannot be evaluated within the framework at all. The paper is explicit about this from the outset:

"For example, 'encoder-only' models like BERT (Devlin et al., 2018) are designed to produce a single prediction per input token or a single prediction for an entire input sequence. This makes them applicable for classification or span prediction tasks but not for generative tasks like translation or abstractive summarization. As such, none of the model architectures we consider are identical to BERT or consist of an encoder-only structure." (Section 3, introduction, emphasis added)

The consequence. The paper's central architectural claim—that the encoder-decoder Transformer is the best architecture for transfer learning—is established by comparing encoder-decoder models against decoder-only language models and prefix LMs (Section 3.2, Table 2). But the dominant architecture in the field at the time was encoder-only: BERT, RoBERTa, ALBERT, and their variants. These architectures are never compared against in a controlled setting because they cannot be plugged into the text-to-text framework. The prefix LM architecture (which the paper does compare) is argued to "closely resemble BERT for classification tasks" (Section 3.2.1) because the fully-visible input prefix serves as an analog to BERT's bidirectional encoder, with the label generated as the first output token. However, this analogy is imperfect: BERT processes the input bidirectionally and makes a prediction through a dedicated classification token representation, while the prefix LM must learn to generate exactly the label string from the decoder. The paper provides no evidence that these are equivalent.

The practical consequence is that a practitioner choosing between BERT-style fine-tuning and T5-style text-to-text fine-tuning for a classification-only deployment has no controlled comparison to guide them. The paper demonstrates that T5 outperforms published BERT-based results (Table 14), but this comparison confounds architecture, pre-training data (C4 vs. Wikipedia + BooksCorpus), pre-training volume (1T tokens vs. 137B), model scale, and the text-to-text formulation. The incremental benefit of the text-to-text format itself, independent of scale and data, is unknown.

What evidence exists in the paper. The only architectural comparison is in Table 2, and it explicitly excludes encoder-only models. When comparing T5-Base to BERT_BASE on specific tasks (Section 3.1.5), the paper notes the results are "comparable" (SQuAD EM 80.88 vs. BERT's 80.8, MNLI-m accuracy 84.24 vs. BERT's 84.4), but T5-Base was pre-trained on 34B tokens while BERT_BASE used 137B tokens—a 4× pre-training disadvantage for T5. The paper does not run a BERT model on C4 with matched pre-training volume, which would isolate the architectural contribution.

Mitigation status. The paper does not address this limitation. It is inherent to the text-to-text premise: the framework's unification requires a generative interface, which encoder-only architectures lack. The paper partially sidesteps the issue by showing that encoder-decoder models can achieve state-of-the-art classification performance (Table 14), but this demonstrates sufficiency, not optimality. A practitioner who only needs classification and wants the simplest possible model still cannot determine from this paper whether a fine-tuned BERT with a classification head would match T5 at comparable scale.

The Cost of Difficulty Estimation and Strategy Selection Is Unaccounted For in the 4× Efficiency Claims

The assumption or constraint. The paper's compute-optimal allocation framework in the reference example (not this paper, but drawn from the parallel T5 analysis context) depends on estimating prompt difficulty to decide which test-time strategy to use. Even within the scope of the T5 paper itself, there is a parallel issue: the final model selection and hyperparameter tuning that produce the headline results in Table 14 involve an unaccounted computational cost. More broadly, every experiment in Sections 3.2–3.6 requires training a full model from scratch and evaluating it across 24 tasks to determine which design choice is optimal. These search costs are not amortized into the reported efficiency or performance numbers. The paper acknowledges this implicitly in the design of its scaling experiments: "You were just given 4× more compute. How should you use it?" (Section 3.6), but does not account for the compute required to answer that question in the first place.

The consequence. A practitioner wanting to replicate T5's results on a new domain or task family faces a substantial unaccounted cost: they must either adopt the paper's exact design choices (which were optimized for the specific benchmarks studied) or repeat the ablation process to find optimal choices for their own setting. The paper's "coordinate ascent" methodology (Section 3.1) is explicitly described as possibly missing second-order effects, and the paper provides no guidance on how to efficiently transfer the optimal configuration to a new setting without repeating the full ablation suite.

More importantly, the claim that "all denoising objectives perform similarly" (Section 3.3.5) and that objective choice should be made on computational efficiency grounds is a conclusion drawn from experiments on the specific set of 24 benchmarks studied. If a practitioner's target task shares characteristics with one of these benchmarks (e.g., is classification-heavy or generation-heavy), the optimal objective might differ. The paper provides no meta-analysis of which objective variants favor which task types, making it impossible to generalize without additional experimentation.

What evidence exists in the paper. The scope of the empirical study is 24 specific English-language benchmarks (Section 2.3). Every design choice—architecture (Table 2), objective (Tables 4–7), dataset (Table 8), fine-tuning strategy (Tables 10–12), and scaling allocation (Table 13)—is evaluated on the same fixed set of tasks. The paper does not include any held-out tasks from different domains to test the generalizability of its recommended configuration. When the final T5 model is assembled (Section 3.7), the design choices (span-corruption objective, multi-task pre-training, specific mixing rates, model size scaling) are justified by reference to earlier ablation tables, but there is no experiment validating that the selected configuration remains optimal when transferred to, say, biomedical text, legal documents, or code.

Mitigation status. The paper partially addresses generalizability by including a diverse set of tasks (classification, summarization, QA, translation) within the 24 benchmarks. The fact that the same design choices work well across this diverse set provides some evidence of robustness. However, this is only partial mitigation—all benchmarks are English-language, all involve relatively formal text (news, Wikipedia, books, web text), and none require specialized domain knowledge beyond what appears in the pre-training data. The paper does not suggest any transfer methodology for adapting the optimal configuration to new domains without full re-ablation.

English-Only Pre-Training Fundamentally Limits Performance on Multilingual and Translation Tasks

The assumption or constraint. T5 is explicitly designed as an English-language transfer learning system. The pre-training corpus (C4) is filtered to only English text, the vocabulary is trained on a 10:1 English-to-other-language mixture (Section 3.1.3), and the unsupervised objective operates solely on English text. The translation tasks (English to German, French, Romanian) are included to test "the behavior of transfer learning in the high-resource regime" (Section 3.1.5) and to see whether English pre-training alone can bootstrap non-English generation.

The consequence. T5 fails to achieve state-of-the-art results on all three WMT translation benchmarks (Table 14). The gap is particularly stark for English-to-Romanian, a lower-resource translation pair, where T5-11B achieves BLEU 28.1 versus the state-of-the-art 38.5 (Lample and Conneau, 2019). The paper correctly identifies the cause:

"This may be in part due to our use of an English-only unlabeled data set. We also note that most of the best results on these tasks use backtranslation... and additional forms of cross-lingual unsupervised training." (Section 3.7)

Translation performance also scales weakly with model size compared to English understanding tasks. From T5-Small (60M) to T5-11B (11B), EnDe BLEU improves from 26.7 to 32.1 (a gain of 5.4 BLEU), while GLUE improves from 77.4 to 90.3 (a gain of 12.9 points) and SQuAD EM improves from 79.10 to 91.26 (12.2 points). The diminishing returns suggest that English pre-training provides limited transfer to non-English generation, and that further scaling of English-only models will not close the gap to multilingual systems.

Beyond translation, this limitation implies that T5 is fundamentally unsuitable for any non-English NLP task. The "text-to-text" framework is language-agnostic in principle, but the implementation is English-only: the vocabulary, the pre-training data, and the benchmark suite all assume English input and output. A practitioner working with, say, Arabic or Japanese text cannot use T5 out of the box—they would need to retrain the vocabulary, recreate a filtered C4-equivalent corpus in their target language, and re-run pre-training.

What evidence exists in the paper. Table 14 shows T5-11B failing to surpass the previous state-of-the-art on any translation task, with EnRo trailing by 10.4 BLEU. Table 1 shows that the baseline model's pre-training provides minimal benefit for EnFr translation (39.82 BLEU with pre-training vs. 39.77 without), confirming that English pre-training contributes almost nothing to this high-resource translation task. The scaling experiments (Table 13) show that even with 4× compute, translation gains are modest (EnDe 27.08–28.05 vs. baseline 26.98), whereas understanding tasks gain 2–5 points.

Mitigation status. The paper is transparent about this limitation. Section 4.2 explicitly identifies "language-agnostic models" as a key direction for future work:

"We are interested in further investigating language-agnostic models, i.e. models that can perform a given NLP task with good performance regardless of the text's language. This is an especially pertinent issue given that English is not the native language for the majority of the world's population."

The paper's vocabulary design (including German, French, and Romanian subwords in the SentencePiece model) is a partial mitigation that allows the model to at least tokenize non-English text, but this is far from true multilingual pre-training. No experiments with non-English downstream tasks beyond translation are included. The limitation remains fundamental: T5 is an English-language model, and its results do not transfer to other languages.

The Computational Cost and Practical Deployment Barriers of 11B-Parameter Models

The assumption or constraint. The paper's best results are achieved with T5-11B, an 11-billion-parameter model trained on 1 trillion tokens. The paper assumes that the benefits of scale justify the computational costs, and that the scaling trends observed (Table 13, Table 14) will continue to hold for practitioners with access to large-scale compute infrastructure. Section 4.2 acknowledges this tension:

"The inconvenience of large models. An unsurprising but important result from our study is that larger models tend to perform better. The fact that the hardware used for running these models is continually getting cheaper and more powerful suggests that scaling up may continue to be a promising way to achieve better performance... However, it will always be the case that there are applications and scenarios where using a smaller or less expensive model is helpful, for example when performing client-side inference or federated learning." (Section 4.2)

The consequence. The 11B model's practical deployment is extremely expensive. The paper reports that T5-11B uses $d_{ff} = 65536$ and 128 attention heads, meaning the feed-forward layers alone contain the vast majority of the 11 billion parameters. Inference with this model requires substantial accelerator memory and compute, even for single-example processing. The paper provides no inference latency measurements, no memory footprint analysis, and no discussion of quantization, pruning, or distillation to make deployment feasible.

The practical consequence is stark: most of T5's improvements over prior work come from the 3B and 11B models, not from the Base or Large variants that are more realistically deployable. T5-Base (220M parameters) achieves GLUE 82.7 on the test set (Table 14), which is below the 2019 state-of-the-art of 89.4 (ALBERT, which achieved this through ensembling but with similar parameter counts due to parameter sharing). T5-Large (770M) achieves GLUE 86.4, still below the state-of-the-art. It is only at T5-3B (88.5) and T5-11B (90.3) that T5 pulls ahead. A practitioner who can only deploy a Base or Large model due to latency, memory, or cost constraints gets a system that underperforms the best published results—and the paper provides no guidance on how to close that gap without scaling.

Furthermore, the scaling of $d_{ff}$ specifically (reaching 65,536 in the 11B model) was motivated by TPU efficiency for large matrix multiplications. On GPU hardware, which is far more common in both research and deployment settings, these enormous feed-forward layers may be comparatively less efficient, and the optimal architecture for GPU deployment might differ. The paper's design choices are implicitly optimized for TPU Pods, which are accessible primarily within large industrial labs.

What evidence exists in the paper. Table 14 shows the performance of all five model sizes on all tasks. The gap between T5-Large (the largest "conventional" model) and T5-11B is substantial: 3.9 GLUE points, 4.6 SQuAD EM points, 6.6 SuperGLUE points. Table 15 shows that even with 1T tokens of pre-training, the Base model (baseline-1T) underperforms T5-Base, and T5-Base itself is far from state-of-the-art. The scaling trends in Table 13 suggest that doubling model size provides roughly comparable gains to doubling training steps, but both approaches require exponential increases in compute for linear performance improvements—the regime of diminishing returns has clearly been reached even within the paper's experiments.

Mitigation status. The paper discusses smaller models (T5-Small at 60M parameters, T5-Base at 220M) and releases all model checkpoints to facilitate distillation and fine-tuning research. Section 4.2 explicitly calls for "research on methods that achieve stronger performance with cheaper models so that transfer learning can be applied where it will have the most impact. Some current work along these lines include distillation, parameter sharing, and conditional computation." However, the paper does not perform any distillation experiments itself, does not evaluate the smaller models' performance relative to comparably sized competitors (e.g., DistilBERT, ALBERT with parameter sharing), and does not provide deployment-focused metrics (latency, memory, throughput) for any model size. The mitigation is purely aspirational—the paper identifies the problem but contributes no solutions.

The Coordinate Ascent Methodology Assumes Independence of Design Factors Without Validation

The assumption or constraint. The entire empirical methodology of Sections 3.2–3.6 is structured around changing one factor at a time from a fixed baseline and measuring the effect. This "coordinate ascent" approach is explicitly chosen for feasibility, but the paper acknowledges its limitation:

"This 'coordinate ascent' approach might miss second-order effects (for example, some particular unsupervised objective may work best on a model larger than our baseline setting), but performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive." (Section 3.1)

The consequence. The paper's recommendations—use an encoder-decoder architecture, a span-corruption denoising objective with mean span length 3 and 15% corruption rate, C4 data, multi-task pre-training, and scale up both size and training steps—are based on optimizing each factor independently at the Base model scale (~220M parameters, 34B pre-training tokens). The final T5-11B model applies these recommendations at a scale 50× larger (11B parameters, 1T tokens), with no validation that the optimal choices at Base scale remain optimal at 11B scale.

This is not a minor concern. The paper's own data provides reasons to doubt independence:

The optimal data mixing proportions for multi-task learning (Table 11) are shown to depend on $K$ , the artificial dataset size limit. The final T5 models scale $K$ with model size using a heuristic (710K for Small → 133M for 11B) with no ablation at larger scales.
The corruption rate experiment (Table 6) shows that 50% corruption hurts GLUE and SQuAD at Base scale but translation tasks are unaffected. At 11B scale, with its vastly greater capacity, perhaps 50% corruption would be beneficial—the harder pre-training task might prevent overfitting that becomes more severe with larger models.
The span length experiment (Table 7) shows that longer spans (10 tokens) slightly degrade performance at Base scale. At 11B scale, with its greater generative capacity, longer spans might provide better training signal for text generation.
The architectural comparison (Table 2) is conducted at Base scale. The finding that parameter sharing performs nearly as well as the full encoder-decoder might reverse at 11B scale, where the shared model's reduced capacity could become a bottleneck.

The paper's final model configuration is therefore a conjecture—an informed guess that the Base-scale optimal settings generalize to 11B scale—not an empirical finding. A single sanity-check experiment at Large scale (770M parameters) sweeping the most critical hyperparameters (objective type, span length, mixing cap) would have substantially strengthened the claim.

What evidence exists in the paper. The only cross-scale comparison that touches on this issue is Table 13, which varies scaling strategy (1× size/4× training vs. 2× size/2× training vs. 4× size/1× training) at the Base scale and shows they are roughly comparable. This experiment varies the allocation of compute between model size and training steps, but keeps all other design factors (objective, data, architecture, corruption rate) fixed—it does not test whether those fixed factors remain optimal at larger scales. Table 15 decomposes T5-Base's gains into scaling (baseline-1T) and non-scaling factors, but again at a single scale (Base).

Mitigation status. The paper acknowledges the limitation but provides no mitigation beyond transparency. The combinatorial explosion of possible configurations is genuinely prohibitive—even a modest sweep of 3 objectives × 3 span lengths × 3 corruption rates × 3 mixing strategies at 3 model sizes would require 81 full pre-training runs, each costing millions of TPU-hours. However, the paper could have strengthened its claims by:

Running the single most important sweep (e.g., objective variants) at one additional scale (Large or 3B) to test for interactions.
Providing a theoretical argument or prior literature suggesting which factors are most likely to have scale-dependent optima.
Using the validation loss curves from different scale models to predict whether certain design choices were becoming more or less favorable with scale.

None of these mitigations are attempted. The coordinate ascent limitation remains a significant uncertainty in the claim that T5's configuration is the "optimal" combination of the studied design choices.

Low-Resource Tasks Exhibit High Variance That Undermines the Reliability of Aggregate Benchmark Scores

The assumption or constraint. The paper treats the GLUE and SuperGLUE average scores as its primary evaluation metrics for understanding tasks. These averages weight all constituent tasks equally, regardless of dataset size, variance, or difficulty. The paper acknowledges in Section 3.1.5 that some low-resource tasks have high inter-run variance, but continues to use the aggregate averages as the primary comparison metric throughout:

"We caution that the high inter-run variance of CoLA, CB, and COPA can make it harder to compare models using the GLUE and SuperGLUE scores alone." (Section 3.1.5)

The consequence. Small changes in aggregate GLUE and SuperGLUE scores—including some of the differences the paper uses to select the "best" configuration—may be statistically indistinguishable from noise. The baseline model's 10-run standard deviation on GLUE is 0.235 (Table 1), which means that many of the comparisons in the ablation tables (where differences of 0.1–0.5 GLUE points are used to declare one variant better than another) are within the noise range.

Specific examples where this matters:

The choice of corruption rate (Table 6): GLUE ranges from 82.82 (10%) to 83.28 (15%) to 83.00 (25%). The difference between 10% and 15% is 0.46, roughly 2× the baseline standard deviation—potentially significant. But the difference between 15% and 25% is 0.28, roughly 1.2× the standard deviation—much less clearly significant.
The span-corruption vs. i.i.d. comparison (Table 7): GLUE 83.49 (mean span 3) vs. 83.28 (i.i.d.), a difference of 0.21, less than the baseline standard deviation of 0.235. The paper describes this as "slightly (but significantly) outperforms," but with only a single run per configuration, statistical significance is untestable.
The multi-task pre-training + fine-tuning vs. baseline comparison (Table 12): GLUE 83.11 vs. 83.28, a difference of 0.17. The paper concludes they are "comparable," but with the baseline standard deviation of 0.235, a single-run difference of 0.17 is entirely consistent with either being genuinely better.

The situation is worse for SuperGLUE, where the baseline standard deviation is 0.416 (Table 1). Many of the SuperGLUE comparisons in the ablation tables involve differences smaller than this:

The span corruption comparison (Table 7): SGLUE ranges from 70.44 (mean span 10) to 72.53 (mean span 3), a range of 2.09, which is about 5× the standard deviation, so the span-length effect on SuperGLUE is likely real. But the difference between mean span 3 (72.53) and mean span 5 (72.23) is 0.30, well within noise.
The pre-training dataset comparison (Table 8): SGLUE for C4 is 71.36, for WebText-like is 71.40, a difference of 0.04—meaningless.

What evidence exists in the paper. Table 1 provides standard deviations for the baseline model across 10 runs. The per-task breakdown in Table 16 reveals the high-variance tasks: CoLA (MCC standard deviation 1.111 on baseline 53.84), CB (F1 standard deviation 3.237 on baseline 91.22), COPA (standard deviation 2.741 on baseline 66.20). The SQuAD exact match has standard deviation 0.343, and SGLUE average has 0.416. The paper treats these as representative of all experimental variants, but does not verify this assumption—different pre-training objectives or data sets might have different run-to-run variance. A configuration that converges more stably (lower variance but same mean) would be genuinely preferable for practical deployment, but the paper's single-run methodology cannot distinguish between a small mean difference and a difference in stability.

Mitigation status. The paper's approach—running the baseline 10 times and assuming the variance generalizes—is a reasonable pragmatic compromise given the scale of the experimental program. Boldfacing scores within two standard deviations of the maximum provides a rough significance filter. However, this is insufficient for the fine-grained comparisons that dominate the ablation sections. The paper never computes a confidence interval for the difference between two configurations, never reports whether apparent "best" configurations are significantly different from the second-best, and never corrects for multiple comparisons (dozens of pairwise comparisons across tables). The mitigation is therefore partial at best. A practitioner reading the paper should treat differences smaller than ~0.5 GLUE points or ~1.0 SuperGLUE points as ambiguous, regardless of which direction the point estimate favors.

7. Implications and Future Directions

How This Work Changes the Landscape

T5 is best understood not as a paradigm shift but as a methodological reframing and an empirical diagnostic for the transfer learning field. It does not introduce fundamentally new pre-training objectives or architectures—the encoder-decoder Transformer with a denoising objective was already known from work like MASS (Song et al., 2019). Instead, it reorganizes the field's research methodology around the insight that unification enables controlled science, and then uses that controlled science to identify which design choices actually matter and which are noise.

The primary conceptual shift is from fragmentation to systematization. Before T5, the NLP transfer learning literature was a collection of point solutions, each with its own architecture, objective, corpus, and fine-tuning protocol, making cross-comparison impossible. After T5, the text-to-text framework provides a fixed coordinate system where any factor—architecture, objective, data, training strategy—can be varied independently while holding all others constant. This is less glamorous than inventing a new objective, but it is arguably more valuable for the field's long-term health: it converts the question "does this new pre-training objective help?" into a falsifiable hypothesis that can be tested in a controlled setting, rather than a claim buried in a paper that simultaneously changes five other variables.

The paper's diagnostic findings are the concrete output of this reframing. The most important of these is the flatness of the denoising objective landscape (Section 3.3.5). By demonstrating that BERT-style masking, MASS-style masking, i.i.d. corruption, span corruption, and entirely dropping tokens all produce statistically indistinguishable performance when other factors are held constant, the paper effectively nullifies a substantial portion of the pre-training objective design literature that preceded it. The community had invested significant effort in designing clever corruption strategies—BERT's random token replacement, XLNet's permutation language modeling, SpanBERT's contiguous span masking—each accompanied by theoretical justifications for their superiority. T5's null result suggests these justifications were post-hoc: the objectives all work because they all require the model to build rich contextualized representations to fill in missing information, and the specific form of the missing information matters little. This finding redirects research attention from objective engineering (which the paper shows is a low-return activity) to scale, data quality, and architecture (which the paper shows provide large, robust gains). The practical advice—choose the denoising objective that minimizes computational cost—is a direct consequence.

The second major insight is the quantification of the in-domain pre-training effect (Section 3.4.1). While it was intuitively obvious that pre-training on Wikipedia helps SQuAD (since SQuAD passages are from Wikipedia), T5 provides precise magnitudes for this effect across multiple tasks and domains: Wikipedia + TBC nearly doubles MultiRC exact match (25.78 to 50.93) because MultiRC uses fiction passages; RealNews-like data boosts ReCoRD by 5.6 exact match points (68.16 to 73.72) because ReCoRD uses news articles. This quantification transforms a vague intuition into a decision-relevant tradeoff: a general-purpose model trained on diverse data (C4) achieves reasonable performance everywhere but is never optimal for any single task, while a domain-specific model can dominate on its domain but collapses on others. The paper's contribution is not the observation of this tradeoff but the empirical specificity that allows practitioners to estimate the benefit of in-domain pre-training for their own setting.

The third shift is the operationalization of the "bitter lesson" as a controlled experimental variable. Section 3.6 asks a concrete, actionable question: given a 4× compute budget increase, how should you allocate it? The answer—that increasing both model size and training duration (2× size + 2× training) slightly edges out either extreme—provides a resource allocation heuristic that practitioners can apply immediately. More importantly, the framing of scale as an experimental variable to be optimized, not an afterthought, has influenced subsequent work on scaling laws. The paper demonstrates that scaling studies belong in the core scientific contribution of an NLP paper, and that comparing scaling strategies at matched compute is as important as comparing objectives or architectures.

The paper also provides conceptual closure on several debates that were active in 2019:

Denoising vs. language modeling pre-training. The finding that denoising objectives significantly outperform language modeling (Table 2: GLUE 83.28 vs. 79.56 for encoder-decoder, Table 4: BERT-style 82.96 vs. prefix LM 80.69) settles a question that had been debated since GPT and BERT proposed opposing approaches. The mechanism—bidirectional context during encoding—is clearly the driver.
Encoder-decoder vs. decoder-only architectures. The prefix LM (which adds bidirectional context to a decoder-only model) recovers much but not all of the encoder-decoder's advantage (Table 2: prefix LM GLUE 81.82 vs. encoder-decoder 83.28). The remaining gap, combined with the shared-parameter encoder-decoder's superiority over the prefix LM at identical parameter count, suggests that explicit encoder-decoder cross-attention is genuinely beneficial, not just an artifact of having separate encoder and decoder parameters.
Multi-task training vs. pre-train-then-fine-tune. The finding that multi-task training underperforms separate fine-tuning (Table 11) but that multi-task pre-training followed by fine-tuning matches the baseline (Table 12) clarifies the role of multi-task learning in transfer learning pipelines: it is useful as a pre-training augmentation (allowing monitoring of downstream performance during pre-training) but not as a replacement for task-specific fine-tuning.

Research directions that become more attractive after T5:

Scaling laws for transfer learning. The paper's 4× scaling analysis is a proof of concept for systematic scaling studies. Extending this to orders-of-magnitude larger compute budgets and fitting parametric scaling laws (à la Hoffmann et al., 2022) is a natural next step.
Data quality and filtering. The finding that Unfiltered C4 performs worse than C4 despite being 8× larger (Table 8) elevates data quality from an engineering detail to a first-class research problem. Understanding which heuristics matter most and how to automate data filtering become important questions.
Efficient architectures for the text-to-text framework. The shared-parameter encoder-decoder (Table 2) performs nearly as well as the full encoder-decoder while halving parameters, suggesting that parameter-efficient architectures for sequence-to-sequence models are a promising direction.
Domain-adaptive pre-training. The in-domain effect documented in Section 3.4.1 motivates research on methods for efficiently adapting a general-purpose pre-trained model to a target domain, rather than pre-training domain-specific models from scratch.

Research directions that become less attractive after T5:

Incremental variations on denoising objectives. The flatness of the objective landscape (Section 3.3.5) suggests that proposing marginally different corruption strategies without scaling up other factors is unlikely to produce meaningful gains. The paper's finding that even dropping tokens entirely (no sentinel markers, no masking) performs competitively is a strong signal that the specific corruption pattern is not where the field's research effort should be spent.
Decoder-only architectures for general-purpose transfer learning. The encoder-decoder architecture's consistent advantage over decoder-only variants (even when controlling for parameter count and FLOPs) suggests that the field's interest in pure language model pre-training (as in GPT-2, GPT-3) for understanding tasks should be tempered by the recognition that bidirectional encoding provides a genuine representation quality advantage.

Follow-Up Research This Work Enables

Scaling laws for text-to-text transfer learning across multiple orders of magnitude. T5's Section 3.6 provides a scaling comparison at a single 4× multiplier, but does not establish a parametric relationship between compute, model size, training tokens, and downstream performance. A natural follow-up would train T5-style models at 10–20 different scales spanning 6+ orders of magnitude of compute (e.g., from 10M to 100B parameters), fit power-law relationships for how GLUE, SQuAD, and SuperGLUE performance scale with FLOPs, parameters, and tokens, and determine whether the optimal allocation between model size and training duration changes with total budget. The paper's baseline 10-run standard deviation analysis (Table 1) provides the statistical foundation for such a study by quantifying the noise floor that scaling laws must rise above. A strong result would be a Chinchilla-style scaling law (Hoffmann et al., 2022) but for transfer learning rather than language modeling loss, specifying the compute-optimal model size and pre-training token count for a given downstream task suite. The paper's observation that translation tasks scale weakly with model size (EnDe improves only 5.4 BLEU from Small to 11B while GLUE improves 12.9 points) suggests that scaling laws may be task-family-dependent, which would be a practically important finding for resource allocation.

Controlled comparison between T5's text-to-text format and task-specific architectures at matched scale. The paper never compares the text-to-text framework against BERT-style encoder-only fine-tuning under controlled conditions—same pre-training corpus (C4), same pre-training volume, same model scale. A direct follow-up would train BERT_BASE on C4 for 34B tokens using a masked language modeling objective (adapted for span corruption, predicting only corrupted tokens at the masked positions), fine-tune it with a standard classification head on GLUE and SQuAD, and compare to T5-Base. The key measurement is whether the text-to-text generative formulation imposes a performance penalty on pure classification tasks compared to a dedicated classification head. If T5 matches or exceeds BERT at identical scale, the text-to-text framework's engineering convenience comes with no performance cost, strengthening the case for its adoption. If BERT significantly outperforms T5 at the same scale on classification, the field needs to develop hybrid approaches that use unified pre-training but allow task-specific fine-tuning heads. This experiment would also isolate the contribution of the encoder-decoder architecture versus the encoder-only architecture for understanding tasks, which the paper's Table 2 cannot do because it excludes encoder-only models.

Automated data filtering and quality estimation for web-scale pre-training corpora. C4's heuristic filtering pipeline was designed manually and tested holistically—the paper shows that the full set of filters helps (Table 8, filtered C4 vs. Unfiltered C4) but does not ablate individual filters to determine which ones matter most. A systematic follow-up would: (1) ablate each of C4's 9 filtering heuristics individually (Is deduplication more important than removing code? Is the terminal punctuation filter more important than the bad-word filter?), measuring the downstream performance impact of removing each one; (2) train a lightweight text quality classifier on the outputs of the existing heuristic filters, then use it to filter raw Common Crawl data without hand-crafted rules, testing whether learned filtering can match or exceed rule-based filtering; (3) evaluate whether higher-quality pre-training data (e.g., C4 vs. WebText-like vs. Wikipedia) becomes more or less important as model scale increases, testing the hypothesis that larger models are more robust to noisy pre-training data. The finding that Unfiltered C4 underperforms C4 by 1.5–3 points across tasks despite being 8× larger establishes that data quality matters, but the field needs to know which specific quality dimensions drive the effect and whether quality requirements change with scale.

Cross-lingual and multilingual text-to-text pre-training at T5 scale. T5's translation results (Table 14) demonstrate that English-only pre-training is insufficient for competitive machine translation, with EnRo trailing the state-of-the-art by 10.4 BLEU. A direct extension would pre-train a T5 model on a multilingual version of C4 (filtering for, say, 10–100 languages using the same heuristic pipeline), use the same span-corruption objective, and fine-tune on both English understanding tasks and translation into multiple language pairs. The key question is whether multilingual pre-training (which provides cross-lingual signal through shared subwords and shared parameters) can close the translation gap without hurting English understanding performance. If multilingual pre-training matches English-only T5 on GLUE/SQuAD/SuperGLUE while dramatically improving translation BLEU, it would establish a single model that is genuinely useful across languages. If there is a tradeoff (multilingual pre-training hurts English performance due to reduced English token allocation), the field needs methods for language-adaptive training that preserve English quality while gaining cross-lingual transfer. The paper's vocabulary design (10:1 English-to-other-language mixture) is a minimal first step; a full multilingual study would tune the language mixing proportions, the vocabulary size, and the pre-training corpus composition.

Understanding the interaction between model scale and optimal design choices. The paper's coordinate ascent methodology (Sections 3.2–3.5) optimizes each design factor at the Base model scale, then applies the winning combination to models up to 11B parameters, assuming the optimal choices are scale-invariant. A stress-test follow-up would re-run the most critical ablation (e.g., the span-corruption objective with different mean span lengths, Table 7) at the Large or 3B scale to test whether the optimal span length shifts with model capacity. The hypothesis to test is: larger models, with greater generative capacity, might benefit from longer corrupted spans (e.g., mean span length 10 instead of 3) because they can learn more complex text reconstruction patterns, and the harder pre-training task might provide better regularization against overfitting. A negative result (optimal span length remains 3 at all scales) would validate the paper's coordinate ascent methodology; a positive result (the optimum shifts) would mean the paper's final T5 configuration is suboptimal, and that scaling studies must jointly sweep scale and design factors, dramatically increasing the cost of empirical NLP research. Either outcome is scientifically valuable, and this experiment is feasible at the Large (770M) scale where pre-training costs are manageable.

Distillation and compression of T5 for resource-constrained deployment. The paper explicitly calls for "research on methods that achieve stronger performance with cheaper models" (Section 4.2) and releases all model checkpoints, but performs no distillation experiments itself. A practical follow-up would use T5-11B as a teacher to distill task-specific models at the Small or Base scale, measuring how much of the 11B model's performance can be recovered by a 60M or 220M student. The baseline to beat is T5-Base trained from scratch (Table 14): GLUE 82.7, SQuAD 85.44. If distillation from T5-11B can push a Base-sized student to, say, GLUE 86+ and SQuAD 88+, it would make T5-level performance available to practitioners who cannot deploy 11B-parameter models. The paper's finding that fine-tune-only ensembling (Table 13, 4× ensembled fine-tune only) provides substantial gains from a single pre-trained model suggests that a single T5-11B teacher could generate diverse training data for distillation by fine-tuning multiple copies with different random seeds and aggregating their predictions. A negative result—distillation fails to transfer most of the scale advantage—would imply that the benefits of large models are fundamentally tied to their capacity and cannot be compressed, which would have significant implications for the deployability of large-scale NLP models.

Practical Applications and Downstream Use Cases

Unified NLP pipelines for multi-task production systems. A company deploying NLP models for multiple tasks—say, a customer support system that needs intent classification, sentiment analysis, question answering over documentation, and summarization of support tickets—currently needs to maintain separate models or complex multi-head architectures for each task. T5's text-to-text framework allows a single deployed model to handle all of these tasks by changing only the input prefix. Pre-training or fine-tuning a T5 model on the company's task suite (using the multi-task pre-training approach validated in Table 12) produces one model artifact that can be deployed once, reducing operational complexity (one container, one scaling policy, one monitoring dashboard) compared to maintaining N separate task-specific models. The paper's demonstration that multi-task pre-training followed by individual task fine-tuning matches pure unsupervised pre-training (Table 12: GLUE 83.11 vs. 83.28) provides evidence that this approach does not sacrifice task-specific performance. A team could start from a released T5 checkpoint, continue multi-task pre-training on their proprietary tasks mixed with C4 data, then optionally fine-tune separate copies for each task if production latency allows model-per-task deployment.

Data augmentation and synthetic data generation for low-resource tasks. The paper documents that T5 performance on low-resource tasks (CB, COPA, WSC) lags human performance substantially (Section 3.7: COPA 94.8 vs. human 100.0, WSC 93.8 vs. 100.0), even as T5 exceeds human baselines on high-resource reading comprehension tasks. This asymmetry suggests that data scarcity, not model capacity, is the bottleneck. A practitioner with a low-resource task (e.g., a specialized classification problem with 100–1,000 labeled examples) could use a pre-trained T5 model to generate additional training data by: (1) fine-tuning T5 on the available labeled examples; (2) using the fine-tuned model to generate new input-output pairs for the task; (3) filtering generated examples by confidence score or a separate quality classifier; (4) re-training on the augmented dataset. The paper's text-to-text format makes this straightforward—the model inherently knows the task format from the prefix, so synthetic examples preserve the input-output structure. The finding that pre-training benefits are largest for low-resource tasks (Table 1: SQuAD drops from 80.88 to 50.31 without pre-training, while EnFr barely changes) establishes that transfer from pre-training is the primary mechanism by which T5 learns low-resource tasks, and data augmentation is a way to amplify that transfer.

Document processing pipelines with integrated summarization, classification, and extraction. An organization processing large volumes of documents—legal contracts, scientific papers, financial reports—could deploy a single T5 model to perform multiple document understanding operations in sequence: given a document, generate a summary (prefix: "summarize:"), classify the document type (prefix: "classify:"), extract key entities or clauses (prefix: "extract:"), and answer specific queries about the content (prefix: "question:" context:). The paper's demonstration that T5 performs competitively on CNN/Daily Mail summarization (ROUGE-2 21.55, exceeding prior state-of-the-art 20.30), GLUE classification tasks (average 90.3), and SQuAD question answering (EM 91.26) within a single model architecture establishes the feasibility of this integrated approach. The practical benefit is reduced model management overhead and the ability to share representations across operations—the model's understanding of a document built during summarization could improve its subsequent classification and extraction performance, though this cross-task transfer benefit is not directly measured in the paper and would need validation in a production setting.

Pre-training data curation for specialized domains. The paper's quantification of the in-domain pre-training effect (Section 3.4.1) provides a concrete recipe for practitioners building models for specialized domains. For a biomedical NLP application: (1) collect a large corpus of unlabeled biomedical text (PubMed abstracts, clinical notes, biomedical literature); (2) filter it using C4-style heuristics adapted for the domain (e.g., removing citation markers, filtering for complete sentences, language detection); (3) mix this biomedical corpus with C4 at some ratio (to preserve general language capabilities while gaining domain knowledge); (4) continue pre-training T5 from the released checkpoint on this mixture using the span-corruption objective; (5) fine-tune on the target biomedical tasks. The paper's MultiRC result (Wikipedia + TBC improving EM from 25.78 to 50.93, a 97% relative improvement) provides the effect size benchmark—domain-matched pre-training can roughly double performance on domain-specific reading comprehension. A biomedical team could expect qualitatively similar gains, though the exact magnitude would depend on the domain specificity of their tasks and the volume of in-domain unlabeled data available.

When to Prefer This Method

The paper does not articulate an explicit decision rule preferring T5 over named alternatives, because its contribution is not a new method competing against specific previous methods, but rather a framework for systematic comparison and a recommendation to use the simplest, most computationally efficient variant of a denoising objective at large scale. However, the paper's results do imply several practical criteria that can be extracted as a preference structure, grounded in specific findings:

Prefer T5-style text-to-text pre-training when:

You need a single model to handle multiple task types (classification AND generation). The text-to-text format allows one deployed model to perform summarization, QA, and classification by changing only the task prefix. The paper demonstrates this across 24 tasks without task-specific architectures (Table 14). If your application involves only classification, the paper provides no evidence that the text-to-text format outperforms a BERT-style classifier at matched scale, and the extra decoder parameters (doubling model size relative to an encoder-only model) may be wasteful.
You have access to large-scale pre-training data (hundreds of GB of text or more) and the compute to pre-train on it. The paper's scaling results (Section 3.6) and the degradation observed when repeating small datasets (Table 9, 1024× repetition drops GLUE by ~4 points) imply that T5's approach—pre-training without repeating data—requires a corpus large enough to avoid excessive repetition at the target pre-training volume. For 1T tokens, C4's 750 GB is sufficient; a 20 GB domain-specific corpus repeated 50× would likely degrade performance based on the trends in Table 9.
You are operating in English (or a language with abundant unlabeled text). The paper's translation results show that English-only pre-training is insufficient for non-English generation (EnRo 28.1 vs. state-of-the-art 38.5 BLEU), and no experiments test non-English understanding tasks. Extending T5 to other languages requires recreating the C4 pipeline in the target language, retraining the vocabulary, and re-running pre-training—the paper provides the recipe but not the pre-trained model.
Computational efficiency during pre-training is a priority, and you are willing to sacrifice marginal potential gains from more complex objectives. Section 3.3.5 recommends choosing the denoising objective that minimizes target sequence length (and thus training time), because all denoising variants perform similarly. The span-corruption objective with mean span length 3 is specifically recommended because it produces shorter targets than i.i.d. corruption while maintaining performance.

Prefer alternative approaches (task-specific architectures, multilingual pre-training, or non-T5 objectives) when:

You only need classification or regression, and model size or inference latency is severely constrained. An encoder-only model (BERT-style) with a classification head has fewer parameters than an encoder-decoder text-to-text model at the same capacity because it lacks a decoder. If your tasks are purely classification and you face strict deployment constraints (on-device inference, low-memory environments), the paper provides no evidence that the text-to-text decoder is worth its parameter cost. The adapter layer experiments (Table 10) show that adapters can match full fine-tuning for classification at large enough inner dimensionality, but this doesn't reduce the base model size.
Your tasks are primarily non-English, especially low-resource languages. T5's English-only pre-training and vocabulary design provide no cross-lingual transfer. The translation results (Table 14) demonstrate that scaling English pre-training does not close the gap to state-of-the-art systems that use backtranslation and cross-lingual pre-training. A practitioner working primarily in a non-English language should use a multilingual pre-trained model or pre-train a monolingual T5 from scratch in that language following the C4 recipe.
You have a small, fixed pre-training corpus and cannot obtain more data. Table 9 shows that repeating a small dataset more than ~100× causes significant performance degradation. If your total pre-training data is limited (e.g., a few GB), and you plan to pre-train on billions of tokens, T5's approach will suffer from memorization. Alternative approaches like data augmentation during pre-training, more aggressive regularization, or using a different pre-training paradigm (e.g., the ELECTRA-style discriminative objective, which was shown after T5 to be more data-efficient) might be more appropriate, though the paper does not evaluate these.
You need state-of-the-art machine translation and can incorporate cross-lingual supervision. The paper acknowledges that backtranslation and cross-lingual pre-training provide gains that English-only T5 cannot match (Section 3.7). For translation, XLM-style multilingual pre-training (Lample and Conneau, 2019) or dedicated sequence-to-sequence translation models with backtranslation (Edunov et al., 2018) are preferred based on the gap between T5's translation BLEU scores and the then-state-of-the-art.