UL2: Unifying Language Learning Paradigms

🎯 Pitch

A single pre-training recipe—mixing span corruption, causal language modeling, and extreme long-span denoising with explicit mode tokens—enables one model to beat both T5 and GPT-style models across the board, including a 20B variant that outperforms 175B GPT-3 on zero-shot SuperGLUE.

1. Executive Summary

This paper introduces UL2, a unified framework for pre-training language models that combines diverse self-supervision objectives into a single Mixture-of-Denoisers (MoD) — a blended objective comprising R-denoising (standard span corruption, as in T5), S-denoising (sequential prefix language modeling, akin to causal LMs), and X-denoising (extreme denoising with long spans and high corruption rates) — along with mode switching, where dedicated sentinel tokens ([R], [S], [X]) bind downstream behavior to specific pre-training paradigms. Through systematic ablations on SuperGLUE and GEM benchmark tasks using PaLM-style decoder-only and encoder-decoder architectures, UL2 outperforms T5-like span corruption models by an average normalized gain of +43.6% and GPT-like causal language models by +76.1% across 9 diverse setups, while the 20B-parameter scaled version achieves state-of-the-art results on approximately 50 supervised NLP tasks and demonstrates emergent chain-of-thought reasoning at a moderate scale. The paper establishes that a single pre-trained model can perform universally well across both fine-tuned discriminative and prompt-based generative tasks only when the pre-training objective explicitly blends complementary denoising paradigms rather than committing to any single one.

2. Context and Motivation

The Core Problem: No Single Pre-Training Paradigm Works Universally Well

At the time this paper was written, the NLP community faced a persistent and deeply frustrating dilemma: the choice of which pre-trained language model to use seemed to depend entirely on what downstream task you cared about. If you were building a classifier for sentence-level entailment or sentiment analysis, the consensus pointed to encoder-decoder models like T5 (Raffel et al., 2019) trained with span corruption — they consistently delivered the strongest supervised fine-tuning results on NLU benchmarks like SuperGLUE. But if you were building an application that required open-ended text generation or in-context few-shot learning (where the model must infer the task from a handful of examples in the prompt), the field had largely converged on decoder-only causal language models like GPT-3 (Brown et al., 2020). These two families of models — T5-like span-corruption models and GPT-like causal LMs — occupied different regions of the performance frontier (as visualized in Figure 1 of the paper), with each excelling where the other struggled.

This dependence on task type created a fragmented research landscape. Organizations building practical NLP systems found themselves maintaining multiple pre-trained models, each optimized for a different category of task — a strategy the authors describe as "diversifying resources across N models" (Section 1). For resource-constrained settings such as on-device deployment, where only one or two models can feasibly be served, this fragmentation was particularly costly. The situation raised a pointed question that the paper sets out to answer (Section 1):

"Why should the choice of the pre-trained LM depend on the downstream task? And how can we pre-train models that work universally well across many tasks?"

The Architecture-Objective Confusion

A key intellectual contribution of the paper's motivation is its diagnosis that the field was systematically conflating architectural choices with pre-training objective choices. The dominant narrative at the time roughly equated "decoder-only architecture = causal LM = good at generation" and "encoder-decoder architecture = span corruption = good at understanding." But these are two independent axes. As the paper notes (Section 3.2):

"We argue that the choice between both architectures (encoder-decoder vs decoder-only) is more of an efficiency trade-off and that architecture choice should not be conflated with the pretraining objective."

The authors point out that T5's span corruption can be trained perfectly well with a decoder-only architecture, and prefix language modeling can be trained with an encoder-decoder. By disentangling these two dimensions, the paper aims to isolate the effect of the pre-training objective — the self-supervision signal — as the primary driver of downstream task performance. This reframing is important because it shifts the focus from architecture engineering (which the paper treats as an efficiency consideration) to objective design (which the paper argues is the key missing piece for universality).

The Inadequacy of Existing Single-Objective Approaches

The paper's motivation draws heavily on well-known limitations of the dominant pre-training objectives when applied in isolation:

Span corruption (T5-style) excels at NLU fine-tuning but struggles with generation and in-context learning. The standard T5 denoising objective blanks out short contiguous spans (mean length 3 tokens, 15% corruption rate) and trains the model to recover them — a relatively easy task that the authors characterize as "more akin to fact completion" (Section 3.1.1). While this produces strong supervised fine-tuning results on discriminative tasks like SuperGLUE (where T5-11B achieves 89.9 average score, as shown in Table 9), it leaves the model poorly equipped for open-ended generation and few-shot prompting. The authors note in Section 2.1 that T5-style encoder-decoder models "have shown limited performance on open-text generation and prompt-based inference (i.e., in-context learning), which motivates the use of decoder-only models." This is quantitatively stark: in their own experiments, a T5-style span corruption encoder-decoder achieves a mere 1.43 ROUGE-L on 1-shot XSUM summarization (Table 2), compared to 11.51 for their proposed UL2. The model simply hasn't been trained to handle the long-range coherence demands of open-ended generation.

Causal language modeling (GPT-style) excels at generation but underperforms on supervised discriminative tasks. Left-to-right autoregressive LMs are the standard for decoder-only models, training on the full sequence and learning to predict the next token at each position. This objective exposes the model to long-range dependencies and open-ended generation, which is why GPT-3 demonstrates impressive few-shot capabilities. However, when fine-tuned on discriminative benchmarks, causal LMs systematically underperform span corruption models. In the paper's own ablations (Table 2), a decoder-only causal LM achieves 62.24 on SuperGLUE versus 72.00 for an encoder-decoder span corruption model — a 14% relative degradation. The causal constraint prevents the model from leveraging bidirectional context during pretraining, which turns out to be important for the kinds of local syntactic and factual reasoning that many NLU tasks demand.

Prefix LM offers a partial compromise but still makes trade-offs. The prefix language modeling objective allows the model to attend bidirectionally over a prefix (or "memory") while generating autoregressively over the target. This gives the model more modeling power than a strict causal LM while retaining autoregressive generation capability. The paper's experiments show that PrefixLM consistently outperforms causal LM across both supervised and few-shot setups (Table 3: PrefixLM decoder outperforms causal LM by +21.4% overall relative gain). However, it still falls short of both span corruption models on pure NLU and span corruption's supervised stronghold — the PrefixLM encoder-decoder achieves 69.30 on SuperGLUE versus 72.00 for span corruption (Table 2). The gap on generation tasks is smaller but still present.

The key insight: no single objective dominates because each one teaches the model to solve a different type of problem. The paper formalizes this in Section 3.1.1 by noting that all these objectives can be viewed through a unified denoising lens where they differ only in what kind of context the model conditions on — and that spanning from short-span corruption to full-sequence language modeling is a spectrum of difficulty and skill that no single point on the spectrum captures completely.

Where Prior Unified Approaches Fall Short

The paper does not propose unification in a vacuum. It acknowledges several prior attempts to combine multiple pre-training objectives, but identifies specific shortcomings that motivate the UL2 design:

UniLM (Dong et al., 2019) trains on multiple objectives but uses a cloze-style (BERT-like) formulation with explicit mask tokens and position-wise loss computation. The UL2 authors note (Section 2.3) that this approach is fundamentally less general than the input-to-target formulation they adopt: it requires task-specific classification heads for downstream use (since cloze-style prediction does not naturally produce autoregressive outputs), which the authors consider "very cumbersome" and contrary to the goal of a universal model (Section 2.1). Their own UniLM implementation (reformulated in the input-to-target style for fair comparison) shows that while the objective helps (UniLM encoder-decoder achieves +19.8% overall gain over T5 in Table 3), it lacks the explicit decomposition into R, S, and X denoisers that UL2 introduces, and it does not incorporate the notion of mode switching via dedicated paradigm tokens.

SCLM (Span Corruption + Language Model) is the most straightforward hybrid baseline — simply mix the two objectives equally. The paper's results (Table 3) show that this indeed helps: SCLM encoder-decoder achieves +28.3% overall gain over T5, outperforming most other baselines. But the authors argue (implicitly, through their MoD design) that a simple 50-50 mix is suboptimal because it doesn't explore the full space of denoising configurations — particularly high-span-length, high-corruption-rate extreme denoising (X-denoising) that pushes the model toward language modeling-like behavior while retaining the multi-span structure of span corruption. The SCLM baseline serves as evidence that mixing objectives is directionally correct, but that the specific mixture composition and the inclusion of intermediate regimes matter.

The T5 paper itself explored higher corruption rates and found them unhelpful. As the authors note (Section 3.1.2), "the original T5 paper explored an option with 50% corruption rate (X-denoising) and found that to not work well." This is a crucial motivational point: taken in isolation, increased corruption rates don't improve performance — in fact, the original T5 experiments suggested they were harmful. The UL2 authors' key conjecture is that these extreme denoising regimes are beneficial only when mixed with easier regimes (short-span, low-corruption R-denoising), not when used as the sole pre-training objective. Their own ablation in Table 6 confirms this: a variant that uses only long spans (Var-H, µ = 8, 64) performs poorly, echoing T5's negative finding. But variants that combine long spans with short spans (Var-D, E, F) achieve substantially better one-shot performance, suggesting a complementarity between denoisers that single-objective studies failed to discover.

The Conceptual Gap: No Framework for Reasoning About Objective Complementarity

Beyond the specific limitations of individual prior works, the paper identifies a higher-level gap: the absence of a unified theoretical perspective from which different pre-training objectives could be understood as points in a continuous design space rather than as discrete alternatives. Section 3.1.1 introduces precisely this perspective by showing that the span corruption function, parameterized by span length (µ) and corruption rate (r), can approximately recover both causal language modeling (by setting µ = L, the full sequence length) and prefix language modeling (by constraining the single corrupted span to end at the sequence boundary). This parameterization reveals that what the field has been treating as categorically different objectives — "span corruption" vs. "language modeling" — are actually endpoints on a spectrum, and that the space between them (corresponding to X-denoising at µ = 64, r = 0.5) might contain useful training signals that neither endpoint captures alone.

This unified perspective allows the authors to design MoD as a principled interpolation across the spectrum rather than an ad-hoc mixture. It also explains why simple mixing (SCLM) helps but leaves room for improvement: SCLM only mixes the two endpoints, missing the intermediate regimes that push the model to simultaneously handle long-range coherence (like an LM), multi-span corruption (like T5), and bidirectional context (like PrefixLM) in varying proportions.

How This Paper Positions Itself

The paper positions UL2 as an intellectual synthesis rather than a completely novel objective. It does not claim to invent span corruption, language modeling, or prefix LM. Instead, it claims three interconnected contributions:

The Mixture-of-Denoisers framework that blends R, S, and X denoisers in specific, empirically validated proportions (Section 3.1.2, Table 1), with the key novelty being the inclusion of X-denoising (long spans, high corruption rates) as a bridge between standard span corruption and language modeling.
Mode switching via paradigm tokens ([R], [S], [X]) that allows the model to distinguish which denoising paradigm it should operate under during both pre-training and downstream fine-tuning (Section 3.1.3). This is conceptually distinct from prior multi-objective training because it makes the task identity explicit in the input, enabling the model to learn distinct behaviors for distinct paradigms rather than averaging across them.
The architecture-agnostic claim: that the choice of encoder-decoder vs. decoder-only is primarily an efficiency consideration, not a determinant of task suitability, and that the pre-training objective is the more important design variable (Section 3.2). The paper supports this by showing UL2 works well in both architectures (Table 2), though encoder-decoder generally performs better on NLU at equal compute.

The paper explicitly frames itself as a response to the fragmentation it diagnoses in Section 1: rather than developing yet another model that's good at one thing (e.g., scaling GPT further for in-context learning, or scaling T5 further for supervised NLU), the goal is to produce "a single pretrained model that can perform well on many types of tasks." This is what distinguishes UL2 from the scaling-focused work happening contemporaneously (Chowdhery et al., 2022; Du et al., 2021) — it's about objective design as an orthogonal axis of improvement to pure scale.

The paper also positions itself as a practical intervention. By demonstrating that UL2 at 20B parameters achieves SOTA on ~50 tasks (Table 8), matches or exceeds T5-11B on SuperGLUE (Table 9), and enables chain-of-thought reasoning at a scale where prior work required 100B+ models (Section 5.2.7), the authors argue that objective design can partially substitute for scale — a claim with direct implications for resource allocation in both research and deployment. This connects to the broader conversation about whether scaling laws (Kaplan et al., 2020) can be shifted by better pre-training recipes, though the paper does not formalize this as a scaling law claim.

Finally, the paper is transparent about its scope limits (Section 1): it does not claim to have solved universality entirely. The focus is on demonstrating that a single model can be consistently strong across a very diverse task suite, not necessarily the absolute best on every individual task. This is a more modest but arguably more practical goal than "one model to rule them all" — it's about shifting the Pareto frontier such that a single checkpoint becomes the sensible default choice instead of task-specific model selection.

3. Technical Approach

3.1 Reader Orientation

UL2 is a pre-training recipe — a specific way of training a language model from scratch on unlabeled text so that it becomes universally useful across many different types of downstream tasks, rather than being specialized to one category. The system solves the problem of "which pre-trained model should I use for my task?" by producing a single model that performs well at everything: supervised fine-tuning on classification tasks, few-shot in-context learning, open-ended text generation, and even multi-step reasoning — tasks that previously required separate models trained with different objectives and architectures.

3.2 Big-Picture Architecture (Diagram in Words)

At its core, UL2 has three major components:

A unified denoising perspective — a mathematical formulation showing that all major pre-training objectives (span corruption, language modeling, prefix language modeling) are special cases of a single parameterized SpanCorrupt function characterized by mean span length ( $\mu$ ), corruption rate ( $r$ ), and number of corrupted spans ( $n$ ). This insight allows the designers to reason about objectives as points on a continuous spectrum rather than as discrete alternatives.
Mixture-of-Denoisers (MoD) — the actual pre-training objective, which blends seven specific denoiser configurations spanning three paradigms: R-denoising (regular — T5-style short spans at 15% corruption), S-denoising (sequential — prefix language modeling where targets always reach the end of the sequence), and X-denoising (extreme — long spans and/or high corruption rates that push toward language-modeling-like behavior). Each denoiser is sampled uniformly during pre-training.
Mode switching via paradigm tokens — special sentinel tokens ([R], [S], [X]) prepended to the input that tell the model which denoising paradigm it should operate under, both during pre-training (where the token matches the sampled denoiser) and during downstream fine-tuning (where the practitioner selects the token best suited to the task).

Information flows as follows: raw text from the C4 corpus enters → a denoiser is uniformly sampled from the seven-configuration mixture → the text is corrupted according to that denoiser's parameters (producing inputs with sentinel tokens and target spans) → the paradigm token corresponding to the sampled denoiser is prepended → the model processes the corrupted input and autoregressively generates the missing spans → the loss is computed only on the target tokens. At fine-tuning time, the practitioner selects a paradigm token based on whether the task is more "understanding-like" (use [R]), "generation-like" (use [S]), or in between (use [X]), and fine-tunes normally.

3.3 Roadmap for the Deep Dive

First, the unified perspective for pre-training tasks (the SpanCorrupt formalism), because it is the intellectual foundation that shows why mixing objectives works — by demonstrating that causal LM and span corruption are endpoints on a continuous spectrum, not separate things.
Second, the three denoiser paradigms (R, S, X) and their concrete configurations, because these are the actual building blocks of the mixture and understanding their distinct characteristics explains why each contributes something unique.
Third, the Mixture-of-Denoisers composition — the specific seven-configuration blend and the design choices behind it (uniform sampling, the inclusion of X-denoising despite T5's negative results, the role of each component).
Fourth, mode switching — how paradigm tokens are used during pre-training and fine-tuning, and why this explicit task-identity signal matters beyond simply mixing objectives.
Fifth, the architecture-agnostic design philosophy — why UL2 is implemented for both decoder-only and encoder-decoder architectures, what the trade-offs are, and how this reinforces the claim that the objective matters more than the backbone.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an objective design paper whose core idea is that a universally effective pre-trained model requires exposure to a diverse mixture of denoising tasks during self-supervised training, combined with an explicit mechanism (paradigm tokens) that allows the model to associate downstream behavior with specific pre-training paradigms.

The Unified Perspective: All Pre-Training Objectives as Parameterized Span Corruption

The intellectual foundation of UL2 is the observation that the dominant pre-training objectives in NLP — causal language modeling, prefix language modeling, and span corruption — can all be expressed as instances of a single function that corrupts an input sequence and then asks the model to recover the missing parts. The authors define this function conceptually rather than with a formal equation, but its parameters are stated precisely.

The SpanCorrupt function takes an input text and produces two outputs — corrupted inputs and target spans — according to three parameters:

$\mu$ (mean span length): the average length (in subword tokens) of each contiguous span that gets removed from the input and turned into a target. The actual span lengths are drawn from a distribution (normal for R- and X-denoising, uniform for S-denoising) centered at $\mu$ .
$r$ (corruption rate): the fraction of the total input tokens that get corrupted — i.e., replaced by sentinel tokens and moved to the targets. For example, $r = 0.15$ means roughly 15% of tokens are masked.
$n$ (number of corrupted spans): how many separate contiguous spans are removed. This is typically derived from the input length $L$ , the mean span length $\mu$ , and the corruption rate $r$ — you need enough spans at the right length to reach the target corruption percentage. For S-denoising, $n$ is always fixed at 1.

After corruption, the original tokens from the removed spans become the target sequence (separated by sentinel tokens that indicate where each span began), and the remaining uncorrupted tokens (with sentinel tokens inserted where spans were removed) become the input. The model's task is to process the corrupted input and autoregressively generate the target spans.

How this recovers different objectives:

Causal language modeling: set $\mu = L$ (the full sequence length), $r = 1.0$ (100% corruption), and $n = 1$ (a single span). The entire sequence is removed and becomes the target, and the model conditions only on a sentinel token. This is approximately (but not exactly — the sentinel token provides a trivial signal) equivalent to generating the full text from scratch. The authors note that this is only "roughly approximate since the model still conditions on a sentinel token," meaning it is not precisely identical to standard LM pre-training where every token is predicted from its prefix.
Prefix language modeling: set $\mu = L - P$ (where $P$ is the prefix length), $r = 1.0 - P/L$ , and $n = 1$ , with the additional constraint that the single corrupted span must extend to the end of the sequence (no uncorrupted tokens after it). This creates a setup where the model sees a bidirectional context (the prefix) and generates the suffix autoregressively — identical to the PrefixLM objective. The prefix length $P$ is sampled during training so the model learns to handle varying amounts of context.
Standard span corruption (T5): set $\mu \approx 3$ (mean span of 3 tokens), $r = 0.15$ (15% corruption), and $n$ determined by $L/\mu$ to achieve the target rate. This is the well-known T5 denoising objective where multiple short spans are corrupted and recovered.

Why this unified perspective matters: it reveals that what the field treated as categorically different objectives — "denoising" vs. "language modeling" — are actually points on a continuous two-dimensional manifold defined by $(\mu, r)$ . A model trained only at one point (say, $\mu = 3$ , $r = 0.15$ for T5-style span corruption) learns skills appropriate to that point but may fail to generalize to tasks that require capabilities more naturally acquired at other points (e.g., long-range coherence from $\mu \rightarrow L$ ). The key design insight is that by training across the manifold — rather than at a single point — the model develops a more complete set of capabilities.

This perspective also explains the complementarity between denoisers that the paper empirically demonstrates. Moving $\mu$ toward longer spans teaches the model to handle long-range dependencies (the hallmark of language modeling), while keeping $r$ below 1.0 and $n > 1$ preserves the multi-span structure that teaches the model to attend to multiple separate context regions (useful for tasks like reading comprehension). No single point on the manifold captures both skills simultaneously.

The Three Denoiser Paradigms: R, S, and X

From this unified perspective, the authors define three prototypical denoising paradigms that span the important regions of the $(\mu, r)$ space. Each is motivated by the specific capability it teaches and the type of downstream task it prepares the model for.

R-Denoiser (Regular Denoising): This is the standard T5 span corruption objective. The configuration uses $\mu = 3$ and $r = 0.15$ , with the number of spans $n$ derived from the input length to achieve the 15% corruption rate. (The paper also includes a variant with $\mu = 8$ , $r = 0.15$ in the mixture — see Table 1, which shows $\mu = 3$ and $\mu = 8$ are both included under R-denoising.) The spans are sampled from a normal distribution with the given mean.

The corrupted input contains sentinel tokens where text was removed, and the targets are the removed spans in the order they appear, each prefixed by the corresponding sentinel token. Because spans are short (2–5 tokens for $\mu = 3$ , roughly 6–10 for $\mu = 8$ ) and corruption is moderate, the model has substantial context available to recover the missing tokens. The authors characterize this as being "more akin to fact completion" — the model learns to fill in missing words using local syntactic and semantic cues.

Why R-denoising teaches useful skills: the multi-span structure forces the model to attend to multiple separate regions of the input simultaneously, since recovering the third corrupted span may require information from the first uncorrupted region and the fifth uncorrupted region that are far apart in the text. This bidirectional, multi-hop attention is precisely what discriminative NLU tasks require — a sentiment classification decision or an entailment judgment often depends on information distributed across the input. The inclusion of $\mu = 8$ alongside $\mu = 3$ provides slightly longer-span signals that begin to stretch into the "sentence completion" regime without fully leaving the multi-span corruption paradigm.

S-Denoiser (Sequential Denoising): This is the prefix language modeling objective. The configuration is $\mu = L/4$ , $r = 0.25$ , and $n = 1$ (a single span). The critical additional constraint is that the single corrupted span must extend to the end of the original sequence — there are no uncorrupted tokens after it. This means the input (prefix) contains the first 75% of tokens (on average), which the model can attend to bidirectionally, and the target (suffix) contains the remaining 25%, which must be generated autoregressively without peeking at future tokens.

The span length $\mu$ is sampled from a uniform distribution rather than a normal distribution, and the constraint that the span reaches the end of the sequence means that for a sequence of length $L$ , the prefix length $P$ is uniformly distributed between 0 and $L - 1$ (with the constraint that the target is approximately $L/4$ on average). When the memory is very short (small $P$ ) or zero, this approximates standard causal language modeling (no bidirectional context, just generate).

Why S-denoising teaches useful skills: the sequential order constraint means the model cannot use future information when generating targets — it must learn to produce coherent, flowing text where each token depends only on what came before it in both the prefix and the already-generated suffix tokens. This is exactly the skill required for open-ended text generation, summarization, and dialogue, where the model produces novel text autoregressively. The bidirectional prefix encoding gives the model more power than a strict causal LM (which can't look at the full prefix bidirectionally), while the autoregressive target generation teaches generation fluency.

The authors note that "since LM is a special case of Prefix-LM, we did not find it necessary to include a causal LM task into the mixture" (Section 3.1.2). When the prefix length is zero, S-denoising reduces to exactly the same computation as standard causal language modeling (with the minor difference of a sentinel token marking the boundary).

X-Denoiser (Extreme Denoising): This is the novel contribution and the crucial bridge between R-denoising and S-denoising. X-denoising pushes one or both of the corruption parameters to extremes — long spans ( $\mu \geq 12$ , up to $\mu = 64$ ) and/or high corruption rates ( $r \geq 0.3$ , up to $r = 0.5$ ). The specific configurations in the mixture are shown in Table 1: $\mu = 3$ at $r = 0.5$ , $\mu = 8$ at $r = 0.5$ , $\mu = 64$ at $r = 0.15$ , and $\mu = 64$ at $r = 0.5$ .

What makes X-denoising "extreme": the model must recover a large portion of the input from very limited context. For example, with $\mu = 64$ and $r = 0.5$ , half of all tokens are corrupted, and the corrupted spans average 64 tokens each. This means the model sees only about 50% of the original text, and the missing portions are very long — requiring the model to generate coherent multi-sentence passages. The authors describe this as a situation "where a model needs to generate long target from a memory with relatively limited information."

Why X-denoising is the key innovation: it serves as an interpolation between R-denoising (short spans, low corruption — essentially "fill in the blanks") and S-denoising/language modeling (full-sequence generation from minimal context). At $\mu = 64$ , $r = 0.5$ , the model is effectively doing language modeling on long stretches while still retaining the multi-span corruption structure (multiple spans can be corrupted). The authors explicitly connect this to sample efficiency: "X-denoisers are also connected to improve sample efficiency since more tokens are learned to be predicted in each sample, in similar spirit to LMs" — since 50% of tokens are targets instead of 15%, each training example provides more learning signal per token of input processed.

Crucially, the paper notes that "the original T5 paper explored an option with 50% corruption rate (X-denoising) and found that to not work well" when used as a standalone objective. The UL2 authors' hypothesis (validated by their ablations) is that X-denoising is only effective when mixed with easier denoisers — it provides a valuable training signal that R-denoising alone cannot, but it is too difficult to serve as the sole objective.

The Mixture-of-Denoisers (MoD) Composition

The MoD objective blends all three paradigms by sampling uniformly from the seven specific denoiser configurations listed in Table 1. The exact configurations are:

Denoiser Category	Configurations
R	$\mu = 3, r = 0.15$ ∪ $\mu = 8, r = 0.15$
S	$\mu = L/4, r = 0.25, n = 1$ (with end-of-sequence constraint)
X	$\mu = 3, r = 0.5$ ∪ $\mu = 8, r = 0.5$ ∪ $\mu = 64, r = 0.15$ ∪ $\mu = 64, r = 0.5$

Each configuration is sampled with equal probability during pre-training, meaning the model sees S-denoising on approximately 1/7 ≈ 14% of training examples, R-denoising on 2/7 ≈ 29%, and X-denoising on 4/7 ≈ 57%. The X-denoising category is thus the dominant component by training volume. Within each X-denoising configuration, span lengths are sampled from a normal distribution with the given mean $\mu$ (for the multi-span variants with $\mu = 3$ and $\mu = 8$ ; the paper doesn't explicitly describe the distribution for $\mu = 64$ , but consistent treatment suggests normal distribution for consistency).

Design choices and their justifications:

Uniform sampling across the seven configurations (as stated in Section 3.1.2: "All tasks have an approximate equal participation in the mixture") — this ensures no single paradigm dominates, and the model must learn to handle all of them. The authors also explore increasing S-denoising to 50% of the mixture in their ablations (Section 4.5) and find that it "generally hurts performance," concluding that "small amounts of S-denoisers (≈20%) are preferred." This is a non-obvious finding: one might intuitively think that since generation is the harder skill, it deserves more training volume, but empirically the balanced mixture works better.
Two R-denoising configurations (µ=3 and µ=8) rather than just the standard µ=3 — this provides a modest stretch toward longer-span corruption without jumping all the way to the X-denoising regime. It smooths the transition across the $\mu$ spectrum.
Four X-denoising configurations spanning the product of {short, long} spans × {moderate, high} corruption — the $\mu = 3$ , $r = 0.5$ configuration is interesting because it uses the same short spans as R-denoising but at much higher corruption (50% vs 15%). This teaches the model to handle dense masking over short distances. The $\mu = 64$ , $r = 0.15$ configuration is interesting because it uses the standard corruption rate of T5 but with very long spans — the model must generate paragraph-length passages but still has 85% of the context available. The $\mu = 64$ , $r = 0.5$ configuration is the most extreme: half the text is gone in very long chunks, pushing closest to pure language modeling.
Only one S-denoising configuration and no explicit causal LM configuration — the authors argue that S-denoising with variable prefix length already covers the LM regime (when prefix is very short), and that the prefix-LM formulation is strictly better than causal LM because it provides bidirectional encoding of the prefix. Their empirical results in the ablations (Table 2) strongly support this: PrefixLM (PLM) consistently outperforms causal LM (CLM) on every single task in both supervised and one-shot setups. For example, PLM decoder achieves 62.44 SuperGLUE vs 62.24 for CLM, and 42.54 one-shot SuperGLUE vs 39.22 for CLM. The gains are even larger on generation tasks: PLM achieves 6.40 one-shot TOTTO vs 0.20 for CLM.

Why the mixture works (the complementarity argument): each denoiser is "difficult in different ways" (Section 3.1) and teaches a different capability. R-denoising teaches bidirectional multi-hop reasoning over multiple short contexts — useful for NLU. S-denoising teaches sequential coherence and autoregressive generation — useful for open-ended generation and in-context learning. X-denoising teaches long-range generation from limited context — useful for tasks that require both understanding (reasoning about the limited available context) and generation (producing the missing text). A model trained only on R-denoising has no practice generating coherent multi-sentence text. A model trained only on S-denoising has no practice reasoning about multiple separate context regions simultaneously. The mixture ensures the model is challenged in all these ways during pre-training.

The authors provide evidence for this complementarity in their ablation study (Table 6). They compare variants with and without long spans (Var-D includes $\mu = 32$ spans; Var-C uses only up to $\mu = 12$ ), with and without S-denoising (Var-K and Var-L have 0% S-denoising), and with varying S-denoising proportions (Var-I and Var-J at 50% S-denoising). Key findings:

Removing S-denoising entirely (Var-K) improves supervised SuperGLUE (73.7 vs 73.1 for the best configuration) but crushes one-shot XSUM performance (2.6 vs 10.4 Rouge-L for the best configuration) — a >75% relative degradation. This is direct evidence that S-denoising teaches a capability (in-context generation) that R- and X-denoising alone cannot provide.
Removing long spans (Var-C with max $\mu = 12$ vs Var-D with $\mu$ up to 32) significantly degrades one-shot performance on both SuperGLUE (38.6 → 42.7) and XSUM (4.1 → 10.6) without helping supervised performance.
Using only long spans without short spans (Var-H with $\mu = 8$ and $\mu = 64$ only) performs poorly on supervised SuperGLUE (72.5 vs 73.1) and one-shot XSUM (10.9 vs 10.4) — showing that extreme denoising must be complemented by regular denoising.
Increasing S-denoising to 50% (Var-I and Var-J) hurts supervised SuperGLUE (71.2–71.3 vs 73.1) while providing at most marginal improvements in one-shot generation — the authors conclude "S-denoisers are necessary but only small amounts of S-denoisers (≈20%) are preferred."

Mode Switching: Paradigm Tokens as Task-Identity Signals

The second key innovation in UL2 is mode switching — the use of dedicated sentinel tokens ([R], [S], [X]) to explicitly signal which denoising paradigm is active. During pre-training, the appropriate paradigm token is prepended to the input before the model processes it: if an R-denoising configuration was sampled, the input starts with [R]; if an S-denoising configuration, [S]; if an X-denoising configuration, [X]. The model thus learns to associate each token with a different mode of operation — different patterns of attending to context, different types of output expected.

Why this matters beyond simply mixing objectives: without paradigm tokens, the model would have to infer from the input pattern alone whether it's doing span corruption (many short gaps), prefix LM (one gap at the end), or extreme denoising (some combination). This inference is possible in principle — the input pattern carries information — but in practice it may lead the model to average across modes, producing behavior that is suboptimal for any single mode. Paradigm tokens make the task identity explicit and discriminable, enabling the model to develop mode-specific circuits (different attention patterns, different internal representations) that can be cleanly activated by the token.

The authors describe this as "paradigm-shifting via mode switching" (Section 3.1.3): during pre-training, the paradigm token tells the model which denoising gear to operate in. During fine-tuning, the practitioner selects the paradigm token that best matches the downstream task:

[R] (Regular denoising mode) for tasks that are primarily about understanding and factual reasoning — the model should operate in its "fill-in-the-blank, attend-bidirectionally-to-everything" mode.
[S] (Sequential denoising mode) for tasks that are primarily about generation — the model should operate in its "generate coherent text autoregressively" mode.
[X] (Extreme denoising mode) for tasks that require both heavy reasoning and substantial generation — a hybrid mode that pushes the model to extract maximal information from limited context.

Empirical validation of mode switching effectiveness: Table 5 shows the effect of different paradigm prompts on one-shot evaluation using an encoder-decoder model pre-trained with UL2 on 7B tokens. The results demonstrate that paradigm token choice has a dramatic, task-dependent effect:

One-shot XSUM (summarization): using no prompt achieves Rouge-L of 10.8. Using [R] achieves 11.1. Using [S] achieves 10.0. Using [X] achieves 7.6. The best prompt ([R]) provides a 48% relative gain over the worst ([X]) on Rouge-1 (1.5 vs 0.9). The takeaway: for summarization, which requires understanding a long input and generating a concise output, the R-denoising mode — despite being associated with "understanding" — actually works best, possibly because the task of condensing information aligns with the span-corruption paradigm of extracting key information from context.
One-shot SuperGLUE: all prompts improve over no prompt (38.3 without prompt → 38.5–38.7 with prompts), but the differences between prompts are small. SuperGLUE appears less sensitive to mode selection — the tasks are discriminative enough that any well-trained mode works reasonably. [X] performs marginally best at 38.7.

The critical insight is that "getting the prompt right seems to be crucial for good performance" on generation tasks (Section 4.4), validating that mode switching is not merely cosmetic — the model genuinely develops mode-specific behaviors, and selecting the wrong mode can cause substantial degradation.

Design choice: three modes, not continuous mode selection. The authors could have designed a more fine-grained mode-selection mechanism (e.g., one token per specific denoiser configuration, or a continuous control signal). They chose three discrete tokens corresponding to the three denoiser paradigms (R, S, X) for simplicity and because their ablations suggest that the key distinction is between these three regimes, not between the specific configurations within each regime. The mapping from the seven denoiser configurations to three paradigm tokens is: all R configurations get [R], the single S configuration gets [S], all four X configurations get [X].

Relationship to prompt engineering: the paradigm tokens function as a form of "model-level prompt engineering" — rather than designing elaborate natural-language prompts to coax different behaviors from the model (as with GPT-3), UL2 bakes the mode-switching capability into pre-training and exposes it through a simple token interface. This makes downstream use simpler and more reliable: the practitioner doesn't need to discover the right prompt through trial and error; they select from a small, well-understood set of tokens that map to known pre-training behaviors.

Architecture-Agnostic Design: Decoder-Only and Encoder-Decoder Variants

UL2 is explicitly designed to be architecture-agnostic — the MoD objective and mode switching can be applied to both decoder-only (single-stack Transformer) and encoder-decoder (two-stack Transformer) architectures. The paper trains and evaluates both variants at equal compute (decoder at 167M parameters, encoder-decoder at 335M parameters — approximately 2× the parameters but similar FLOPs per token due to the differing structures).

Why architecture agnosticism matters: the paper argues (Section 3.2) that the field systematically conflates architecture choice with objective choice. The standard narrative is "decoder-only = causal LM = good at generation" and "encoder-decoder = span corruption = good at understanding." By showing that UL2 outperforms T5 (span corruption encoder-decoder) and GPT-like models (causal LM decoder) in both architectures, the authors isolate the objective as the primary driver of universality.

The decoder-only variant uses a single Transformer stack that processes the concatenation of inputs and targets. However — and this is a crucial detail — the decoder-only implementation does not use a full causal attention mask over the entire sequence. Instead, it uses a prefix-LM architecture: the input portion of the sequence receives bidirectional attention (every input token can attend to every other input token), and only the target portion is autoregressive (each target token attends to all input tokens and all previous target tokens). This is stated in Section 4.2.3: "our experiments always use a bidirectional receptive field only in its input segment and autoregressive decoding at the targets segment. This is essentially the PrefixLM-type architecture which we find to be consistently better than a full causal decoder model."

The encoder-decoder variant uses separate encoder and decoder stacks. The encoder processes the entire input bidirectionally, producing contextualized representations. The decoder processes the target autoregressively, attending to the encoder output via cross-attention at each layer. This architecture has approximately 2× the parameters of the decoder-only variant at equal compute (Table 2: ED at 335M params vs Dec at 167M params).

The efficiency trade-off: the paper frames the choice between architectures as primarily an efficiency consideration (Section 3.2):

Encoder-decoder models have more parameters for the same compute because the encoder and decoder can run in parallel during pre-training (the encoder processes inputs, the decoder processes targets, and they share compute across the batch). This "intrinsic sparsity" (different parameters for input processing vs target generation) can be beneficial for performance.
Decoder-only models have fewer parameters for the same compute because the single stack must process both inputs and targets sequentially. This makes them more parameter-efficient (better performance per parameter) but potentially worse at equal compute budgets.
For deployment where parameter count matters (storage constraints, on-device inference), decoder-only is preferred — you get a smaller model that's still quite capable. For deployment where throughput matters more than parameter count, encoder-decoder is preferred.

The paper's results (Table 2) show that UL2 encoder-decoder generally outperforms UL2 decoder on supervised tasks (73.10 vs 65.50 SuperGLUE, 31.86 vs 28.90 XSUM Rouge-L) but the gap narrows on one-shot tasks (41.30 vs 42.30 SuperGLUE, 11.51 vs 8.01 XSUM Rouge-L). Notably, UL2 decoder still outperforms T5 (encoder-decoder span corruption) on the absolute scale — 65.50 vs 72.00 on SuperGLUE (UL2 decoder loses, but is close given half the parameters) and dramatically on one-shot XSUM (8.01 vs 7.49 for T5).

The architectural implementation details: UL2 uses "a pretty standard vanilla T5 Transformer that have been enhanced with modifications that have withstood the test of time, i.e., GLU layers (Shazeer, 2020) and T5-style relative attention" (Section 3.2). GLU (Gated Linear Unit) layers replace standard feed-forward layers with a gating mechanism: $\text{GLU}(x) = (xW_1 + b_1) \odot \sigma(xW_2 + b_2)$ followed by an output projection. This has been shown to improve Transformer performance consistently. T5-style relative attention encodes positional information as relative offsets between tokens rather than absolute positions, which improves generalization to unseen sequence lengths.

For the tokenizer, UL2 uses "the default T5 English 32K sentencepiece" (Section 4.2.3), meaning it inherits T5's subword vocabulary without modification. This ensures comparability with T5 baselines and avoids confounds from tokenizer changes.

Pre-Training Implementation Details

The pre-training pipeline is implemented using the open-source T5X framework (Roberts et al., 2022) and Flaxformer library in JAX/Flax. The key hyperparameters for the ablative experiments (Section 4.2.3) are:

Training steps: 500K steps
Batch size: 128
Sequence length: 512 tokens for inputs, 512 tokens for targets (1024 total per example)
Total tokens seen: approximately 32 billion tokens (500K steps × 128 batch × 512 target tokens ≈ 32.8B)
Corpus: C4 (Colossal Clean Crawled Corpus), the same dataset used by T5
Optimizer: Adafactor (Shazeer & Stern, 2018) with inverse square root learning rate schedule
Hardware: 64–128 TPUv4 chips
Model sizes: decoder-only at ~167M parameters, encoder-decoder at ~335M parameters

For the scaled-up 20B model (Section 5.1), the configuration is:

Training steps: 2 million steps (approximately 1 trillion tokens on C4)
Batch size: 1024
Sequence length: 512 inputs / 512 targets
Dropout: set to 0 during pre-training
Hardware: 512 TPUv4 chips
Model parallelism: 8
Architecture: 32 encoder layers, 32 decoder layers, $d_{\text{model}} = 4096$ , $d_{\text{ff}} = 16384$ , 16 attention heads with $d_{\text{head}} = 256$ per head
Pre-training duration: "slightly more than one month for about 1 trillion tokens"
Loss spikes: the authors note that "spiking and instabilities are common when scaling up models" and that they "did not specifically control or put in place any mitigation strategies such as occasional restarts" — the 20B model experienced occasional loss spikes during training, but the resulting checkpoints still achieved SOTA performance on many tasks.

Implementation simplicity: the authors emphasize (Section 3.1.2) that "the implementation of UL2's mixture of denoiser is simple and easy to implement using a library like seqio." The appendix provides a complete Python function ul2_objective (Section 9.2) that implements the MoD objective in roughly 130 lines of code. The core logic is straightforward: define a set of (mean_noise_span_length, noise_density) pairs, apply the span corruption preprocessor to each shard of the dataset with the corresponding parameters, then mix the shards using tf.data.experimental.sample_from_datasets for uniform sampling. The S-denoising (prefix LM) component is added by calling t5.data.preprocessors.prefix_lm on an additional shard.

The Relationship Between MoD and Prior Multi-Objective Training

The paper explicitly contrasts MoD with simpler multi-objective approaches to highlight the non-obvious design choices that make it work:

Vs. SCLM (Span Corruption + Language Model, 50-50 mix): SCLM simply mixes T5's standard span corruption with causal language modeling at equal proportions. This is the most obvious baseline and indeed performs well — SCLM encoder-decoder achieves +28.3% overall gain over T5 (Table 3), outperforming most other baselines. However, SCLM misses two things that MoD provides:

The intermediate regimes (X-denoising): SCLM only has the two endpoints — short-span/low-corruption (T5) and full-sequence (LM). MoD adds configurations at $\mu = 64$ , $r = 0.5$ and other intermediate points that bridge the gap, providing a smoother learning curriculum.
Mode switching: SCLM does not use paradigm tokens to distinguish objectives, so the model must infer from input patterns alone which mode to operate in. This may cause the model to average across modes, producing behavior suboptimal for any specific one.

The quantitative comparison (Table 3) shows that UL2 encoder-decoder achieves +43.6% overall gain vs T5, while SCLM encoder-decoder achieves +28.3% — a substantial gap. The pattern is similar compared to the GPT-like baseline (Table 4): UL2 decoder achieves +45.7% vs SCLM decoder at +15.8%.

Vs. UniLM (Dong et al., 2019): UniLM trains on unidirectional LM, bidirectional LM, and seq2seq LM using a single Transformer model, but with a cloze-style (BERT-like) formulation that predicts masked tokens in-place rather than autoregressively generating them. The UL2 authors reimplement UniLM in the input-to-target style for fair comparison (Section 4.1). UniLM encoder-decoder achieves +19.8% overall gain vs T5 (Table 3) — better than T5 but substantially behind UL2's +43.6%. The gap likely arises because:

UniLM's mixture of causal LM, bidirectional LM, and seq2seq LM doesn't include the extreme denoising regimes that MoD emphasizes.
UniLM (in its original formulation) uses position-wise loss computation and requires task-specific heads, which the authors consider incompatible with the goal of a universal input-to-target model.
The lack of explicit mode switching tokens means the model doesn't cleanly separate the different paradigms.

Vs. T5's internal ablation of high corruption rates: The T5 paper (Raffel et al., 2019) tested a 50% corruption rate and found it performed worse than the standard 15% rate. This had been interpreted as evidence that high corruption is harmful. UL2's ablation Var-H (Table 6) — which uses only long spans at 15% corruption without short spans — confirms that extreme denoising as a standalone objective does not perform well. But UL2 variants that combine extreme denoising with regular denoising (Var-D, E, F) achieve substantially better one-shot performance (e.g., 10.6 Rouge-L for Var-D vs lower for variants without long spans). This demonstrates that the negative result in T5 was an artifact of testing extreme denoising in isolation rather than as part of a mixture — a classic example of interaction effects that would be invisible in single-factor ablation studies.

Summary of Key Design Choices and Their Justifications

Uniform sampling across seven denoiser configurations rather than weighted sampling: ensures all paradigms receive sufficient training signal; weighted sampling (e.g., 50% S-denoising) was tested and found to hurt overall performance (Section 4.5).
Three paradigm tokens ([R], [S], [X]) rather than seven (one per specific configuration) or one: three tokens provide enough granularity to distinguish the major operational modes while keeping the interface simple and avoiding over-fragmentation of the training signal across too many modes.
Prefix-LM decoder architecture rather than full causal decoder for the decoder-only variant: bidirectional encoding of the input is strictly better than unidirectional encoding (as shown by PrefixLM consistently outperforming causal LM across all tasks in Table 2), and the input-to-target separation still enables autoregressive generation of targets. The causal LM baseline is the worst-performing configuration across the board (Tables 3 and 4).
C4 corpus rather than a curated mixture: keeps comparability with T5 and avoids confounds from dataset quality differences. The authors acknowledge that this puts UL2 at a disadvantage compared to models trained on curated data (PaLM, GLaM) but choose consistency over absolute performance maximization.
Input-to-target (next-token prediction on targets) rather than in-place (cloze-style) formulation: the autoregressive target generation is more general — it subsumes both classification (by generating the class label) and generation tasks without requiring task-specific output heads. The authors are explicit that they consider BERT-style masked language modeling "somewhat deprecated" for most use cases (Section 2.1).
No dropout during 20B pre-training (set to 0): a common practice at large scale where the vast data quantity provides sufficient regularization, and dropout would slow convergence.
No dedicated causal LM configuration in the mixture: S-denoising with zero or very short prefix naturally reduces to causal LM behavior, so a separate configuration would be redundant. This also means the model never sees the "no bidirectional context at all" regime that pure causal LMs operate in, which may be why UL2's generation quality remains strong without sacrificing NLU performance.

4. Key Insights and Innovations

Innovation 1: Pre-Training Objectives as Points on a Continuous Spectrum, Not Discrete Alternatives

The paper's most fundamental conceptual move is reframing all dominant pre-training objectives — span corruption, prefix language modeling, and causal language modeling — as instances of a single parameterized function (SpanCorrupt, characterized by mean span length µ and corruption rate r) rather than as categorically different approaches that require different architectures or training pipelines. Prior to UL2, the field largely treated these objectives as discrete design choices: you picked T5-style span corruption for NLU, or GPT-style causal LM for generation, and the choice entangled with architecture selection (encoder-decoder vs decoder-only). The paper diagnoses this as a category error — conflating objective with architecture — and demonstrates through the unified perspective that the full two-dimensional (µ, r) space contains intermediate regimes (long spans at moderate corruption, short spans at high corruption) that no prior work had systematically explored or mixed.

What makes this reframing distinctive is not the mathematical formalism (which is straightforward) but the design it enables. Once you see that causal LM is approximately µ = L, r = 1.0 and T5 span corruption is µ ≈ 3, r = 0.15, it becomes natural to ask: what about µ = 64, r = 0.5? What capabilities does that teach that the endpoints miss? The paper's answer — X-denoising — emerges directly from this reframing. A practitioner working within the discrete-objective paradigm would never think to try "span corruption with paragraph-length spans and 50% masking" because that doesn't correspond to any named objective. The spectrum perspective makes it obvious.

This innovation is fundamental rather than incremental. It changes the question from "which objective is best?" (a discrete choice problem that prior work attacked through empirical comparison — e.g., Wang et al., 2022a) to "which points on the spectrum, and in what proportion, produce the most universally capable model?" (a continuous design problem). The shift parallels how the scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) reframed model sizing from "pick a size" to "optimize along a compute-optimal frontier" — it turns a discrete menu into a continuous optimization landscape. The empirical validation comes from the MoD ablation study (Table 6), which shows that interpolating along the spectrum (Var-D with µ up to 32) substantially outperforms either endpoint alone (Var-C without long spans, Var-H with only long spans) — direct evidence that the intermediate region contains useful signal invisible to single-point training.

Innovation 2: Complementarity as the Mechanism Behind Objective Mixing, Not Mere Ensembling

A natural (and common) approach to combining multiple pre-training objectives is simple ensembling: train on objective A half the time and objective B half the time, and hope the model averages to something useful. The paper's SCLM baseline (span corruption + language model, 50-50 mix) represents this philosophy. It performs well — +28.3% overall gain over T5 in the encoder-decoder setting (Table 3) — which might suggest that objective mixing is straightforward and the remaining gains are incremental.

UL2's key insight is that the benefit comes not from averaging across objectives but from complementarity — different denoisers teach fundamentally different capabilities that a single objective cannot provide, and the mixture must be carefully composed to cover the full capability space rather than simply blending the two most popular endpoints. The evidence for this is the asymmetric sensitivity to removing individual components observed in the MoD ablations (Table 6). Removing S-denoising entirely (Var-K: 0% S-denoising) improves supervised SuperGLUE from 73.1 to 73.7 but crushes one-shot XSUM from 10.4 Rouge-L to 2.6 — a 75% relative degradation. This is not the behavior of an ensemble where each component contributes incrementally; it is the signature of a necessary capability (sequential generation) that R- and X-denoising fundamentally cannot teach. Conversely, removing long spans (Var-C with max µ = 12 vs Var-D with µ up to 32) degrades one-shot performance without helping supervised scores — the long spans teach a capability (long-range generation from limited context) that short spans alone cannot.

This diagnostic is more than an empirical finding; it is a methodological contribution about how to design multi-objective pre-training. Prior multi-objective work (UniLM, SCLM) mixed objectives based on architectural convenience or precedent, without a theory of which capabilities each objective provides and whether the mixture covers all necessary capabilities. UL2's MoD composition (Table 1) is explicitly designed to cover three capability dimensions: multi-hop bidirectional reasoning (R-denoising), autoregressive coherence (S-denoising), and long-range generation under information scarcity (X-denoising). The ablation study functions as a capability audit: remove one component, observe which downstream tasks collapse, and you learn which capability that component was providing. This framework — designing mixtures based on capability coverage rather than precedent — is a conceptual advance that generalizes beyond UL2's specific denoiser set.

The innovation is fundamental for the objective design subfield but incremental for the broader transfer learning literature (which has long understood multi-task complementarity — e.g., multilingual training provides cross-lingual transfer that monolingual training cannot). What distinguishes it is the application to self-supervised pre-training objectives rather than supervised tasks, and the explicit decomposition of denoisers by capability rather than by task format.

Innovation 3: Paradigm Tokens as Lightweight Task-Identity Signals That Enable Mode-Specific Computation

The concept of mode switching via dedicated sentinel tokens ([R], [S], [X]) appears deceptively simple — just prepend a token to the input — but it represents a distinctive approach to a problem that had been tackled very differently in prior work. The standard approach to getting a single model to handle multiple types of tasks was either (a) architectural partitioning (different output heads for classification vs generation, as in UniLM's original cloze-style formulation with task-specific classifiers) or (b) natural language prompting (describe the task in a natural language prompt, as with GPT-3, T0, and FLAN). UL2's paradigm tokens occupy a middle ground between these extremes: they are more structured than natural language prompts (they have fixed, well-defined meanings tied to specific pre-training paradigms), but more flexible than architectural partitioning (they reuse the same model weights for all modes, changing behavior purely through the input).

What makes this innovative is the recognition that the model needs to know which pre-training paradigm is active not just during pre-training (where the input pattern provides some signal) but especially during downstream fine-tuning, where the input format may not clearly distinguish whether a R-denoising-like or S-denoising-like behavior is appropriate. The paradigm tokens bind downstream behavior to specific pre-training modes by making the mode identity an explicit, discriminable feature of the input. The ablation in Table 5 validates this: on one-shot XSUM (generative summarization), using the wrong paradigm token ([X]) causes a 48% relative degradation compared to the right one ([R]) on Rouge-1, demonstrating that the model has learned mode-specific computation that can be cleanly toggled.

This is a modularity advance: it separates the problem of "what capabilities should the model have" (solved by MoD composition) from "which capability should be active for this task" (solved by paradigm token selection). Prior work conflated these — to change the model's behavior for a new task, you either changed the architecture (added a new head) or engineered a new prompt. Paradigm tokens provide a small, interpretable interface for mode selection that is learned during pre-training and transferable to any downstream task.

The innovation is incremental in its mechanism (adding special tokens is not new — BERT's [CLS] and [SEP] tokens serve similar structural roles) but fundamental in its implications: it suggests that future universal models should be designed with explicit mode interfaces that partition the model's capability space into discrete, selectable behaviors, rather than relying on the model to infer the desired mode from context or natural language instructions alone. This connects to the broader research direction of controllable generation and model steerability, but approaches it from the pre-training design perspective rather than the post-hoc fine-tuning or prompting perspective.

Innovation 4: X-Denoising as a Novel Pre-Training Regime That Reconciles Conflicting Prior Findings

The inclusion of X-denoising (extreme denoising with long spans and high corruption rates) is not merely another point in the MoD mixture — it represents a diagnostic resolution of a confusing empirical contradiction in the prior literature. The T5 paper (Raffel et al., 2019) had tested a 50% corruption rate as a standalone objective and found it performed worse than the standard 15% rate, leading to the received wisdom that high corruption is harmful. Wettig et al. (2022), studying BERT-style masked language modeling, had meanwhile found that increased corruption rates improved sample efficiency for larger models — a seemingly contradictory finding. The UL2 paper's ablation Var-H (Table 6: only long spans at 15% corruption, no short spans) confirms T5's negative result — extreme denoising alone underperforms. But Var-D (which includes both long spans at high corruption and short spans at standard corruption) substantially outperforms the standard T5 configuration on one-shot tasks (10.6 vs 6.5 Rouge-L on XSUM) while matching on supervised tasks.

The innovation here is the identification of an interaction effect: extreme denoising is beneficial only when mixed with easier denoising regimes, not when used in isolation. This is a classic example of a superadditive combination where the whole is greater than the sum of its parts — each denoiser in isolation performs worse than the best single denoiser, but the mixture outperforms any single denoiser by a substantial margin. This pattern is invisible to standard ablation studies that test each component in isolation and compare against a single-component baseline. It explains why T5's negative result was correct but incomplete — the experiment design (compare standalone 50% corruption to standalone 15% corruption) could not detect the complementary benefit that emerges from mixing.

This innovation is fundamental as a methodological lesson for pre-training objective design: the standard paradigm of "propose a new objective, test it in isolation against baselines" can systematically miss superadditive combinations. The success of MoD suggests that future work should explore the interaction space between pre-training objectives rather than treating each new objective as a candidate to replace existing ones. This parallels a broader trend in machine learning — the recognition that ensembles and mixtures often outperform individual components not through averaging but through complementarity — but applies it at the level of self-supervised pre-training design rather than model ensembling.

The practical significance is substantial: it means the search space for effective pre-training recipes is not the set of individual objectives but the set of possible mixtures, which is combinatorially larger. UL2's specific mixture (seven configurations across R, S, X) represents one point in this space, empirically validated but not claimed to be optimal. The paper's ablation study (Table 6) provides a partial map of this interaction space — varying span lengths, corruption rates, and S-denoising proportions — that future work can build on. This shifts the research agenda from "design a better objective" to "design a better mixture of objectives," which is a qualitatively different optimization problem.

Innovation 5: Empirical Demonstration That Objective Design Can Partially Substitute for Scale

While not framed as the primary contribution, the paper's demonstration that UL2 20B achieves capabilities previously associated with much larger models represents a significant empirical finding with practical implications. Specifically:

Chain-of-thought reasoning (Section 5.2.7): Prior work (Wei et al., 2022b) had shown that chain-of-thought prompting was an "emergent ability of scale" — it only improved performance for very large models (100B+ parameters) and actually hurt performance for smaller ones. UL2 20B not only benefits from chain-of-thought prompting (+2.2 points averaged across five arithmetic benchmarks, from 13.5 to 15.3 without calculator, Table 12) but also enables self-consistency to deliver substantial further gains (15.3 → 36.0 with calculator + self-consistency). This makes UL2 "the first publicly available pre-trained model (without any fine-tuning) to successfully leverage CoT prompting" at this scale, democratizing access to a capability previously locked behind proprietary 100B+ models.
Zero-shot SuperGLUE (Table 10): UL2 20B achieves 65.2 average score (best checkpoint), outperforming GPT-3 175B's published result of 61.2 — meaning the objective design compensates for a roughly 8.75× reduction in parameter count. Notably, UL2 20B also outperforms all other compute-matched models (T5-XXL + LM adaptation at 52.5, PaLM-Dense 8B at 63.9, GLaM-Dense 8B at 61.3), isolating the effect of the objective from the effect of scale.
FLAN instruction tuning (Section 5.3): After FLAN training, FLAN-UL2 20B achieves 55.7 on MMLU and 45.9 on BBH (direct prompting), "almost reaching the performance of Flan-PaLM 62B" (59.6 and 47.5 respectively, Table 15) — a model with 3.1× more parameters. On the CoT variants, the gap narrows further (MMLU-CoT: 52.2 for FLAN-UL2 20B vs 56.9 for FLAN-PaLM 62B; BBH-CoT: 42.7 vs 44.9; Table 16).

What makes this finding innovative is not the raw numbers but the implication for resource allocation. Prior to UL2, the dominant narrative (driven by scaling laws work) was that capability improvements come primarily from scaling model size and data, with architectural and objective choices providing marginal gains at best. UL2 demonstrates that, at least for the specific capability profile of "universal competence across both understanding and generation," objective design can shift the performance frontier by an amount equivalent to a 3–9× increase in parameter count. This doesn't contradict scaling laws — UL2's performance still improves with scale — but it suggests that the intercept of the scaling curve can be substantially shifted by better pre-training recipes, making smaller models more capable than they "should" be based on parameter count alone.

The innovation is incremental as a conceptual claim (the idea that training recipe matters is not new — see the long history of BERT optimization in Liu et al., 2019, or the T5 paper's own extensive ablation of pre-training objectives) but fundamental in its magnitude and specificity. Prior work had shown that better pre-training recipes could improve performance by a few percentage points; UL2 shows that the right mixture of objectives can triple one-shot summarization performance (T5-XXL + LM achieves 10.7 Rouge-L; UL2 20B achieves 19.8; Table 11) and enable qualitatively new capabilities (chain-of-thought reasoning at 20B scale). The paper doesn't formalize this as a scaling law (e.g., "UL2's objective is equivalent to a 3× parameter multiplier"), but the empirical pattern is clear enough to be practically actionable: for organizations deciding between training a larger model with a standard objective and training a smaller model with a carefully designed objective mixture, the latter may be more cost-effective.

A critical caveat, which the paper does not emphasize but is evident from the results, is that the substitution is not uniform: UL2's gains over T5 are much larger on generation and few-shot tasks (+363% on one-shot SGD, +210% on one-shot TOTTO; Table 3) than on supervised NLU (+1.5% on SuperGLUE). The objective design primarily addresses T5-style models' weakness at generation, not their strength at understanding. This means the "substitute for scale" finding is task-dependent — for pure NLU, scaling T5 may still be more efficient than training UL2. This nuance is important because it prevents overgeneralization: objective design shifts the Pareto frontier, but does not eliminate the trade-off entirely.

5. Experimental Analysis

Evaluation Methodology

Dataset. The ablative experiments (Section 4) use a custom suite of 9 diverse tasks drawn from two benchmark collections. For language understanding, the authors use the SuperGLUE benchmark (Wang et al., 2019), which comprises 8 NLU sub-tasks: BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. For language generation, they select three datasets from the GEM benchmark (Gehrmann et al., 2021): XSUM (extreme summarization), ToTTo (table-to-text generation), and Schema Guided Dialog (SGD). In addition, they evaluate open-text generation quality using perplexity on the C4 validation set. The scaled 20B experiments (Section 5) expand dramatically to cover approximately 50+ NLP tasks spanning language generation (CNN/DM, XSUM, MultiNews, SAMSum, WebNLG, E2E, CommonGen), human evaluation (GENIE benchmark), language understanding (RACE, QASC, OpenBookQA, TweetQA, QuAIL, IMDB, AgNews, DocNLI, AdversarialNLI, VitaminC), commonsense reasoning (HellaSwag, SIQA, PIQA, CosmosQA, AbductiveNLI, CommonsenseQA, CSQA2), long-range reasoning (Scrolls benchmark), structured knowledge grounding (UnifiedSKG tasks), and information retrieval (DSI-NQ). The authors deliberately selected this suite to provide "good coverage across many setups in the literature including supervised and conditional few-shot learning" (Section 4.2.1).
Base model(s). All ablative experiments use a standard Transformer enhanced with GLU layers (Shazeer, 2020) and T5-style relative attention, implemented in two architectures: decoder-only at ~167M parameters and encoder-decoder at ~335M parameters (approximately 2× parameters but similar FLOPs due to architectural differences). For the scaled experiments, the authors train UL2 20B (19.5B parameters): 32 encoder layers, 32 decoder layers, $d_{\text{model}} = 4096$ , $d_{\text{ff}} = 16384$ , 16 attention heads with $d_{\text{head}} = 256$ , using a T5 32K sentencepiece tokenizer. The paper argues that PaLM 2-S* — wait, that's from the prior paper. Let me be precise. The paper states the model uses "a pretty standard vanilla T5 Transformer" with the GLU and relative attention modifications, and the 20B model was pretrained on C4 using the T5X framework. The architecture choice for scaling follows from the ablative findings that encoder-decoder with UL2 objective works best for universality.
Metrics. The paper uses task-appropriate metrics throughout. For SuperGLUE, it reports the standard composite metrics: accuracy (BoolQ, COPA, RTE, WiC, WSC), F1 and Exact Match (MultiRC, ReCoRD), and a macro-average across all 8 tasks. For GEM benchmark tasks (XSUM, ToTTo, SGD), it uses ROUGE-L. For language modeling quality, it reports negative log perplexity on C4 validation — though notably, the tables present this as a raw negative log perplexity score where higher is better. For the 20B experiments, metrics vary by dataset: ROUGE-2 for summarization, accuracy/Exact Match/F1 for classification and QA tasks, BLEU for generation tasks (WebNLG, E2E, DART), human evaluation ratings for GENIE tasks, and Hits@10 for retrieval. The paper notes that "reporting rouge-1/rouge-2/rouge-L for all tasks may be a bit excessive" so it often reports a representative metric.
Holistic aggregate metric. Because the tasks span widely different metric types (accuracy, F1, ROUGE, perplexity), direct averaging is meaningless. Instead, the paper computes a normalized relative gain against two prime baselines: T5 (encoder-decoder span corruption) and GPT-like (decoder causal LM). For each task, the paper computes the percentage improvement of each method over the baseline: $(\text{score} - \text{baseline\_score}) / \text{baseline\_score} \times 100$ . These per-task percentage gains are then averaged (equally weighted) to produce the "All" column in Tables 3 and 4. The authors argue this "is the most suitable method for comparing these models since it is easy to reason about how much a new model is generally better than a popular setting." This metric normalizes away the different scales of different tasks and answers the question: on average, how much better is method X than the established baseline across this diverse suite?
Baselines. The paper compares against five pre-training objectives, each trained in both decoder-only and encoder-decoder architectures: (1) Causal Language Model (CLM) — standard left-to-right autoregressive LM, described as "GPT-like" (Radford et al., 2019; Brown et al., 2020); (2) Prefix LM (PLM) — causal LM where the input prefix receives bidirectional attention, with the prefix length uniformly sampled (Liu et al., 2018; Raffel et al., 2019); (3) Span Corruption (SC) — the standard T5 denoising objective with $\mu = 3$ , $r = 0.15$ (Raffel et al., 2019); (4) Span Corruption + LM (SCLM) — equal mixture of CLM and SC objectives, representing the most straightforward multi-objective baseline; (5) UniLM — the multi-objective approach from Dong et al. (2019), reimplemented in the input-to-target style (rather than the original cloze-style) to ensure architectural comparability. For the 20B scaled experiments, additional comparison points include T5-XXL (11B), T5-XXL with LM adaptation (Lester et al., 2021), GPT-3 175B, GLaM, PaLM at various scales, and ST-MoE-32B.
Generation budget / compute accounting. For the ablative experiments, all models are trained for 500K steps with batch size 128 and sequence length 512/512 (inputs/targets), totaling approximately 32 billion tokens on the C4 corpus. The compute-matching between architectures deserves careful attention: the decoder-only models have ~167M parameters while encoder-decoder models have ~335M, meaning encoder-decoders have approximately 2× the parameter count. The authors justify this by noting that "encoder-decoder models in these set of results have approximately twice the number of parameters as the decoder models but have similar speeds" due to the parallel processing of encoder and decoder stacks during training (Section 4.3.1). This means the comparison is compute-matched but not parameter-matched — encoder-decoder models get more parameters for the same FLOPs, which the authors characterize as a form of "intrinsic sparsity." For the 20B model, pretraining uses a batch size of 1024, 512 TPUv4 chips, 2 million steps, totaling approximately 1 trillion tokens on C4.
Cross-validation / statistical protocol. The ablative experiments do not employ cross-validation in the traditional sense — models are pretrained once and evaluated on standard benchmark splits (SuperGLUE dev set, GEM test/dev sets). For the 20B experiments, the paper uses a continuous fine-tuning protocol: "after each Nk steps of pretraining, we finetune on each downstream task and record its results" (Section 5.2.1), with fine-tuning typically conducted for 50K–100K steps. The best result across checkpoints is reported. This introduces a potential for checkpoint selection bias — the paper effectively selects the best-performing checkpoint per task, which may overstate performance relative to methods that report from a single fixed checkpoint. The authors acknowledge this implicitly: "since many finetuning experiments using these checkpoints still often result in sota performance, we let it be for now." For leaderboard submissions, results are reported on official test sets and represent single submissions. For the mode-switching ablation (Table 5), statistical significance is not reported, but the observed differences are very large (48% relative gap on XSUM Rouge-1 between best and worst paradigm token), making the effects practically unambiguous.

Main Quantitative Results

Ablative Comparison: UL2 vs. Baselines Across 9 Tasks (Tables 2, 3, 4)

The core ablation results are presented in three interconnected tables. Table 2 reports raw scores on all 9 individual tasks (SuperGLUE supervised, XSUM supervised, SGD supervised, TOT supervised, SuperGLUE one-shot, XSUM one-shot, SGD one-shot, TOT one-shot, and LM perplexity) for every combination of objective and architecture. Tables 3 and 4 convert these raw scores into relative percentage gains over the two canonical baselines — T5 (encoder-decoder span corruption, Table 3) and GPT-like (decoder causal LM, Table 4) — and report both the per-task gains and a normalized overall average.

Headline finding: UL2 encoder-decoder achieves +43.6% overall gain over T5 and +76.1% over GPT-like, outperforming all baselines on all 9 tasks (Tables 2, 3, 4). This is the central quantitative claim of the ablation study. Reading from Table 3's "All" column: UL2 encoder-decoder achieves 43.6% overall gain over the T5 baseline, compared to +28.3% for the next-best multi-objective baseline (SCLM encoder-decoder), +19.8% for UniLM encoder-decoder, and +16.7% for PrefixLM encoder-decoder. Against the GPT-like baseline (Table 4), UL2 encoder-decoder achieves +76.1% overall gain, compared to +60.3% for SCLM encoder-decoder and +48.6% for PrefixLM encoder-decoder. The "Win" columns in Tables 3 and 4 report how many of the 9 tasks each method wins against the baseline — UL2 encoder-decoder wins 9 out of 9 against T5 and 8 out of 9 against GPT-like (the only loss being on LM perplexity, where the causal LM baseline has a natural advantage).

UL2's consistency is as important as its average. While other methods sometimes outperform UL2 on individual tasks — for example, SCLM encoder-decoder achieves +508% on one-shot SGD vs UL2's +363% (Table 3, "SGD" one-shot column) — UL2 is the only method that never loses substantially on any task. Quoting Section 4.3.4: "Even when it loses to another method on a task, the loss is relatively marginal (e.g., 6.5 vs 7.3 on one-shot TOTTO). Conversely, when UL2 outperforms a baseline like T5, the gain can be as large as +363%." This property — being consistently strong rather than spiky — is what makes UL2 a "universal" model.

The gains are asymmetric across task types. Breaking down Table 3 by task category reveals a clear pattern: UL2's gains over T5 are modest on supervised NLU (+1.5% on SuperGLUE) but dramatic on one-shot generation (+363% on one-shot SGD, +210% on one-shot TOTTO, +53.6% on one-shot XSUM). This asymmetry is expected given the design: T5 was already strong at supervised NLU, so improvements there are hard-won, but T5 was genuinely weak at in-context generation, which is precisely the gap that S-denoising and X-denoising in MoD address. The gains over GPT-like (Table 4) show the mirror image: UL2 improves supervised SuperGLUE by +17.4% but one-shot generation by enormous margins (+3150% on one-shot TOTTO, +1754% on one-shot XSUM) — because the GPT-like baseline was weak at NLU and essentially non-functional at certain generation formats.

Architecture matters but less than the objective. Comparing UL2 decoder (167M) vs UL2 encoder-decoder (335M) in Table 2: the encoder-decoder variant outperforms on supervised tasks (SG: 73.10 vs 65.50; XS: 31.86 vs 28.90) but the decoder variant is competitive on one-shot tasks (SG: 41.30 vs 42.30; XS: 11.51 vs 8.01) even with half the parameters. Most importantly, UL2 decoder actually outperforms T5 encoder-decoder on the overall metric (+14.6% in Table 3's "All" column), despite having half the parameters — a striking demonstration that the objective can compensate for architectural disadvantages. The authors draw the conclusion: "we were able to push the UL2 decoder to outperform the T5 encoder-decoder setup by +14.6%. That said, this UL2 decoder does not outperform our UL2 encoder-decoder. However, this reinforces our point that the self-supervision objective may be intrinsically more important than the backbone architecture" (Section 4.3.1).

GPT-like is the worst configuration; PrefixLM is the best single-objective baseline. Table 2 shows that the causal LM decoder (GPT-like) is uniformly the worst or near-worst on every task. Among single-objective baselines, PrefixLM encoder-decoder emerges as the strongest, achieving +16.7% overall gain over T5 (Table 3), with particularly large advantages on one-shot generation (+397% on SGD, +86% on TOTTO) while sacrificing only modestly on supervised NLU (-3.7% on SuperGLUE). This supports the paper's recommendation to "always at least training with Prefix-LM or UniLM whenever possible" instead of pure causal LM.

Scaled UL2 20B: Supervised Fine-Tuning Results (Table 8)

The 20B model was evaluated on approximately 50+ supervised NLP tasks through per-task fine-tuning. Table 8 reports the results, comparing UL2 20B against the previously published state-of-the-art for each dataset.

Headline finding: UL2 20B achieves SOTA on the vast majority of tasks, often by substantial margins. Scanning Table 8: on summarization, UL2 achieves 21.9 ROUGE-2 on CNN/DM (vs 21.7 SOTA), 29.6 on SAMSum (vs 28.3), and 21.7 on MultiNews (vs 21.1). On data-to-text, UL2 reaches 55.4 ROUGE-2 on WebNLG (vs 53.5), 46.5 on E2E (vs 45.8), and 37.4 on CommonGen (vs 32.5 — a 4.9-point improvement). On Schema-Guided Dialog, the improvement is dramatic: 44.1 ROUGE-2 vs 36.8 SOTA, a 7.3-point absolute gain. On the Scrolls benchmark (long-range reasoning), UL2 20B achieves an aggregate score of 37.9 vs 29.2 SOTA — an 8.7-point absolute improvement (29.8% relative). Individual Scrolls tasks show gains ranging from modest (+0.1 on QMSum) to enormous (+19.8 on QuALITY EM, nearly doubling the prior SOTA). On structured knowledge grounding (UnifiedSKG), UL2 outperforms prior work on all 10 tasks, with gains of 8.1 points on SQA (62.4 → 70.5), 3.7 points on TabFat (83.4 → 87.1), and 3.2 points on DART BLEU (47.2 → 50.4).

The exceptions where UL2 does not achieve SOTA are informative. On XSUM, UL2 achieves 26.6 ROUGE-2 vs 27.1 SOTA (a 0.5-point miss). On several GENIE human evaluation metrics (WMT19 German-to-English translation, XSUM overall and fluency), UL2 matches or slightly trails prior SOTA — the authors note that for WMT19, their model was "pretrained on only English C4 then finetuned on only the provided WMT19 data (no German pretraining, parallel data or backtranslation)," making the comparison unfair but demonstrating robustness. On CosmosQA (91.6 vs 91.8), Winogrande (90.1 vs 91.3), and AbductiveNLI (87.5 vs 89.8), UL2 falls short by small margins — these are commonsense reasoning tasks where specialized models (UNICORN, Lourie et al., 2021) had set high bars. On GovRep within Scrolls (36.2 vs 37.2), the paper explains that the SOTA model used 16K sequence lengths vs UL2's 2K, and "it is expected that increasing the length to UL2 would significantly improve our scores."

Human evaluation results (GENIE) validate generation quality. On the GENIE leaderboard (Table 8, middle section), UL2 achieves SOTA on aNLG (77.0 vs 76.0 human rating), matches SOTA on ARC-DA (72.0), and posts competitive scores on XSUM (overall 50 vs 51 SOTA, with better no-hallucination at 54 vs 53). The WMT19 result (67 vs 71 SOTA) is the only clear loss, but as noted above, the model had no German pretraining. These results matter because they confirm that UL2's generation improvements on automated metrics (ROUGE) are not an artifact of metric exploitation — human raters also prefer UL2's outputs or find them comparable to prior SOTA.

The breadth of evaluation is unusual and strengthens credibility. Rather than cherry-picking tasks where UL2 excels, the authors evaluate on a remarkably diverse set spanning summarization, data-to-text, question answering, commonsense reasoning, long-document tasks, structured knowledge, retrieval, and classification — roughly 50 distinct benchmarks. Achieving SOTA on most of them with a single pretrained checkpoint (though fine-tuned per-task) is strong evidence for the universality claim.

Fine-Tuning vs. Zero-Shot Trade-off on SuperGLUE (Table 9)

Headline: UL2 20B achieves 90.7 average SuperGLUE score, outperforming T5-11B (89.9) and PaLM 8B (83.4), while trailing larger models. Table 9 reports supervised fine-tuning results on the SuperGLUE dev set. UL2 20B's 90.7 average outperforms T5-11B by 0.8 points, demonstrating that the MoD objective does not sacrifice T5's core strength (NLU fine-tuning) while adding generation capabilities. However, UL2 trails ST-MoE-32B (93.2, a 269B-parameter sparse model costing equivalent to 32B dense) and PaLM 540B (92.6), as expected given the parameter and compute disparities. The per-task breakdown shows UL2 excelling on certain tasks (CB: 98.7/98.2 F1/EM vs ST-MoE's 100/100; WSC: 98.1 vs 96.2 for T5-11B) while underperforming on others (ReCoRD: 93.7/93.2 vs 95.0/95.6 for ST-MoE; RTE: 92.1 vs 95.7 for ST-MoE).

The authors acknowledge the limit: "UL20B still lacks behind the SOTA model ST-MoE-32B given two main reasons. Firstly, ST-MoE-32B has 200B+ parameters and is costs equivalent to a 32B dense model. Secondly, ST-MoE-32B is trained solely on span corruption using an encoder-decoder architecture which is known to be very advantageous on NLU finetuning." This honesty is important — it confirms that UL2's universality comes with a modest NLU tax compared to a model optimized purely for that regime.

Zero-Shot SuperGLUE: UL2 20B Outperforms GPT-3 175B (Table 10)

Headline: UL2 20B achieves 65.2 average on zero-shot SuperGLUE (best checkpoint), outperforming GPT-3 175B's published 61.2 and all compute-matched models. Table 10 reports zero-shot performance on SuperGLUE, where models receive no fine-tuning and must infer the task from the provided format and examples (or, for zero-shot, from the task description alone). UL2 20B's 65.2 average (using the best checkpoint per task) substantially exceeds GPT-3 175B despite having 8.75× fewer parameters. The single-checkpoint result (63.0) — which is more representative of typical deployment — still outperforms GPT-3 13B (57.2), GLaM-Dense 8B (61.3), PaLM-Dense 8B (63.9), and T5-XXL with LM adaptation (52.5), placing UL2 20B as the strongest model in its approximate compute class.

The per-task pattern reveals where UL2's strengths lie. On BoolQ (63.1), UL2 outperforms GLaM-MoE 64E (72.2) but trails PaLM 540B (88.0). On ReCoRD (88.1), UL2 is competitive with much larger models. The authors note that "UL20B remains competitive at a compute-matched setup despite only training on C4 corpus which is known to be less effective than specially curated datasets used in PaLM and GLaM." This is a significant caveat — the zero-shot comparison is somewhat unfavorable to UL2 because it was trained on less carefully curated data, meaning its performance may underestimate what the UL2 objective could achieve with better pretraining data.

The "best checkpoint" vs "single checkpoint" distinction matters. UL2 20B (best) at 65.2 is computed by selecting the best performing checkpoint per task, which inflates the result relative to methods reporting from a single fixed checkpoint. The single-checkpoint result of 63.0 is more conservative and still competitive. The gap between these numbers (2.2 points) gives a rough estimate of checkpoint variance, though without formal confidence intervals this remains qualitative.

One-Shot Summarization: Tripling T5-XXL Performance (Table 11)

Headline: UL2 20B achieves 19.8 ROUGE-L on one-shot XSUM summarization, approximately 3× the performance of LM-adapted T5-XXL (10.7) and competitive with PaLM 62B (11.2). Table 11 reports one-shot XSUM results, where the model sees a single example before generating a summary. UL2 20B's 19.8 ROUGE-L dramatically exceeds T5-XXL's 0.6 (without LM adaptation) and 10.7 (with LM adaptation). It also outperforms LaMDA 137B (5.4 ROUGE-2, though ROUGE-L is not reported for LaMDA) and PaLM 8B (7.9 ROUGE-2). However, UL2 20B still trails PaLM 62B (11.2 ROUGE-2 vs UL2's 8.6) and PaLM 540B (12.2 ROUGE-2) on the ROUGE-2 metric specifically, suggesting that at larger scales, the benefits of pure parameter count begin to dominate for this particular task.

This result is one of the paper's most dramatic demonstrations of objective-driven capability improvement. T5-XXL (11B parameters, similar scale to UL2 20B) is essentially non-functional at one-shot summarization without LM adaptation (0.6 ROUGE-L) — it wasn't trained to handle this format. Even with LM adaptation (a separate fine-tuning stage designed to make T5 better at in-context learning), it reaches only 10.7 ROUGE-L. UL2 achieves 19.8 without any adaptation — the MoD objective baked the necessary skills into pretraining itself.

Chain-of-Thought Reasoning at 20B Scale (Tables 12, 13)

Headline: UL2 20B is the first publicly available model at this scale to successfully leverage chain-of-thought prompting, with self-consistency improving average arithmetic reasoning by 22.5 points over standard prompting. Table 12 reports performance on five arithmetic reasoning benchmarks (GSM8K, SVAMP, ASDiv, AQuA, MAWPS) under three conditions: standard prompting, chain-of-thought (CoT) prompting, and CoT + calculator + self-consistency (SC). Standard prompting with UL2 20B achieves only 13.5 average accuracy — the model is not naturally strong at math word problems. CoT prompting alone provides a modest boost to 15.3. Adding a calculator (to handle arithmetic) substantially improves performance to 27.2. Adding self-consistency (sampling multiple CoT reasoning paths and taking the majority answer) further boosts to 36.0 — a 22.5-point improvement over standard prompting. The largest gains are on MAWPS (16.6 → 57.9, a 3.5× improvement) and SVAMP (10.1 → 41.4, a 4.1× improvement).

Table 13 reports similar experiments on five commonsense reasoning benchmarks (CSQA, StrategyQA, Date Understanding, Sports Understanding, ARC-easy/challenge). CoT prompting improves the average from 37.8 to 48.1, and self-consistency further boosts to 52.2 — a 14.4-point improvement. The gains are most dramatic on ARC-easy (32.2 → 69.8) and CSQA (34.2 → 55.7).

Why this matters: Wei et al. (2022b) had established that chain-of-thought reasoning is an "emergent ability of scale" — it only improved performance for models above ~100B parameters and actually hurt smaller models. UL2 20B not only benefits from CoT (the improvement from 13.5 to 15.3 on arithmetic, while small, is positive — not negative as would be expected at this scale from prior work) but also enables self-consistency to deliver further gains. The authors hypothesize that "the mixture of denoisers may contribute to the ability of UL2 to leverage CoT prompting at 20B parameters" (Section 5.2.7) — a plausible but unverified conjecture. The practical implication is that CoT reasoning, previously locked behind access to 100B+ models, becomes feasible with a publicly available 20B checkpoint.

A caveat on the absolute numbers: Even with self-consistency, UL2 20B's arithmetic performance (36.0 average) is far below what large models achieve (PaLM 540B reaches 56.9 on GSM8K with CoT + calculator + SC, vs UL2's 10.2). The paper is demonstrating that the capability exists at 20B scale, not that it matches large-scale performance.

MMLU Zero-Shot: Outperforming T0 and T5 (Table 14)

Headline: UL2 20B achieves 39.2% zero-shot accuracy on MMLU, outperforming T0 Strawberry (36.9%), T0 Vanilla (34.5%), and T5-XXL with LM adaptation (27.5%). Table 14 reports zero-shot MMLU performance. UL2 20B's 39.2% exceeds T0 models that were specifically fine-tuned for zero-shot task generalization, as well as T5-XXL with LM adaptation. The authors interpret this as evidence that MoD pre-training provides better zero-shot knowledge probing than either pure span corruption (T5) or multitask fine-tuning (T0). However, the absolute number (39.2%) leaves substantial room for improvement — MMLU has 4 answer choices, so random guessing would achieve 25%, and large models at the time were reaching 60–70%.

FLAN-UL2 20B: Competitive with Larger Instruction-Tuned Models (Tables 15, 16)

Headline: After FLAN instruction tuning, FLAN-UL2 20B achieves 55.6 MMLU and 45.3 BBH (direct prompting), competitive with FLAN-PaLM 62B and outperforming FLAN-T5 XXL by 1.1–1.3 points. Table 15 reports few-shot MMLU and BBH (Big-Bench Hard) results after FLAN instruction tuning. FLAN-UL2 20B's MMLU score of 55.6 (best checkpoint for both tasks) exceeds FLAN-T5 XXL's 54.5 and approaches FLAN-PaLM 62B's 59.6 — a model with 3.1× more parameters. On BBH, FLAN-UL2 achieves 45.3, matching FLAN-T5 XXL and trailing FLAN-PaLM 62B by 2.2 points. The individual-task-best results (46.0 BBH, 58.1 MMLU dev) show that with optimal checkpoint selection, FLAN-UL2 can nearly close the gap to FLAN-PaLM 62B on BBH (46.0 vs 47.5) and surpass it on MMLU dev (58.1 vs 59.6 — actually trailing by 1.5, not surpassing; my reading correction: the paper says "the best dev scores of FLAN-UL2 is almost reaching the performance of Flan-PaLM 62B on both MMLU and BBH").

Table 16 further breaks down performance by direct prompting vs. CoT prompting. FLAN-UL2 20B outperforms FLAN-T5 XXL on all four settings, with the largest relative gain on MMLU-CoT (+7.4%, 48.6 → 52.2). The average across all four settings is 49.1 for FLAN-UL2 vs 49.9 for FLAN-PaLM 62B — a 0.8-point gap despite the 3.1× parameter difference. FLAN-PaLM 540B remains far ahead at 67.2 average.

These results establish that UL2's pre-training recipe is compatible with and benefits from instruction tuning — it is not made obsolete by FLAN. The paper demonstrates that the best approach is to combine MoD pre-training with FLAN fine-tuning, achieving performance that approaches models 3× larger.

Ablation Studies and Robustness Checks

Decoder vs. encoder-decoder architectures (Table 2, Section 4.3.1): When compute-matched (decoder at 167M, encoder-decoder at 335M), encoder-decoder models consistently outperform decoder-only models on supervised tasks — the best decoder baseline (PrefixLM) is about 10% worse overall than the T5 baseline in Table 3 (PLM Dec at -11.0% vs T5 at 0.0%). However, UL2 decoder reverses this pattern, achieving +14.6% overall gain over T5 — showing that the right objective can make a decoder-only model competitive with or superior to encoder-decoder models on aggregate. The paper concludes that architecture choice is "mainly about efficiency trade-offs that can be studied independently" from objective design. The key robustness implication: UL2's benefits are not tied to a specific architecture, reducing the risk that the results are an artifact of encoder-decoder inductive biases.

GPT-like vs. T5-like as optimal configurations (Section 4.3.2): The paper explicitly tests whether the well-established defaults (causal LM decoder = GPT-like; span corruption encoder-decoder = T5-like) are already near-optimal. The answer is clearly no. Causal LM decoder is the worst configuration overall, losing to all baselines. PrefixLM encoder-decoder actually outperforms T5 on the overall metric (+16.7% in Table 3) despite being worse on SuperGLUE specifically — suggesting that the field's default (T5) was suboptimal for tasks beyond pure NLU. The paper makes "the straightforward recommendation of always at least training with Prefix-LM or UniLM whenever possible" over pure causal LM.

UniLM and SCLM baselines validate that objective mixing helps, but specific composition matters (Section 4.3.3): Both UniLM encoder-decoder (+19.8% overall) and SCLM encoder-decoder (+28.3%) outperform T5, confirming that multi-objective training is directionally beneficial. However, neither matches UL2 (+43.6%), demonstrating that how the objectives are mixed — specifically, the inclusion of X-denoising and mode switching — accounts for the remaining ~15 percentage points of gain. This is a critical ablation: it shows that the benefit of MoD is not simply "mixing objectives is good" (which SCLM already demonstrated) but rather "this specific mixture composition is substantially better than simple 50-50 mixing."

Mode switching effectiveness (Table 5, Section 4.4): Ablating the paradigm token choice on an encoder-decoder model pre-trained with UL2 on 7B tokens reveals two key findings. First, using the right prompt ([R] for XSUM summarization) vs. the wrong prompt ([X]) causes a 48% relative gap on Rouge-1 (1.5 vs 0.9) — "getting the prompt right seems to be crucial for good performance" on generation tasks. Second, SuperGLUE is much less sensitive to prompt choice (all prompts achieve 38.5–38.7 vs 38.3 with no prompt), suggesting that discriminative tasks benefit less from explicit mode selection because the bidirectional R-denoising mode is already the default appropriate behavior. The ablation validates that mode switching is not cosmetic — the model genuinely learns mode-specific computation that can be toggled.

Mixture-of-Denoisers composition (Table 6, Section 4.5): This is the most detailed ablation, systematically varying span lengths, corruption rates, and S-denoising percentages across 12 configurations (labeled A through L). Key findings include:

X-denoising is complementarily effective but insufficient alone (Var-H vs Var-D, F): A mixture using only long spans (Var-H: µ = 8, 64 at r = 15%, no short spans, 25% S-denoising) achieves 72.5 SuperGLUE and 10.9 one-shot XSUM vs 73.1 and 10.4 for the best overall configuration (Var-E). The supervised SuperGLUE performance is actually lower than Var-D (71.0) despite using longer spans — suggesting that the combination of short and long spans, not long spans alone, drives the benefit. The authors note this "corroborates the result from Raffel et al. (2019) that shows that a 50% corruption rate does not perform well" when used in isolation.
Small amounts of S-denoising (~20%) are preferred (Var-I, J vs Var-D, F): Scaling S-denoising to 50% of the mixture (Var-I: 71.2 SG, 11.7 one-shot XS; Var-J: 71.3 SG, 11.8 one-shot XS) reduces supervised SuperGLUE by ~2 points compared to the ~14% S-denoising in Var-F (70.6 SG — wait, Var-F has 70.6, which is lower than Var-I and J's 71.2–71.3 on SG. Let me re-read carefully. Var-F: µ = 3, 8, 64 at r = 15, 50, SD = 17% — this means 1/6 = 16.7% S-denoising. Var-F achieves 70.6 SG, 11.5 one-shot XS. Var-I: same spans and rates but SD = 50% — achieves 71.2 SG, 11.7 one-shot XS. So 50% S-denoising actually improves SuperGLUE marginally (71.2 vs 70.6) while one-shot XSUM is comparable (11.7 vs 11.5). Var-J: µ = 3, 8, 64 at r = 15, 50, SD = 50% — achieves 71.3 SG, 11.8 one-shot XS. The statement that 50% S-denoising "generally hurts performance" needs contextualization. Looking at overall patterns: Var-D (11% SD) achieves 71.0 SG and 10.6 one-shot XS. Var-E (11% SD with µ = 3, 8, 32, 64) achieves 73.1 SG and 10.4 one-shot XS — substantially better SG. So the highest SG comes from Var-E at 11% SD, not the 50% variants. The authors' claim is that lower SD percentages allow better supervised performance while maintaining one-shot gains. Var-K and Var-L (0% SD) achieve 73.7 and 70.1 SG respectively — mixed supervised results — but 2.6 and 7.3 one-shot XS — dramatic degradations. The conclusion that "S-denoisers are necessary but only small amounts of S-denoisers (≈20%) are preferred" is supported by the pattern where 0% SD fails catastrophically on one-shot tasks while 11–17% SD provides the best balance.
Long spans improve one-shot performance substantially (Var-C vs Var-D): Var-C (µ = 3, 8, 12; max span 12) achieves 38.6 one-shot SG and 4.1 one-shot XS. Var-D (µ = 3, 8, 12, 32; max span 32) achieves 42.7 (+4.1 points) and 10.6 (+6.5 points) respectively. The addition of µ = 32 spans more than doubles one-shot XSUM performance while leaving supervised SG essentially unchanged (71.9 → 71.0). This cleanly isolates the effect of long spans on in-context generation capability.
Zero S-denoising catastrophically hurts one-shot generation (Var-K, L): Var-K (no S-denoising, similar to Var-C's spans/rates but SD = 0%) achieves 73.7 SG (excellent) but only 2.6 one-shot XS — a 37% reduction from Var-C's 4.1 (which had 14% SD) and dramatically worse than the 10.4–11.5 range achieved by configurations with S-denoising. Var-L (no S-denoising, similar to Var-F) achieves 70.1 SG but 7.3 one-shot XS — better than Var-K but still substantially below the 11.5 of Var-F with 17% SD. This is the clearest ablation result in the entire paper: S-denoising is not merely helpful for one-shot generation — it is essential. Without it, the model lacks the sequential generation capability that one-shot tasks demand, regardless of how good its span corruption skills are.

Scaling behavior at moderate scale (Table 7, Section 4.6): When scaled to ~1B parameters (encoder-decoder) / 0.5B (decoder-only) and 0.5T tokens, the relative patterns persist. UL2 encoder-decoder achieves 83.3 SuperGLUE vs T5's 84.7 — a 1.4-point deficit, the first instance where UL2 underperforms T5 on this metric. However, UL2 dramatically outperforms on one-shot tasks: 45.4 vs 29.4 on SG (+54%), 15.4 vs 8.9 on XSUM Rouge-1 (+73%), and 7.8 vs 1.4 on one-shot TOTTO (+5.6×). The paper notes that "UL2 drops the SuperGLUE suite against T5 (1B)" but compensates with generation gains — an early indication that the universality trade-off becomes more pronounced at scale, with supervised NLU potentially requiring dedicated optimization.

Continuous fine-tuning and checkpoint selection (Section 5.2.1): The paper's 20B results use a continuous fine-tuning protocol where downstream tasks are fine-tuned from multiple pretraining checkpoints and the best result is reported. The authors are transparent about this: "after each Nk steps of pretraining, we finetune on each downstream task and record its results... we stop finetuning on a task once it has reached sota to save compute." This protocol risks overfitting to the validation sets through checkpoint selection, which could inflate the reported SOTA numbers. The release of three checkpoints (1.87M, 2.05M, 2.65M steps) provides a partial mitigation — practitioners can evaluate on held-out data — but the paper does not provide single-checkpoint results for the supervised tasks, making it difficult to quantify how much checkpoint selection contributes to the reported performance.

Critical Assessment

Claim 1 (from Executive Summary): UL2's MoD + mode switching produces a model that is universally effective across diverse tasks, outperforming T5-like models by +43.6% and GPT-like models by +76.1% overall gain across 9 diverse setups.

This claim is well-supported by the ablative experiments (Tables 2–4), with specific and replicable numbers. However, the following qualifications apply:

The "overall gain" metric is normalized but has limitations. The percentage-gain aggregation equally weights all 9 tasks regardless of their practical importance or the absolute magnitude of improvement. A +363% gain on one-shot SGD (from a low base of 1.43 ROUGE-L for T5 to 6.63 for UL2 — a 5.2-point absolute improvement) counts equally alongside a +1.5% gain on SuperGLUE (72.00 to 73.10 — a 1.1-point absolute improvement). The headline numbers (+43.6%, +76.1%) are heavily influenced by tasks where the baseline was near-zero and small absolute gains produce enormous percentages. This doesn't invalidate the claim — UL2 genuinely improves on these tasks — but readers should understand that the normalized percentage gain metric amplifies improvements on tasks where the baseline was weakest. The complementary perspective is that UL2's absolute gains on strong-baseline tasks (SuperGLUE) are modest (+1.1 points), while its absolute gains on weak-baseline tasks (one-shot generation) are also modest in absolute terms (+5.2 ROUGE-L on one-shot XSUM). The metric choice makes these look like enormous relative improvements when they are meaningful but not transformative absolute improvements.

The 9-task suite, while diverse, is small. Nine tasks provide reasonable coverage but are insufficient to confidently claim "universality." The 20B experiments expand to ~50 tasks, which substantially strengthens the claim, but the +43.6% / +76.1% headline numbers come from the 9-task ablation, not the 50-task evaluation. The 20B SOTA results (Table 8) use per-task fine-tuning and checkpoint selection, making them a different type of evidence than the controlled ablation comparison.

The compute-matching between architectures favors encoder-decoders. At equal compute, encoder-decoder models get ~2× the parameters of decoder-only models. This means the strong performance of UL2 encoder-decoder relative to UL2 decoder is partially attributable to parameter count, not purely the objective. The paper is explicit about this (Section 4.3.1: "this may slightly favor encoder-decoders since this can be interpreted as a form of model sparsity"), but the headline comparison against T5 (also an encoder-decoder) is fair since both use the same architecture at the same parameter count. Against GPT-like baselines, the comparison is less clean because the architectures differ.

Claim 2 (from Executive Summary): UL2 20B achieves SOTA on approximately 50 supervised NLP tasks and demonstrates emergent chain-of-thought reasoning at a moderate scale.

The SOTA claim is supported by Table 8, but several caveats temper it:

The SOTA baseline is the "best published" result as of April 2022. The paper notes this explicitly (Section 5.2.2): "results and comparisons are accurate as of 15th April 2022 where we stopped experiments to focus on polishing this paper." By the time of publication (February 2023 on arXiv v3), some of these SOTA claims may have been superseded. The paper acknowledges this for Scrolls: "we later realized... that there have been new results on Scrolls benchmark using a model using 16k sequence lengths." The fast-moving nature of benchmark leaderboards means the SOTA claim is time-stamped and should not be interpreted as a permanent achievement.

Continuous fine-tuning with checkpoint selection inflates SOTA claims. As discussed above, selecting the best checkpoint per task introduces a multiple-comparisons problem: if you test 10 checkpoints on 50 tasks, the probability of finding a "SOTA" result by chance on at least some tasks is non-trivial, even if the model is no better than the baseline. The paper does not apply Bonferroni correction or report single-checkpoint results for supervised tasks, making it difficult to assess how much of the SOTA achievement is due to objective quality vs. checkpoint optimization. For zero-shot and few-shot results, single-checkpoint numbers are reported alongside best-checkpoint numbers (e.g., UL2 20B zero-shot SuperGLUE: 63.0 single vs 65.2 best; FLAN-UL2 MMLU: 55.6 best for both tasks vs 55.1/58.1 individual task best), providing partial transparency.

The SOTA comparisons are against published results, not against rerun baselines. When UL2 achieves 21.9 ROUGE-2 on CNN/DM vs. SOTA 21.7, the SOTA number comes from a different paper using potentially different evaluation protocols, preprocessing, or metric computation. Differences of 0.2 ROUGE-2 could easily fall within evaluation noise. The paper does not rerun baselines under controlled conditions for the 20B experiments, unlike the ablative study where all models were trained identically. This is understandable given the scale, but it means the SOTA claims are weaker than the controlled ablation claims.

The chain-of-thought claim is genuinely novel but limited. UL2 20B's CoT performance (Table 12: 15.3 average without calculator, 27.2 with calculator) demonstrates that CoT can work at 20B scale — a meaningful finding given prior work's emphasis on scale. However, the absolute performance is low (GSM8K: 10.2% with CoT + SC vs. PaLM 540B's 56.9%), and the paper does not provide the crucial control experiment: would a T5-XXL or GPT-3 13B, when equipped with the same CoT prompts and self-consistency, show similar improvements? Without this control, we cannot attribute the CoT capability specifically to UL2's objective rather than to general properties of 20B-scale Transformers that were simply not tested in prior CoT work (which focused on larger models). The authors' hypothesis that "the mixture of denoisers may contribute" is plausible but untested.

Claim 3 (implicit): UL2's objective design can partially substitute for scale, making 20B models competitive with much larger models.

This claim is supported by specific comparisons (zero-shot SuperGLUE: UL2 20B > GPT-3 175B; FLAN-UL2 20B ≈ FLAN-PaLM 62B) but has important boundary conditions:

The substitution is task-dependent and asymmetric. UL2 20B's advantage over larger models is most pronounced on generation and few-shot tasks where the baseline models (T5, GPT-3) were weak. On pure NLU fine-tuning (SuperGLUE Table 9), UL2 20B (90.7) underperforms ST-MoE-32B (93.2) and PaLM 540B (92.6), and only marginally beats T5-11B (89.9). This means objective design compensates for model size primarily on tasks where the baseline objective was mismatched to the task, not universally across all tasks.

The "14× larger model" comparison from the example paper is not present here. The FLOPs-matched comparison framework used in the prior paper example does not appear in UL2. The paper claims competitiveness with larger models based on parameter-count comparisons (20B vs. 62B, 175B, 540B), but these comparisons do not control for total training FLOPs. UL2 20B trained on 1T tokens uses less total compute than PaLM 540B trained on 780B tokens (roughly 27× fewer parameters means lower per-token cost, but total cost depends on data volume and model size jointly). Without a FLOPs-matched comparison, we cannot determine whether UL2's objective provides better "FLOPs efficiency" or simply demonstrates that a model trained with less total compute can sometimes outperform a model trained with more compute on specific tasks — which is unsurprising.

The C4 corpus limitation works against UL2 in some comparisons but is not controlled. The paper notes that UL2 was trained on C4, while PaLM and GLaM used "specially curated datasets" — meaning UL2 is at a data quality disadvantage in those comparisons. This makes UL2's competitiveness more impressive (it's winning despite worse data), but it also confounds the analysis: if UL2 were trained on the same curated data as PaLM, would it improve further, or is C4 actually beneficial for the MoD objective? We cannot tell without controlled experiments.

Missing experiments that would strengthen the paper:

A dedicated scaling study comparing UL2 and T5 at multiple model sizes (e.g., 100M, 300M, 1B, 3B) with FLOPs-matched training. This would directly test whether UL2's benefits persist, diminish, or grow with scale, and would enable extrapolation to even larger models. The paper's single scaling comparison (Table 7, 1B scale) shows UL2 losing to T5 on SuperGLUE — an early warning signal that the universality advantage may narrow at scale for NLU tasks. A multi-scale study would characterize this trade-off quantitatively.
Ablation of the paradigm token mechanism vs. simply mixing objectives without tokens. The paper demonstrates that paradigm tokens affect performance (Table 5), but does not compare MoD with tokens against MoD without tokens (where the model would see all 7 denoiser configurations but no explicit mode signal). This ablation would isolate whether mode switching provides benefits beyond the mixture composition itself. If MoD without tokens performs similarly to MoD with tokens, then the innovation is purely in the denoiser composition, not in the mode-switching mechanism. The current results support that tokens matter for generation tasks (XSUM shows 48% gap between best and worst token), but don't show whether having any token is better than having no token.
Comparison against a rerun PaLM or GPT-3 at matched scale. The zero-shot comparisons (Table 10) use published numbers from different papers with potentially different evaluation protocols. A controlled experiment training a 20B causal LM on the same C4 data with the same compute budget would provide a cleaner comparison of objective vs. scale.
Single-checkpoint results for all supervised SOTA tasks. The paper reports SOTA based on per-task checkpoint selection. Releasing aggregate results from a single fixed checkpoint across all 50 tasks would provide a clearer picture of the model's genuine universal capability and the extent to which checkpoint selection contributes to the SOTA claims.
Statistical significance testing or confidence intervals. No confidence intervals, standard deviations, or significance tests are reported anywhere in the paper. For the ablative experiments (Table 2), the raw scores are point estimates from single training runs. Given the known variability in deep learning training (random seeds, data order), some of the smaller gaps (e.g., UL2 decoder 65.50 vs. SCLM decoder 63.36 on SuperGLUE) could be within noise. The paper would be strengthened by reporting run-to-run variance or at minimum acknowledging the limitation.

Overall, the experiments provide strong support for the claim that carefully designed objective mixtures outperform single-objective baselines across a diverse task suite, with particularly dramatic gains on tasks that require capabilities mismatched to the baseline objective. The evidence for "universality" is robust at the 9-task ablation level and suggestive (though less controlled) at the 50-task 20B level. The evidence for "substituting for scale" is real but conditional — the substitution works primarily on tasks where standard objectives are poorly matched, not uniformly across all task types. The paper's transparency about limitations (C4 data disadvantage, continuous fine-tuning, architecture trade-offs) strengthens credibility, but several missing experiments would be needed to fully characterize the scaling behavior and isolate the contributions of MoD composition vs. mode switching.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted For in the Headline Efficiency Gains

The assumption or constraint. The compute-optimal framework in UL2 depends critically on knowing which downstream task paradigm (R, S, or X) the model should operate under. The paper introduces mode switching via paradigm tokens ([R], [S], [X]) precisely to address this — but selecting the right token for a given downstream task currently relies on empirical trial-and-error or heuristic matching. The paper provides only limited guidance: for one-shot XSUM summarization, [R] is best (Rouge-L 11.1 vs 7.6–10.8 for other tokens, Table 5); for one-shot SuperGLUE, the choice matters less (all tokens in 38.5–38.7 range vs 38.3 with no token). The zero-shot SuperGLUE experiments (Section 5.2.5) use a per-task optimal prompt selection reported in Appendix Table 17 — revealing that different SuperGLUE sub-tasks prefer different tokens: BoolQ uses [S2S] (S-denoising), CB uses [NLU] (R-denoising), RTE uses [S2S]. There is no principled method described for selecting the paradigm token for a new downstream task without running an ablation over all three tokens.

The consequence. In a production deployment, a practitioner faced with a novel task must either (a) run a 3-way ablation over [R], [S], and [X] tokens — effectively tripling the fine-tuning or evaluation cost — or (b) rely on heuristic mapping (e.g., "generation tasks use [S], understanding tasks use [R]") which the paper's own results show is unreliable: XSUM, a generation task, performs best with [R] (the "understanding" token), contradicting the intuitive heuristic. The 48% relative gap between best and worst token on XSUM Rouge-1 (Table 5) means that choosing the wrong token can cause substantial performance degradation, making this selection problem practically consequential. The cost of token selection — whether through ablation or trial-and-error — is not amortized into any of the reported performance numbers, meaning the headline gains over T5 and GPT-like models assume the token selection problem is already solved.

What evidence exists in the paper. Table 5 explicitly demonstrates the sensitivity of generation performance to paradigm token choice (48% gap on XSUM Rouge-1 between [R] and [X]). Appendix Table 17 shows that the optimal zero-shot SuperGLUE prompt varies per sub-task (BoolQ → [S2S], CB → [NLU], RTE → [S2S], ReCoRD → [S2S], WiC → [S2S], WSC → [S2S], COPA → [NLU], MultiRC → [S2S]) — 6 of 8 tasks prefer the S-denoising token, but 2 prefer R-denoising, and none prefer X-denoising. The paper does not provide an experiment evaluating how much performance degrades when using a single fixed token across all SuperGLUE tasks, which would be the realistic deployment scenario for a practitioner who cannot run per-task token ablation.

Mitigation status. The paper does not attempt to solve the token selection problem — it treats it as a design choice left to the practitioner. Section 3.1.3 states that "for fine-tuning and downstream few-shot learning, to trigger the model to learn better solutions, we also add a paradigm token with respect to the setups and requirements of the downstream task," but provides no automated method for determining which token matches "the setups and requirements." The released checkpoints (Section 9.1) include the mapping [NLG] = X-denoising, [NLU] = R-denoising, [S2S] = S-denoising, and practitioners are left to determine which to use. No future work is suggested on automated token selection, though the finding that task-specific optimal tokens exist implies that learning a token-selection policy (perhaps from task metadata or few-shot performance) is a natural extension.

Limitation 2: Single Dataset (C4) and Single Model Family for All Ablations

The assumption or constraint. Every experiment in the paper — from the 9-task ablative study through the 20B scaling experiments — uses models pre-trained exclusively on the C4 corpus and evaluated primarily within the T5-derived architecture family (encoder-decoder or decoder-only Transformers with T5-style relative attention and GLU layers). The paper explicitly acknowledges the C4 limitation (Section 5.2.5): "UL20B remains competitive at a compute-matched setup despite only training on C4 corpus which is known to be less effective than specially curated datasets used in PaLM and GLaM." However, there is no experiment that tests UL2 pre-training on an alternative corpus (e.g., The Pile, a web-text + books + code mixture, or multilingual data) to determine whether the MoD benefits are specific to C4's characteristics or generalize across data distributions. Similarly, all models use the T5 English 32K sentencepiece tokenizer (Section 4.2.3), and no experiment varies the tokenization scheme to test whether UL2's gains depend on subword granularity.

The consequence. The paper's central claim — that MoD + mode switching produces "universally effective" pre-trained models — is supported only for models trained on a single English-language corpus (C4, which is filtered web text) using a single tokenizer and evaluated on English-language benchmarks. Several aspects of the findings could be C4-specific:

C4's relative cleanliness and moderate domain diversity may make the MoD objective easier to optimize than it would be on noisier or more diverse corpora. X-denoising with 50% corruption rate and 64-token spans might be substantially harder on code-heavy or multilingual data where long-range dependencies cross syntactic boundaries differently.
The T5 tokenizer's 32K subword vocabulary produces relatively short token sequences for English text, making µ=64 spans correspond to roughly 2–4 sentences. With a character-level or byte-level tokenizer (as in ByT5, Xue et al., 2021), the same µ=64 would cover much less text, potentially changing the difficulty gradient across denoisers.
The 9-task ablation suite is entirely English-language, and the 50-task 20B evaluation includes only English benchmarks (with the partial exception of WMT19 German-to-English translation — which the paper notes UL2 did not have German pretraining data for, Section 5.2.3 footnote). The paper does not demonstrate that UL2's universality extends to non-English languages or cross-lingual transfer.

What evidence exists in the paper. The limitation is primarily an absence of evidence rather than contradictory evidence. The paper's own WMT19 result (GENIE human evaluation: 67 for UL2 vs 71 SOTA, Table 8) demonstrates that UL2 can perform reasonably on translation despite no German pretraining, but the 4-point deficit also suggests that MoD pre-training on English C4 does not magically confer cross-lingual capability. The paper provides no results on multilingual benchmarks (e.g., XTREME, TyDi QA) or code generation tasks, leaving the scope of "universality" bounded to English NLP.

Mitigation status. The paper makes no claim of multilingual universality — the title and abstract refer to "NLP tasks" without specifying language, but the experimental scope is explicitly English. The authors' decision to use C4 is deliberate for comparability with T5 (Section 4.2.3: "We pre-train all models... using the C4 corpus"), which is a valid methodological choice for controlled ablation but limits the generality of the conclusions. The paper does not frame this as a limitation to be addressed in future work, beyond the implicit acknowledgment that PaLM and GLaM's curated data provides advantages that UL2 does not benefit from. A natural extension — pre-training UL2 on the same curated mixture as PaLM — is not discussed.

Limitation 3: The 20B Scaling Results Use Per-Task Checkpoint Selection Without Statistical Controls

The assumption or constraint. The 20B SOTA results (Table 8, covering ~50 tasks) are obtained through a continuous fine-tuning protocol described in Section 5.2.1: "after each Nk steps of pretraining, we finetune on each downstream task and record its results... we stop finetuning on a task once it has reached sota to save compute." The model was pre-trained for 2 million steps, and fine-tuning runs were initiated from multiple intermediate checkpoints (the paper releases three: at 1.87M, 2.05M, and 2.65M steps; Section 9.1). For each downstream task, the best result across all evaluated checkpoints is reported as the UL2 score in Table 8. This protocol introduces a multiple-comparisons problem: if you evaluate 5–10 checkpoints on 50 tasks, the expected maximum performance across checkpoints for each task will exceed the expected performance of any single fixed checkpoint, even if all checkpoints are drawn from the same distribution.

The consequence. The reported SOTA numbers in Table 8 are upper bounds on what a practitioner would achieve by downloading the released checkpoint and fine-tuning once. The amount of inflation depends on the variance of fine-tuning performance across checkpoints, which the paper does not measure. If checkpoint-to-checkpoint variance is small (e.g., 0.5% relative), the inflation is negligible; if variance is large (e.g., 2–3% relative on some tasks), the SOTA claims for tasks where UL2's margin is thin could be entirely attributable to checkpoint selection rather than genuine model superiority. For example:

CNN/DM: UL2 21.7 vs SOTA 21.7 ROUGE-2 — a margin of 0.0 (tie). Even small checkpoint variance could flip this result.
MultiNews: UL2 21.7 vs SOTA 21.1 — a 0.6-point margin.
PIQA: UL2 90.7 vs SOTA 90.1 — a 0.6-point margin.
CosmosQA: UL2 91.6 vs SOTA 91.8 — UL2 loses by 0.2, despite checkpoint optimization.

For tasks where UL2's margin is large (e.g., Schema-Guided Dialog: 44.1 vs 36.8, a 7.3-point gain; QuALITY: 45.8 vs 26.0, a 19.8-point gain; DocNLI: 88.2 vs 76.9, an 11.3-point gain), checkpoint selection is very unlikely to be the sole explanation — these gains are too large to be artifacts of variance.

The paper's zero-shot and few-shot results provide partial transparency: for zero-shot SuperGLUE (Table 10), they report both "UL2 20B (single ckpt)" at 63.0 and "UL2 20B (best)" at 65.2. The 2.2-point gap between single and best checkpoint is a direct measurement of the checkpoint selection benefit for that benchmark. Similar single-checkpoint baselines are not provided for the supervised SOTA tasks in Table 8, making it impossible to quantify the inflation for those results.

What evidence exists in the paper. The paper is explicit about the protocol (Section 5.2.1: "we stop finetuning on a task once it has reached sota to save compute"), which is transparency about the method rather than evidence of the limitation's magnitude. The zero-shot SuperGLUE single-vs-best gap (2.2 points) provides an existence proof that checkpoint selection matters, but the gap may differ across tasks — tasks that benefit from later pretraining checkpoints (e.g., those requiring more world knowledge) might show larger gaps than tasks that plateau early.

Mitigation status. The paper does not attempt to correct for multiple comparisons (e.g., via Bonferroni correction or by reporting aggregate single-checkpoint results across all tasks). The release of three checkpoints (Section 9.1) enables external researchers to evaluate single-checkpoint performance, but the paper itself does not provide this analysis. The authors describe this as a practical constraint: "not much thought was put into the exact parameter count of this model, i.e., we were training a 20B model already for some time and decided to see it to convergence... we did not specifically control or put in place any mitigation strategies such as occasional restarts." The loss spikes observed during 20B training (Section 5) further suggest that checkpoint quality may vary substantially, potentially increasing the gap between best-checkpoint and typical-checkpoint performance. The paper frames the 20B results as "a proof-of-concept resembling a hint of what UL2 can do at a relatively larger scale," which partially tempers the SOTA claims but doesn't resolve the methodological concern.

Limitation 4: No Direct Evidence That Mode Switching Improves Over Simple Objective Mixing Without Tokens

The assumption or constraint. Mode switching is presented as one of UL2's two key innovations (alongside MoD): "we introduce mode switching, a new concept that associates pre-training tasks with dedicated sentinel tokens and allows dynamic mode switching via discrete prompting" (Section 1). The intuition is that paradigm tokens enable the model to develop mode-specific computation that can be cleanly activated downstream. However, the paper never compares MoD with paradigm tokens against MoD without paradigm tokens — i.e., a model trained on the same 7-denoising mixture but without [R], [S], [X] tokens prepended to the input. Such a model would still see diverse denoising configurations during pre-training and would have to infer the appropriate behavior from the input pattern alone, but would not receive an explicit mode signal. This ablation is necessary to attribute the benefits of UL2 specifically to the interaction between MoD and mode switching, rather than to MoD composition alone.

The consequence. The paper's claim that mode switching is a distinct innovation rests on incomplete evidence. The existing ablations demonstrate that:

Paradigm token choice matters for downstream performance (Table 5): using the wrong token hurts. This shows that the model has learned token-conditioned behavior, but does not show that having tokens at all is better than having no tokens.
MoD composition matters (Table 6): varying the denoiser mixture changes performance. This shows that the mixture design is important, but does not show that tokens are necessary to realize the mixture's benefits.

Without the no-tokens baseline, it is possible that a model trained on MoD without paradigm tokens would perform comparably to UL2 — the tokens might simply provide a convenient interface for controlling behavior that the model could learn to infer from context. Alternatively, the tokens might be essential for preventing mode interference (where training on multiple objectives without explicit task identification causes the model to average across behaviors). The paper's current evidence cannot distinguish these possibilities.

What evidence exists in the paper. Table 5 is the only ablation that directly tests mode switching, and it varies which token is used (or whether any token is used) during evaluation of a model that was trained with tokens. The "UL2 / None" row in Table 5 (no paradigm token at evaluation time) achieves 13.2/1.4/10.8 Rouge-1/2/L on one-shot XSUM vs. 13.5/1.5/11.1 with the [R] token — a small gap, suggesting that the model can partially infer the appropriate mode even without the explicit token. However, this is a model that was pre-trained with tokens; it may have learned to rely on them. The missing comparison is a model pre-trained entirely without paradigm tokens but with the same MoD mixture.

Mitigation status. The paper does not acknowledge this missing ablation as a gap. The mode-switching concept is presented as a key contribution alongside MoD (Section 3.1.3), but the empirical support establishes only that token selection is a meaningful knob, not that the knob itself is necessary. Future work that pre-trains MoD without tokens and compares against UL2 with tokens would directly test whether mode switching provides benefits beyond mixture composition. The paper's finding that SuperGLUE is relatively insensitive to token choice (Table 5: 38.3–38.7 across all tokens) while XSUM is highly sensitive (Rouge-1: 0.9 to 1.5) suggests that the value of explicit mode signaling may be task-dependent — crucial for generation, less important for understanding — but this is a post-hoc observation, not a controlled experimental result.

Limitation 5: The Universality Gains Are Asymmetric — Large on Generation Tasks, Modest on NLU Tasks

The assumption or constraint. The paper's framing emphasizes "universal effectiveness across datasets and setups" (abstract) and "a single pretrained model that can perform well on many types of tasks" (Section 1). The headline result — UL2 outperforms T5 by +43.6% overall normalized gain — aggregates improvements across 9 tasks that include both supervised NLU and one-shot generation (Table 3). However, this aggregate metric masks extreme asymmetry in where the gains occur. On the tasks where T5 was already strong (supervised SuperGLUE), UL2 encoder-decoder improves by only +1.5% (from 72.00 to 73.10). On tasks where T5 was weak (one-shot generation), the gains are orders of magnitude larger: +363% on one-shot SGD, +210% on one-shot TOTTO, +53.6% on one-shot XSUM. The +43.6% headline number is heavily driven by tasks where the baseline was near-floor, making large percentage gains achievable with modest absolute improvements.

The consequence. A practitioner whose primary use case is supervised fine-tuning on NLU tasks (classification, entailment, reading comprehension) will see only marginal benefits from switching from T5 to UL2 — a 1.1-point SuperGLUE improvement (Table 2: 72.00 → 73.10) that may not justify the increased complexity of the MoD training pipeline. Moreover, when the UL2 encoder-decoder is scaled to 1B parameters (Table 7), it actually underperforms T5 on SuperGLUE (83.3 vs 84.7) — a -1.4-point deficit — suggesting that the NLU benefit may not persist at larger scales. The universality claim is thus qualified: UL2 dramatically improves performance on tasks that T5 was bad at, but provides negligible improvement (and possibly degradation at scale) on tasks that T5 was already good at. This is a Pareto improvement over T5 (it gets better on some axes without getting much worse on others, at the ablation scale), but it is not a uniform improvement across all task types — the gains are concentrated in the generation and few-shot regime.

The consequence for GPT-like model users is different: since GPT-like models are weak at NLU, UL2 provides large improvements there (+17.4% on supervised SuperGLUE, Table 4), making it a clear win for practitioners currently using decoder-only causal LMs who also need NLU capabilities. But for practitioners already using T5 for NLU, the case for switching is weaker — it depends on how much they value the added generation and few-shot capabilities.

What evidence exists in the paper. The asymmetry is visible directly in Table 2's raw scores and Table 3's per-task percentage gains. The scaling comparison in Table 7 (1B parameters) provides the critical data point that the SuperGLUE gap reverses: UL2 83.3 vs T5 84.7. The paper acknowledges this: "At large scale, we find that the proposed UL2 encoder-decoder model is still competitive. A key difference now is that UL2 drops the SuperGLUE suite against T5 (1B). However, this is compensated by not only out-performing on 7 out of 8 tasks but also improving performance by 2-4 times on one-shot evaluation" (Section 4.6). This framing — that the NLU deficit is "compensated" by generation gains — implicitly acknowledges the trade-off but treats it as acceptable rather than as a limitation. For a practitioner who only cares about NLU, the trade-off is not compensated; it is a net loss.

Mitigation status. The paper does not frame this asymmetry as a limitation — it presents universality as a success. The 20B results partially address the concern by showing that UL2 20B still achieves competitive SuperGLUE (90.7, outperforming T5-11B's 89.9 in Table 9), suggesting the 1B-scale reversal may not extrapolate linearly. However, the 20B comparison is against T5-11B, not a hypothetical T5-20B (which the paper does not train), so we cannot determine whether UL2 20B would outperform a compute-matched T5 20B on pure NLU. The ST-MoE-32B result (93.2 SuperGLUE, Table 9) — a model trained with pure span corruption — demonstrates that dedicating the same architecture to a single objective still yields better NLU performance, confirming that UL2's universality comes at some NLU cost compared to specialized models. The paper does not propose a method for recovering this NLU gap while retaining generation capabilities, such as adaptive objective weighting during pre-training or staged training (more R-denoising early, more S-denoising late).

Limitation 6: No Combination of MoD with Search, Revision, or Inference-Time Compute Scaling

The assumption or constraint. UL2 is a pre-training recipe — it defines how the model is trained, not how it is used at inference time. All downstream evaluations use standard fine-tuning (supervised), standard prompt-based few-shot learning (one exemplar + query), or standard zero-shot prompting. The paper does not explore whether UL2's mixture-of-denoising pre-training interacts with inference-time techniques such as beam search, best-of-N sampling, verifier-guided selection, iterative revision, or chain-of-thought with self-consistency (the CoT + SC experiments in Tables 12 and 13 use a single approach without comparing against alternative inference strategies). This means the reported performance represents a lower bound on what UL2 could achieve if combined with inference-time compute scaling — but also that the paper cannot characterize how MoD pre-training changes the effectiveness of these techniques.

The consequence. Two important practical questions are left unanswered:

Does UL2's multi-paradigm pre-training make the model more or less amenable to inference-time search? A model trained on diverse denoisers might have a broader proposal distribution (generating more diverse candidates in best-of-N sampling) because it has been exposed to multiple generation modes. Conversely, the mode-switching mechanism might constrain diversity when a specific paradigm token is used, reducing the benefit of search. Without experiments, practitioners cannot determine whether UL2's gains are complementary to or redundant with inference-time compute scaling.
Can paradigm token selection be integrated with inference-time strategies? For example, a verifier could score candidate outputs generated under different paradigm tokens ([R], [S], [X]) and select the best — effectively automating the token selection problem (Limitation 1) by treating it as an inference-time search over modes. The paper does not explore this direction.

The chain-of-thought experiments (Section 5.2.7) provide a partial exception: they demonstrate that self-consistency (a form of inference-time search via majority voting over multiple CoT reasoning paths) substantially improves UL2's performance (e.g., arithmetic average from 15.3 to 36.0, Table 12). However, this is a single technique applied to a specific task type, not a systematic study of how MoD pre-training interacts with inference-time compute allocation.

What evidence exists in the paper. The paper provides no experiments that vary inference-time compute strategies for a fixed UL2 model (e.g., comparing best-of-1 vs best-of-8, or greedy vs beam search, for UL2 vs T5 on the same tasks). The CoT + SC results (Tables 12, 13) demonstrate that UL2 can benefit from inference-time search, but there is no comparison showing whether UL2 benefits more or less than a T5 or GPT model would from the same technique. The FLAN instruction tuning experiments (Section 5.3) use standard prompting without search, leaving the interaction between MoD pre-training, instruction tuning, and inference-time strategies unexplored.

Mitigation status. The paper does not address this as a limitation — its scope is explicitly pre-training objective design, not inference-time optimization. The omission is understandable given the paper's focus, but it limits the practical guidance for deployment: a practitioner adopting UL2 does not know whether they should also invest in sophisticated inference strategies or whether the MoD objective already captures most of the available gains. The FLAN-UL2 CoT results (Table 16: CoT + SC improves MMLU-CoT from 53.9 to 57.1 for a specific run) suggest that combining UL2 with inference-time search is beneficial, but the paper explicitly leaves this exploration incomplete: "We did not experiment further since this increases the search space to a point where it was more time consuming than we would have liked. We leave any future experiments as an exercise for the reader" (Section 5.3.2). This candid admission confirms that the interaction between MoD and inference-time compute is an open question.

7. Implications and Future Directions

How This Work Changes the Landscape

UL2 instigates a reframing of pre-training objective design rather than a paradigm shift or a new architectural contribution. Prior to this work, the dominant mental model in NLP treated pre-training objectives as discrete, architecture-bound choices: you picked T5-style span corruption for NLU, GPT-style causal language modeling for generation, and the architecture (encoder-decoder vs. decoder-only) followed from that choice. UL2 systematically dismantles this conflation along two dimensions.

First, the unified denoising perspective (Section 3.1.1) demonstrates that causal language modeling, prefix language modeling, and span corruption are points on a continuous two-dimensional manifold parameterized by mean span length $\mu$ and corruption rate $r$ , not categorically different approaches. This is a conceptual contribution that changes how researchers should think about pre-training objectives: the design space is not a discrete menu of named methods but a continuous landscape where interpolation is possible and — as the MoD ablation demonstrates (Table 6) — beneficial. Prior work tested individual objectives in isolation (T5's exploration of 50% corruption rates; Wettig et al., 2022's study of increased masking) and reached contradictory conclusions about whether extreme denoising helps. UL2 resolves this by showing that X-denoising with $\mu = 64$ , $r = 0.5$ is harmful as a standalone objective (Var-H in Table 6, confirming T5's negative result) but substantially beneficial when mixed with easier denoisers (Var-D, E, F achieve ~10.6 ROUGE-L on one-shot XSUM vs. ~6.5 for the pure T5 baseline, while matching on supervised SuperGLUE). This is an interaction effect invisible to single-objective ablation studies — a diagnostic resolution that explains why prior findings appeared contradictory and opens the door to exploring the mixture space systematically rather than testing objectives one at a time.

Second, UL2 disentangles architecture from objective in a way that, while not entirely new (Raffel et al., 2019 noted that decoder-only models could be trained with span corruption), had not been systematically demonstrated. The finding that UL2 decoder (167M parameters) outperforms T5 encoder-decoder (335M parameters) by +14.6% on the overall normalized metric (Table 3) — despite having half the parameters — is a concrete existence proof that objective design can compensate for architectural disadvantages. This reshapes the conversation around model selection: architecture becomes an efficiency consideration (throughput vs. parameter count, as discussed in Section 3.2) rather than a task-suitability determinant. The implication for practitioners is that choosing the right pre-training objective matters more than choosing the right architecture — a claim the paper supports directly but that had not been demonstrated with this level of controlled, multi-architecture evidence before.

The work also shifts the research frontier for smaller-scale models. The demonstration that UL2 20B enables chain-of-thought reasoning (Section 5.2.7) — a capability previously documented only at 100B+ scale (Wei et al., 2022b) — and achieves zero-shot SuperGLUE performance (65.2, Table 10) exceeding GPT-3 175B (61.2) suggests that objective design can partially substitute for scale in the 20B parameter regime. This does not overturn scaling laws — larger UL2 models would presumably perform even better — but it shifts the intercept of the scaling curve upward, making moderate-scale models more capable than scaling laws based on standard objectives would predict. The FLAN-UL2 results (Tables 15, 16) reinforce this: FLAN-UL2 20B approaches FLAN-PaLM 62B (a 3.1× larger model) within 0.8 points on average across MMLU/BBH with direct and CoT prompting, suggesting that the benefits of objective design compound with instruction tuning rather than being made obsolete by it.

A less obvious but important shift is that UL2 makes pre-training objective design a search problem over mixtures rather than a search problem over individual objectives. The MoD ablation (Table 6, 12 configurations) provides a partial map of the interaction space — showing, for example, that the optimal S-denoising proportion is ~15–20% (higher proportions degrade supervised NLU; lower proportions catastrophically collapse one-shot generation, as in Var-K with 0% S-denoising and 2.6 one-shot XSUM ROUGE-L) — but the paper explicitly does not claim to have found the optimal mixture. This opens the door to more systematic exploration, potentially using automated search or Bayesian optimization over the $(\mu, r, \text{mix\_weight})$ space, rather than the manual, intuition-driven design that produced UL2's specific 7-configuration blend.

What becomes less attractive as a result of this work is the practice of developing pre-training objectives in isolation and testing them against single-objective baselines. If superadditive interactions between denoisers are the norm rather than the exception — and the dramatic difference between SCLM (simple 50-50 mix, +28.3% overall gain) and UL2 (+43.6%, Table 3) suggests they are — then the standard "propose a new objective, ablate it alone" methodology systematically misses the most promising regime. Future objective-design papers will need to include mixture baselines (at minimum, mixing with standard span corruption and prefix LM) to convincingly demonstrate that a new objective contributes beyond what mixing existing ones already provides. Similarly, single-architecture evaluation becomes harder to justify: UL2's results show that the objective benefit is visible in both decoder-only and encoder-decoder architectures, but the magnitude differs, and reporting only one architecture could misrepresent the objective's contribution.

Follow-Up Research This Work Enables

Automated mixture-of-denoisers search via Bayesian optimization or evolutionary methods. The paper's MoD composition (Table 1: 2 R-denoising, 1 S-denoising, 4 X-denoising configurations) was designed through manual intuition and validated by a 12-configuration ablation (Table 6) that varied span lengths, corruption rates, and S-denoising proportions. However, the space of possible mixtures is combinatorially large — varying $\mu$ (continuous or many discrete values), $r$ (continuous), the number of configurations per paradigm, and the mixing weights — and it is unlikely that the paper's specific blend is optimal. A natural follow-up would apply Bayesian optimization or population-based training to search over the mixture space directly, using a multi-task objective (e.g., weighted average of supervised SuperGLUE and one-shot XSUM ROUGE-L) as the optimization target. The key question is whether there exist Pareto-optimal mixtures that improve on UL2's trade-off — e.g., matching UL2's generation performance while recovering the 1.4-point SuperGLUE deficit observed at 1B scale (Table 7: UL2 83.3 vs T5 84.7). The paper's result that the optimal S-denoising proportion is task-dependent (0% S-denoising maximizes SuperGLUE but destroys one-shot XSUM; 50% S-denoising underperforms on both relative to ~15%) suggests the trade-off is real but potentially optimizable. A strong follow-up would report the full Pareto frontier over mixture hyperparameters at a fixed compute budget, enabling practitioners to select a mixture based on their task distribution rather than accepting a one-size-fits-all blend.

Scaling UL2 to 100B+ parameters with a FLOPs-matched comparison against PaLM and T5. The paper's 20B results demonstrate competitiveness with larger models (GPT-3 175B on zero-shot SuperGLUE, FLAN-PaLM 62B on MMLU/BBH), but these comparisons are not FLOPs-matched — UL2 20B on 1T tokens uses less total compute than PaLM 540B on 780B tokens. The critical missing experiment is a controlled scaling study that trains UL2, T5 (span corruption), and a GPT-like causal LM at multiple scales (e.g., 1B, 10B, 100B parameters) with matched total FLOPs and evaluates on the full 9-task suite from the ablation study. This would directly test whether UL2's benefits persist, diminish, or grow with scale. The 1B-scale result (Table 7) showing UL2 losing to T5 on SuperGLUE (83.3 vs 84.7) is an early warning signal — does this trend continue, such that at 100B scale UL2 would substantially underperform T5 on NLU while dominating on generation, or does the gap stabilize? A FLOPs-matched comparison would also determine whether UL2 is genuinely more compute-efficient or simply achieves better performance on specific tasks because those tasks were mismatched to the baseline objectives. The hypothesis from the paper is that "the self-supervision objective may be intrinsically more important than the backbone architecture" (Section 4.3.1), and a multi-scale FLOPs-matched study would quantify how much more important — e.g., "UL2 at 10B FLOPs matches T5 at 30B FLOPs on NLU while exceeding it on generation" — providing a scaling law for objective choice analogous to Hoffmann et al. (2022) for model size vs. data.

Training a "mode oracle" that automatically selects the optimal paradigm token per task or per example. The mode-switching mechanism (Section 3.1.3) relies on the practitioner selecting the correct paradigm token ([R], [S], or [X]) for each downstream task — a choice that Table 5 shows can cause a 48% relative gap on XSUM Rouge-1 between best and worst token. The paper's zero-shot SuperGLUE results (Appendix Table 17) reveal that different SuperGLUE sub-tasks prefer different tokens (BoolQ → [S2S], CB → [NLU], with 6 of 8 tasks preferring [S2S]), confirming that per-task token selection is necessary for optimal performance. A natural follow-up would train a lightweight classifier — perhaps a linear probe on top of UL2's frozen encoder representations — that takes a few examples from a new task (or even the task description alone) and predicts which paradigm token will yield the best performance. The training data for this classifier already exists: the paper's own experiments evaluating all three tokens on multiple tasks (Table 5, Appendix Table 17). A strong result would show that the predicted token achieves >95% of the oracle token's performance on held-out tasks, eliminating the need for per-task token ablation. A negative result — finding that token preference is not predictable from task metadata or few-shot performance — would suggest that mode switching requires further architectural innovation (e.g., learnable per-task mode embeddings rather than hard discrete tokens) to be practically deployable.

Investigating whether MoD pre-training improves sample efficiency of downstream fine-tuning. The paper evaluates UL2 primarily on final fine-tuned performance after standard fine-tuning budgets (typically 50K–100K steps for the 20B model, Section 5.2.1), not on the rate at which performance improves during fine-tuning. A plausible hypothesis — consistent with the multi-paradigm pre-training philosophy — is that UL2 learns more transferable representations that enable faster adaptation to new tasks, because the model has already been exposed to a wider variety of input-output patterns during pre-training. A concrete experiment would compare UL2 against T5 on a few-shot fine-tuning setup across tasks: fine-tune both models on {10, 50, 100, 500, 5000} examples and measure the learning curve. If UL2 achieves higher accuracy at low data regimes (e.g., 50 examples), this would position MoD pre-training as beneficial not just for final performance but for data efficiency — a practically important property for domains with limited labeled data. The paper's one-shot and zero-shot results (Tables 10, 11) already hint at this: UL2 20B dramatically outperforms T5-XXL on one-shot XSUM (19.8 vs 10.7 ROUGE-L), suggesting better few-shot adaptation. Extending this to a systematic few-shot fine-tuning study with multiple data points per task would quantify the effect and determine whether it holds across task types.

Stress-testing UL2 on distribution shift and out-of-domain generalization. The paper evaluates UL2 on standard benchmarks with training and test sets drawn from similar distributions (C4 pre-training, then fine-tuning on in-domain labeled data for each benchmark). A critical robustness question — especially given that MoD trains on corrupted inputs that differ substantially from clean downstream inputs — is whether UL2's multi-paradigm pre-training makes it more or less robust to distribution shift. The experiment would evaluate UL2 vs. T5 vs. GPT-like models on established out-of-distribution benchmarks (e.g., WILDS, or domain-shifted variants of SuperGLUE where the test set is from a different domain than the training set). The hypothesis could go either way: X-denoising's training on heavily corrupted inputs might make the model more robust to noisy or incomplete inputs at test time (a form of accidental robustness training), or the diversity of pre-training paradigms might cause the model to rely on surface-level cues from the paradigm token rather than learning genuinely invariant representations, making it more brittle under distribution shift. A negative result (UL2 underperforms T5 on OOD generalization despite matching on in-domain) would be an important boundary condition on the universality claim, indicating that the benefits are confined to in-distribution task formats and that specialized models may still be preferable for safety-critical OOD applications.

Extending MoD to multilingual and code pre-training to test generality of the mixture composition. All of UL2's experiments use English C4 pre-training and English downstream tasks (with the partial exception of WMT19 German-to-English translation, where UL2 lacked German pre-training data and underperformed SOTA by 4 points on human evaluation, Table 8). An important generalization test is whether the specific MoD mixture that works for English (7 configurations with $\mu \in \{3, 8, 64\}$ , $r \in \{0.15, 0.25, 0.5\}$ ) transfers to multilingual pre-training (e.g., on mC4) and code pre-training (e.g., on The Stack). The span-length semantics differ across languages and modalities: a $\mu = 64$ span in English corresponds to ~2–4 sentences, but in code it might cross function boundaries or entire code blocks, and in languages with different information density per token (e.g., character-based scripts) the optimal span lengths may shift systematically. A multilingual UL2 experiment would pre-train with the same MoD mixture on mC4, evaluate on XTREME or TyDi QA, and test whether the English-derived mixture generalizes or whether language-specific or script-specific mixture tuning is necessary. A negative result (the English mixture performs poorly on certain languages or scripts) would indicate that the unified denoising perspective is universal in principle but the specific mixture parameters are language-dependent, motivating adaptive mixture strategies based on tokenizer granularity or data characteristics.

Practical Applications and Downstream Use Cases

Single-model deployment for platforms serving diverse NLP workloads. Many production NLP systems — cloud APIs, on-device assistants, enterprise search platforms — must handle a mix of task types: classification (intent detection, toxicity screening), extraction (named entity recognition, structured data parsing), generation (summarization, dialogue response), and retrieval-augmented question answering. Prior to UL2, these platforms typically deployed separate models optimized for each task category (a T5 variant for NLU, a GPT variant for generation) or accepted degraded performance on some tasks from a single model. UL2's headline result — outperforming T5 on all 9 diverse tasks in the ablation study (Table 2) and achieving SOTA on ~50 tasks at 20B scale (Table 8) — directly supports a unified deployment architecture where a single UL2 checkpoint serves all task types. The practical benefit is operational simplification: one model to maintain, monitor, update, and optimize, rather than N separate models with different architectures, tokenizers, and serving requirements. The cost of this simplification — a modest NLU performance gap vs. specialized models (e.g., UL2 20B scores 90.7 on SuperGLUE vs. ST-MoE-32B's 93.2, Table 9) — must be weighed against the operational savings, but for many deployments the 2.5-point gap on SuperGLUE is an acceptable price for eliminating the complexity of a multi-model serving infrastructure.

Cost-efficient fine-tuning for organizations with limited labeled data across diverse task types. The paper's demonstration that UL2 achieves strong one-shot and few-shot performance (Tables 10, 11) without task-specific architectural modifications or extensive prompt engineering positions it as a strong default choice for organizations that need to fine-tune models on small labeled datasets across diverse task types. The practical workflow — download the UL2 checkpoint, select the paradigm token based on the task's primary modality ([R] for understanding-heavy tasks, [S] for generation-heavy tasks, [X] for mixed), fine-tune with standard hyperparameters — is simpler than the alternative of testing multiple model architectures and objectives. The 20B model's release as an open-source T5X checkpoint (Section 9.1) makes this workflow immediately accessible to practitioners with TPU or GPU infrastructure. The specific efficiency gain can be quantified from the paper's results: on one-shot XSUM summarization, UL2 20B achieves 19.8 ROUGE-L vs. 10.7 for T5-XXL with LM adaptation (Table 11) — a ~1.85× improvement at similar model scale, translating directly to better summarization quality from the same amount of labeled data (or equivalent quality from fewer labeled examples).

Enabling chain-of-thought reasoning on moderate-scale hardware. The finding that UL2 20B successfully leverages chain-of-thought prompting with self-consistency (Tables 12, 13) — reaching 36.0 average accuracy on arithmetic reasoning benchmarks with CoT + calculator + SC, vs. 13.5 with standard prompting — makes multi-step reasoning feasible on hardware that cannot serve 100B+ models. Prior to UL2, the CoT literature (Wei et al., 2022b) had demonstrated that CoT was an emergent property of scale, with benefits only appearing for models above ~100B parameters; smaller models were actively harmed by CoT prompting. UL2 20B breaks this pattern: CoT prompting alone provides a modest gain (13.5 → 15.3), and when combined with an external calculator and self-consistency, the improvement is dramatic (13.5 → 36.0). For applications requiring multi-step reasoning on resource-constrained hardware — on-device math assistance, educational tools running on edge devices, research environments without access to large-scale compute — UL2 20B provides a publicly available model that can perform structured reasoning without requiring the infrastructure for 100B+ parameter models. The practical caveat is that the absolute performance remains modest (GSM8K: 10.2% with CoT + SC vs. PaLM 540B's 56.9%), so the use case is for applications where moderate accuracy is acceptable in exchange for local deployment, not for high-stakes reasoning tasks where accuracy requirements demand the largest available models.

Pre-training foundation for instruction-tuned models that approach larger-scale performance. The FLAN-UL2 results (Tables 15, 16) demonstrate that UL2's MoD pre-training produces a strong foundation for instruction tuning: FLAN-UL2 20B achieves 55.6 MMLU and 45.3 BBH (direct prompting), approaching FLAN-PaLM 62B (59.6 and 47.5) despite 3.1× fewer parameters, and outperforming FLAN-T5 XXL across all four CoT/direct evaluation settings. For organizations building instruction-following models — a rapidly growing application area following FLAN, T0, and InstructGPT — starting from a UL2 checkpoint rather than a T5 or GPT checkpoint provides a better performance-per-parameter trade-off, enabling smaller, cheaper-to-serve instruction-tuned models that remain competitive with larger alternatives. The specific workflow is: pre-train UL2 with MoD → apply FLAN instruction tuning (following the protocol in Chung et al., 2022, with the paper's suggested 100K-step adaptation without mode tokens, Section 5.3) → deploy. The released FLAN-UL2 checkpoint (Section 9.1) provides an off-the-shelf starting point for further domain-specific instruction tuning.