PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

ArXiv: 1912.08777

🎯 Pitch

A summarization model that masks and regenerates the most important sentences from a document during pre-training matches human-level performance on multiple benchmarks. This Gap Sentences Generation objective enables the model to beat prior work on all 12 tested datasets, and with just 1000 fine-tuning examples it can outperform fully-supervised baselines trained on 100x more data.

1. Executive Summary

This paper proposes Gap Sentences Generation (GSG), a self-supervised pre-training objective tailored specifically for abstractive text summarization, which masks whole sentences from an input document and trains a Transformer encoder-decoder to generate them as a single output sequence from the remaining text. Evaluating on 12 downstream summarization datasets spanning news, science, stories, instructions, emails, patents, and legislative bills, the authors demonstrate that selecting important sentences for masking—measured via ROUGE1-F1 between each sentence and the rest of the document—substantially outperforms random or lead-based selection, with the resulting PEGASUS model achieving state-of-the-art ROUGE scores on all 12 datasets. The best PEGASUSLARGE model (568M parameters, pre-trained on a corpus of 1.5B news-like articles) matches or exceeds prior work across the board, including a 4× improvement in low-resource settings (achieving comparable quality to a fully-supervised non-pretrained TransformerBASE with only 100–1,000 fine-tuning examples), while human evaluation establishes that model summaries are not measurably worse than reference summaries on XSum, CNN/DailyMail, and Reddit TIFU—though the approach shows that pre-training corpus domain alignment matters critically, with HugeNews benefiting news datasets and C4 benefiting informal domains.

2. Context and Motivation

The Core Problem: No Pre-training Objective Designed for Abstractive Summarization

The fundamental gap this paper addresses is deceptively simple: Transformer-based sequence-to-sequence models had revolutionized NLP through self-supervised pre-training, but no one had designed a pre-training objective specifically for abstractive text summarization. By late 2019, the field had converged on a powerful formula—pre-train large Transformer models on massive text corpora using self-supervised objectives, then fine-tune on downstream tasks—but the pre-training objectives themselves were generic, designed for broad language understanding rather than the specific demands of generating concise, coherent summaries from longer documents.

This matters because abstractive summarization imposes unique requirements on a model that generic pre-training may not adequately address. A good summarizer must: (1) identify which information in a long document is important enough to include in a summary, (2) understand the document holistically—not just individual sentences in isolation—to avoid redundancy and maintain coherence, and (3) generate fluent, condensed text that may use novel words and phrasing not present in the source (the defining characteristic of abstractive vs. extractive summarization). Pre-training objectives like BERT's masked language modeling (which randomly masks individual words) or T5's span corruption (which masks contiguous text spans) teach models about local context and token-level reconstruction, but they don't explicitly train the skill of identifying and generating important content from a longer text—which is precisely what summarization requires.

The authors frame this gap explicitly in the introduction: "pre-training objectives tailored for abstractive text summarization have not been explored." This is not a claim that prior pre-training failed at summarization—models like BART, T5, and UniLM had shown strong results—but rather that the objectives were not designed with summarization in mind, leaving open the question of whether a task-aware objective could yield better performance, faster fine-tuning, or stronger generalization.

Why This Problem Is Important: The Diverging Trajectories of Summarization Datasets and Pre-training Methods

The paper identifies two parallel trends that made this gap increasingly consequential:

First, the summarization community had been rapidly expanding the diversity of available datasets. Prior to 2018–2019, neural abstractive summarization research was heavily concentrated on two news datasets: CNN/DailyMail (Hermann et al., 2015) and Gigaword (Rush et al., 2015). But by the time of PEGASUS, new datasets had emerged covering scientific papers (arXiv, PubMed; Cohan et al., 2018), patents (BIGPATENT; Sharma et al., 2019), legislative bills (BillSum; Kornilova & Eidelman, 2019), instructional text (WikiHow; Koupaee & Wang, 2018), informal stories (Reddit TIFU; Kim et al., 2019), multi-document summaries (Multi-News; Fabbri et al., 2019), and email subject lines (AESLC; Zhang & Tetreault, 2019). These datasets vary enormously in domain, writing style, abstractiveness (measured by extractive fragment coverage and density), document length, and summary length. Critically, the authors note that "there has been little work on systematic evaluation of models across these broad settings." A pre-training objective that performed well on news might not transfer effectively to patents or informal Reddit stories, and understanding why would require evaluation across the full diversity of available tasks.

Second, the pre-training revolution had been advancing rapidly with generic objectives: BERT (Devlin et al., 2019) used masked language modeling, MASS (Song et al., 2019) reconstructed masked sentence fragments, UniLM (Dong et al., 2019) combined multiple language modeling objectives, BART (Lewis et al., 2019) used a denoising autoencoder with arbitrary text corruption, and T5 (Raffel et al., 2019) treated everything as a text-to-text problem with span corruption. All of these showed that pre-training helped summarization, but none asked: what if the pre-training objective itself mimicked summarization?

The practical importance is amplified by the low-resource summarization problem the paper explicitly highlights. The authors observe that "large-scale document-summary datasets are rare and in practice there is a mismatch between research datasets and real-world use-cases where collecting summaries is expensive." Most real-world summarization applications—internal corporate documents, niche technical domains, medical records—cannot rely on hundreds of thousands of human-written summaries for fine-tuning. A pre-training objective that more closely resembles summarization could make models dramatically more sample-efficient, needing far fewer supervised examples to reach usable quality. This is not a minor concern: collecting high-quality abstractive summaries requires skilled human annotators who must read entire documents (which may be thousands of words long), synthesize key information, and produce fluent condensed text—a far more expensive annotation task than, say, sentiment classification or named entity recognition.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior pre-training objectives when viewed through the lens of summarization:

1. Token-Level and Span-Level Masking Misses the Structure of Summarization

BERT's masked language modeling (MLM) randomly masks individual tokens (15% of tokens in the input) and trains the model to predict the original token from surrounding context. This teaches bidirectional understanding of local context but operates at the word level—the model never learns to identify which sentences carry important information or to generate multi-sentence coherent text from a partial document. When used as a pre-training objective for a sequence-to-sequence model (following Rothe et al., 2019, where encoder and decoder share parameters), MLM provides useful representations but no task-specific preparation for condensing documents into summaries.

MASS (Song et al., 2019) improved on this by masking a continuous fragment of a sentence (rather than random individual tokens) and training the model to reconstruct it, forcing the decoder to generate coherent multi-token sequences. However, MASS operates at the sentence-fragment level: it randomly selects a single contiguous span from a single sentence. It never masks multiple whole sentences, never selects sentences based on importance, and never generates output that resembles a multi-sentence summary—the masked fragment is a piece of a single sentence, not a condensed representation of a longer document.

BART (Lewis et al., 2019) generalized this further with arbitrary noising functions, primarily using text infilling (masking random spans of text with single mask tokens). While BART's denoising approach can mask longer spans than MASS, it still uses random selection—there is no notion of choosing spans because they are "important" or "principal" to the document. The model learns to reconstruct the original text, not to select and condense its key content.

T5 (Raffel et al., 2019) used corrupted text spans with varying mask ratios and sizes, turning everything into a text-to-text format. The authors note that T5 and PEGASUS share one design choice: both generate only the masked portions as output, rather than reconstructing the full input sequence. But T5's masking is still random and span-based—there is no sentence-level, importance-based selection.

The key distinction PEGASUS draws is that these objectives train models to reconstruct corrupted input (denoising), whereas summarization requires selecting and condensing important content. Reconstruction treats all tokens as equally worth recovering; summarization is inherently selective.

2. No Explicit Training for Content Selection

None of the prior pre-training objectives incorporate the concept of importance or salience. In extractive summarization research, it is well-known that not all sentences contribute equally to a summary: the first few sentences of news articles often carry the most information (the "lead bias"), and methods like TextRank or LexRank explicitly model sentence importance via graph-based centrality. Pre-training objectives that mask randomly—whether tokens, spans, or whole sentences—train the model that any piece of text could be missing and needs reconstruction, which is fundamentally misaligned with the summarization task, where only the important information should be in the output and the rest should be discarded.

The paper emphasizes this with a pointed observation about a naive option: "a naive option would be to pre-train as an extractive summarizer; however, such a procedure would only train a model to copy sentences, thus not suitable for abstractive summarization." The authors recognize that while extractive pre-training (masking sentences and having the model copy them back) would teach content selection, it would fail to teach the abstractive generation that distinguishes modern summarization. Their GSG objective is designed to sit in the sweet spot: the selection step (choosing gap sentences based on importance) trains the model to identify salient content, while the generation step (producing the gap sentences as a single output sequence) trains abstractive generation, since the model must produce the sentences—not just point to them—and the output can be formatted as a single contiguous pseudo-summary.

3. No Systematic Cross-Domain Evaluation

Perhaps most significantly, the authors argue that the field lacked "systematic evaluation across diverse domains." Prior work on pre-training for summarization typically evaluated on 2–3 datasets, almost always including CNN/DailyMail (news) and sometimes XSum (extreme news summarization) or Gigaword (headline generation). But summarization needs vary dramatically by domain: scientific papers require the model to understand specialized terminology and extract findings from dense technical prose; legislative bills require parsing legal language and identifying policy changes; email subject line generation requires extreme compression (one short phrase) from informal conversational text.

Without broad evaluation, it was impossible to know whether pre-training improvements on news datasets would generalize to other domains, or whether domain mismatch between pre-training corpus and downstream task would hurt performance. This is not a hypothetical concern: the paper itself shows (Figure 3) that HugeNews pre-training helps news datasets but hurts informal ones like WikiHow and Reddit TIFU compared to C4 pre-training. A pre-training strategy that appears optimal on CNN/DailyMail might actually be harmful for an email summarization deployment.

How This Paper Positions Itself

The paper draws a direct line to contemporaneous pre-trained encoder-decoder models—MASS, UniLM, BART, T5—and positions itself as addressing a specific, underexplored question: what if the pre-training objective looked like summarization? The authors are explicit about this lineage, noting that "most similar to our approach are Transformer encoder-decoder models pre-trained on some masked input pre-training objective," then systematically distinguishing PEGASUS from each:

MASS: masks a single sentence fragment; PEGASUS masks multiple whole sentences
UniLM: jointly trains on multiple language modeling tasks including word-level mask; PEGASUS focuses on sentence-level gap generation
BART: uses arbitrary noising with random spans; PEGASUS deterministically selects sentences based on importance
T5: shares the text-to-text format and the choice to generate only masked portions, but uses random corruption rather than importance-based sentence selection

The paper's framing is not that prior approaches are wrong, but that they are generic, and that summarization-specific design choices—sentence-level rather than token-level masking, importance-based rather than random selection, generating multiple sentences as a single output—can yield better performance. This is a "domain-informed pre-training" argument: the closer the pre-training task is to the downstream task, the better the transfer.

The paper also positions itself in the low-resource summarization lineage, citing two prior efforts: Radford et al. (2018b), where GPT-2 prompted with "TL;DR" achieved ROUGE-2 of 8.27 on CNN/DailyMail, and Khandelwal et al. (2019), where a Wikipedia-pretrained language model fine-tuned on 3,000 examples achieved ROUGE-2 of 13.1. PEGASUS aims to dramatically exceed both, and does—achieving ROUGE-2 of 13.28 in zero-shot (no fine-tuning) and 19.35 with only 1,000 examples, beating Khandelwal's result with one-third the supervision.

Finally, the paper acknowledges a conceptual tension that shapes its design: the desire for the pre-training objective to "closely resemble the downstream task" must be balanced against the reality that pre-training corpora (C4, HugeNews) contain only raw documents, not summaries. The gap-sentences generation objective resolves this tension by creating pseudo-summaries from the documents themselves: select important sentences from the document, mask them, and generate them from the remaining text. This is self-supervised—requiring no human annotations—yet structured to mimic the input-output format of summarization. The authors explicitly hypothesize that "this objective is suitable for abstractive summarization as it closely resembles the downstream task, encouraging whole-document understanding and summary-like generation."

3. Technical Approach

3.1 Reader Orientation

The system is a Transformer encoder-decoder model pre-trained on massive text corpora to generate "gap sentences"—sentences that have been removed from a document because they appear to be important. It solves the problem that no prior pre-training objective was designed specifically for abstractive summarization: the solution's shape is a self-supervised objective that creates pseudo-summaries from raw documents by selecting important sentences, masking them, and training the model to generate them from the remaining text.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Sentence Selection Module — given a raw document from the pre-training corpus, selects which sentences will become the "gap sentences" (the pseudo-summary) and which will remain as input. Uses ROUGE1-F1 between each sentence and the rest of the document as a proxy for importance.
Transformer Encoder-Decoder (PEGASUS) — a standard sequence-to-sequence Transformer that takes the remaining (non-selected) sentences as input and generates the selected gap sentences as output. The encoder processes the partial document; the decoder generates the missing sentences autoregressively.
Masked Language Model (MLM) Head (optional, not used in the final model) — an auxiliary objective that randomly masks individual tokens in the input and trains the model to predict them, applied to the encoder during pre-training.
Fine-Tuning Module — adapts the pre-trained PEGASUS model to specific downstream summarization datasets using supervised document-summary pairs.

Information flows as follows: a raw document enters the system → the sentence selection module computes ROUGE1-F1 importance scores for each sentence → the top-m sentences are selected as gap sentences and masked with [MASK1] tokens → the remaining sentences form the encoder input, with some tokens optionally masked with [MASK2] (if MLM is used) → the encoder produces contextualized representations of the partial document → the decoder generates the gap sentences autoregressively as a single output sequence → the model is trained to maximize the likelihood of the actual gap sentences.

3.3 Roadmap for the Deep Dive

First, the gap sentences generation (GSG) objective itself: why sentence-level masking, why importance-based selection, and how the objective is formulated as a sequence-to-sequence task. This is the core innovation.
Second, the sentence selection strategies: the three methods (Random, Lead, Principal), the four variants of Principal selection, and the ROUGE1-F1 calculation that underlies importance scoring. This is where the "summarization-mimicking" property comes from.
Third, the Masked Language Model (MLM) auxiliary objective: how it works, how it interacts with GSG, and why it was ultimately excluded from the final model.
Fourth, the model architecture and scaling: the specific Transformer configurations (PEGASUS_BASE and PEGASUS_LARGE), vocabulary choices, and pre-training hyperparameters.
Fifth, the pre-training corpora (C4 vs. HugeNews) and their domain-specific effects.
Sixth, the fine-tuning procedure: how the pre-trained model is adapted to downstream tasks, handling variable-length inputs, and hyperparameter sweeps.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a pre-training methodology paper whose core idea is that a self-supervised objective resembling the downstream summarization task—selecting and generating important sentences—yields better transfer than generic denoising objectives, with the additional insight that choosing which sentences to mask matters substantially.

The Gap Sentences Generation (GSG) Objective

The central innovation is that the pre-training task itself is structured to mimic abstractive summarization. Given a document $D = \{x_i\}_{i=1}^n$ consisting of $n$ sentences, the objective selects $m$ sentences (called "gap sentences") according to some selection strategy, removes them from the document, and trains the model to generate them as a single concatenated output sequence from the remaining $n - m$ sentences.

What the input and output look like. The input to the encoder is the document with the selected sentences replaced by a special mask token [MASK1] at their original positions. The target output for the decoder is the concatenation of the selected gap sentences in their original order. Figure 1 illustrates this: originally three sentences, one is masked with [MASK1] and becomes the target, while the other two remain in the input.

Gap Sentences Ratio (GSR). The key hyperparameter is $\text{GSR} = m / n$ , the fraction of sentences selected as gaps. A low GSR makes the task easier (fewer sentences to generate, more context available) and computationally more efficient (shorter target sequences). A high GSR loses contextual information necessary to guide generation—if too many sentences are removed, the remaining text may not contain enough information to reconstruct what was removed. The authors sweep GSR from 15% to 75% and find optimal values are always below 50%, with different downstream datasets preferring different ratios: CNN/DailyMail does best with 15%, XSum and Reddit TIFU with 30%, WikiHow with 45%.

Why whole sentences rather than spans or tokens? The authors argue that masking whole sentences encourages "whole-document understanding and summary-like generation." When the model must generate an entire sentence (potentially multiple sentences) from surrounding context, it must understand the document's overall content and structure—not just local co-occurrence patterns. Token-level masking (as in BERT or T5's span corruption) teaches the model that any piece of text could be missing, which is not how summarization works: in summarization, most of the text is omitted and only the important parts appear in the output. Sentence-level masking more closely mirrors this asymmetry.

Why generate only the gaps rather than reconstruct everything? Like T5, PEGASUS does not reconstruct the full input sequence; it only generates the masked sentences as the output. The authors note this is because summarization is inherently selective—the model should learn to produce only the important content, not to reproduce everything. If the objective required reconstructing the full document, the model might learn to be extractive (simply reproducing the input) rather than learning the compression and selection that summarization demands.

The design choice against extractive pre-training. The authors explicitly consider and reject a naive alternative: "a naive option would be to pre-train as an extractive summarizer; however, such a procedure would only train a model to copy sentences, thus not suitable for abstractive summarization." An extractive pre-training objective (selecting sentences, then having the model point to or copy those sentences) would teach content selection but not abstractive generation—the model would learn to identify important sentences but not to produce novel phrasing, which is the defining characteristic of abstractive summarization. GSG is designed to sit at the intersection: the selection step (choosing which sentences to mask) teaches content selection, while the generation step (producing the sentences from scratch) teaches abstractive generation.

Sentence Selection Strategies

The question of which sentences to select as gap sentences is where the summarization resemblance is created. The authors consider three primary strategies for selecting $m$ sentences without replacement from a document of $n$ sentences:

Random. Uniformly select $m$ sentences at random. This is the baseline: it teaches the model to generate coherent text from partial documents but provides no signal about which sentences are "summary-worthy."

Lead. Select the first $m$ sentences. This exploits the well-known "lead bias" in news articles—the fact that journalists typically front-load the most important information. The authors note that Lead "had decent performance on the two news datasets but was significantly worse on the two non-news datasets," consistent with prior findings that the lead bias is domain-specific.

Principal. Select the top- $m$ scored sentences according to importance, where importance is measured as the ROUGE1-F1 score between the sentence and the rest of the document:

$s_i = \text{rouge}(x_i, D \setminus \{x_i\}), \quad \forall i$

where $s_i$ is the importance score for sentence $x_i$ , and $\text{rouge}(\cdot, \cdot)$ computes ROUGE-1 F1 between the sentence and the remaining sentences in the document (treated as a single block of text).

What this computes operationally. For each sentence in the document, ROUGE-1 F1 is calculated between the unigrams (individual words) in that sentence and the unigrams in all other sentences combined. A high ROUGE-1 F1 means the sentence shares many words with the rest of the document—it is topically central, containing vocabulary that appears throughout the document. This is the proxy for "importance" or "principal-ness."

Why ROUGE-1 F1? The choice of ROUGE-1 F1 (rather than ROUGE-2 or ROUGE-L) is deliberate: unigram overlap captures topical centrality without requiring exact phrasal matches, which would be too strict. A sentence that uses key terminology appearing throughout the document will score highly even if it phrases things differently; a sentence about an unrelated tangent will score low. The F1 formulation balances precision (what fraction of the sentence's words appear in the rest of the document?) and recall (what fraction of the document's important terms appear in this sentence?).

Four variants of Principal selection. The authors consider two orthogonal choices in how the ROUGE-based selection is performed:

Independent (Ind) vs. Sequential (Seq) selection:
- Ind: Score each sentence independently against the full remaining document (as in the formula above), then select the top $m$ by score. This is computationally simpler but can select redundant sentences—two sentences that both score highly because they share vocabulary with the same document sections.
- Seq: Greedily select sentences one at a time, each time recomputing ROUGE1-F1 between the union of already-selected sentences plus the candidate, and the remaining unselected sentences. Formally, at step $j$ : $s_i = \text{rouge}(S \cup \{x_i\}, D \setminus (S \cup \{x_i\}))$ , where $S$ is the set of sentences already selected. This is Algorithm 1 in the paper. The sequential approach encourages diversity: once a sentence covering certain content is selected, other sentences covering that same content add less marginal ROUGE score.
Original (Orig) vs. Unique (Uniq) n-gram counting:
- Orig: Uses the standard ROUGE implementation, which double-counts identical n-grams. If the word "climate" appears 10 times in the sentence and 100 times in the rest of the document, it contributes 10 to the numerator (the count of matches).
- Uniq: Treats n-grams as a set, counting each unique n-gram only once regardless of frequency. This prevents high-frequency words from dominating the importance score and gives more weight to the presence of diverse vocabulary.

These choices produce four Principal variants: Ind-Orig, Ind-Uniq, Seq-Orig, Seq-Uniq. The results (Figure 4a) show that Ind-Orig achieves the best overall performance, followed by Seq-Uniq. Both Ind-Orig and Seq-Uniq consistently outperform Random and Lead across all four evaluation datasets (XSum, CNN/DailyMail, WikiHow, Reddit TIFU). Ind-Orig is selected for the final PEGASUS_LARGE model.

Stochastic sampling in the final model. For the final PEGASUS_LARGE (mixed, stochastic) model described in Section 6.6, the paper introduces a refinement: importance scores are perturbed with 20% uniform noise before selection, meaning the selected sentences are not strictly the top-ranked ones but are sampled with some randomness based on the noisy scores. This stochasticity prevents the model from overfitting to a deterministic selection pattern and provides a form of data augmentation. The gap sentences ratio is also dynamically sampled uniformly between 15% and 45% for each training example.

Example illustrating the strategies. Figure 2 shows a document about a Labour Party event, with sentences selected by Random, Lead, and Ind-Orig. The Ind-Orig selection picks the sentences that are most topically central: "Speakers include Rajesh Agrawal, the London Deputy Mayor for Business, Alison McGovern, the Chair of Progress, and Seema Malhotra MP" and "The Labour Business Fringe at this year's Labour Annual Conference is being co-sponsored by Labour in the City and the Industry Forum." These contain the key names and organizations that appear throughout the document. Random and Lead selections capture different (and arguably less central) content.

Masked Language Model (MLM) Auxiliary Objective

In addition to GSG, the authors experiment with applying BERT's masked language model objective to the encoder input. Following the standard BERT recipe, 15% of tokens in the input text are selected, and of those selected tokens: 80% are replaced with a special [MASK2] token (distinct from [MASK1] to avoid confusion with gap sentence masks), 10% are replaced with a random token, and 10% are left unchanged. The model must predict the original identity of the masked tokens.

How MLM interacts with GSG. Figure 1 shows both objectives applied simultaneously: one sentence is masked with [MASK1] and serves as the GSG target, while additional tokens in the remaining sentences are masked with [MASK2] for the MLM objective. When MLM is used as the sole pre-training objective (without GSG), the Transformer decoder shares all parameters with the encoder during fine-tuning, following the approach of Rothe et al. (2019).

Why MLM was ultimately excluded. The paper reports a nuanced finding: "MLM improved fine-tuning performance at early pre-training checkpoints (100k–200k steps), but inhibited further gains with more pre-training steps (500k)." At 500k steps, models trained with GSG alone (Ind-Orig) outperformed those trained with both GSG and MLM (MLM & Ind-Orig), and the combined objective performed similarly to Random GSG selection. Furthermore, MLM as the sole objective performed "significantly worse" than any GSG variant.

The authors hypothesize that the MLM objective may compete with GSG for the encoder's representational capacity: MLM encourages the encoder to retain fine-grained token-level information (to predict individual masked words), while GSG encourages the encoder to develop a more holistic, sentence-level understanding of document content. At scale (500k steps, large models), the GSG objective alone is sufficient and the MLM auxiliary task becomes a distraction rather than a benefit. Since PEGASUS_LARGE is trained for 500k steps, MLM is not included in the final model.

Model Architecture and Scaling

PEGASUS uses a standard Transformer encoder-decoder architecture (Vaswani et al., 2017) with sinusoidal positional encodings. Two model sizes are used:

PEGASUS_BASE (223M parameters): L = 12 encoder layers, L = 12 decoder layers, H = 768 hidden size, F = 3072 feed-forward layer size, A = 12 self-attention heads. This model is used for all ablation studies to save computation before scaling up.

PEGASUS_LARGE (568M parameters): L = 16 encoder layers, L = 16 decoder layers, H = 1024 hidden size, F = 4096 feed-forward layer size, A = 16 self-attention heads. This is the final model reported in all main results.

The notation follows the standard Transformer convention: L is the number of Transformer blocks in both encoder and decoder (each block contains self-attention followed by feed-forward layers), H is the dimensionality of the hidden representations, F is the inner dimensionality of the feed-forward layers (typically 4×H), and A is the number of parallel attention heads per layer.

Sinusoidal positional encodings. The paper uses sinusoidal positional encodings following Vaswani et al. (2017) rather than learned positional embeddings. This is a deliberate choice: sinusoidal encodings can extrapolate to sequence lengths not seen during training. This property proves critical during fine-tuning, where some downstream datasets (CNN/DailyMail, Multi-News, arXiv, PubMed, BIGPATENT) have input documents longer than the maximum input length used in pre-training (L_input = 512 tokens). The authors confirm that sinusoidal positional encodings "generalize well when fine-tuning PEGASUS_LARGE beyond the input lengths observed in training up to L_input = 1024 tokens."

Pre-training optimization. Both pre-training and fine-tuning use the Adafactor optimizer (Shazeer & Stern, 2018) with square root learning rate decay. The learning rate for pre-training is 0.1 with 0.0 label smoothing. The dropout rate is 0.1 throughout. PEGASUS_BASE uses a batch size of 256; PEGASUS_LARGE uses a much larger batch size of 8192. Both are trained for 500k steps (the PEGASUS_LARGE mixed,stochastic variant is trained for 1.5M steps). Neither model observes all examples in the pre-training corpora at 500k steps—the corpora are simply too large.

Maximum sequence lengths. During pre-training, the maximum input length (L_input) is 512 tokens and the maximum target length (L_target) is 256 tokens. Documents are truncated to 400 words before tokenization, which with a typical token-to-word ratio means most documents fit within 512 tokens. During fine-tuning, these lengths are adjusted per dataset: for example, XSum uses 512 input / 64 target, CNN/DailyMail uses 1024 input / 128 target, and BIGPATENT uses 1024 input / 256 target (full details in Table C.1).

A note on preventing exploitation of the mask token. For PEGASUS_LARGE, the authors introduce a small but important modification: "to encourage the model to copy, which is an important aspect of the more extractive datasets, we left 20% of selected sentences unchanged in the input instead of replacing with [MASK1]." This means that for 20% of gap sentences, the sentence remains visible in the input but is still generated as output. This teaches the model that sometimes the correct behavior is to reproduce input text verbatim (as in extractive summarization), while still training abstractive generation for the other 80% of cases. The effective GSR is increased to 45% to compensate—the model sees more gaps, but 20% of those gaps don't actually hide information, resulting in a similar number of truly masked sentences as the optimal 30% GSR found in the ablation studies.

Vocabulary Choices

The paper compares two subword tokenization methods, both implemented in the SentencePiece library:

Byte-Pair Encoding (BPE) (Wu et al., 2016; Sennrich et al., 2016) with a vocabulary size of 32k. BPE iteratively merges the most frequent pairs of tokens, building a vocabulary of common subword units.

SentencePiece Unigram (Kudo, 2018) with vocabulary sizes ranging from 32k to 256k. The Unigram algorithm starts with a large vocabulary and prunes it by removing tokens whose removal least increases the training loss, producing a vocabulary where each token's probability under the unigram model is optimal.

Figure 5 shows the results. BPE and Unigram are comparable on news datasets (XSum, CNN/DailyMail), but Unigram outperforms BPE on non-news datasets, particularly WikiHow. The optimal vocabulary size varies by task: Unigram 96k achieves the highest ROUGE scores on XSum and CNN/DailyMail, 128k on WikiHow, and 64k on Reddit TIFU. The authors select Unigram 96k as the overall best option for PEGASUS_LARGE.

The larger vocabulary of Unigram is particularly beneficial for informal and technical domains (WikiHow, Reddit TIFU) because it reduces the number of out-of-vocabulary tokens and the frequency of rare words being split into many subword pieces. For the PEGASUS_LARGE (mixed, stochastic) model, the tokenizer is updated to encode the newline character, which is a minor refinement for handling document structure.

Pre-Training Corpora

Two large text corpora are used:

C4 (Colossal and Cleaned version of Common Crawl), introduced in Raffel et al. (2019). It consists of text from 350 million web pages, totaling approximately 750GB. C4 is publicly available and is used for most ablation studies for reproducibility.

HugeNews, a newly collected dataset of 1.5 billion articles (3.8TB) from news and news-like websites spanning 2013–2019. The collection process involves: (1) curating a whitelist of domains from high-quality news publishers to lower-quality sites (high-school newspapers, blogs); (2) using this whitelist to seed a web-crawler; (3) applying heuristics to identify news-like articles; and (4) extracting only the main article text as plain text. This corpus is substantially larger than C4 (1.5B vs. 350M documents, 3.8TB vs. 750GB) and is domain-specific to news and journalism.

Corpus effects on downstream performance (Figure 3). Pre-training on HugeNews is more effective than C4 for the two news downstream datasets (XSum, CNN/DailyMail), while C4 is better for the two informal, non-news datasets (WikiHow, Reddit TIFU). This demonstrates that "pre-training models transfer more effectively to downstream tasks when their domains are aligned better"—a finding that motivates the final PEGASUS_LARGE (mixed, stochastic) model, which pre-trains on a mixture of C4 and HugeNews weighted by their number of examples.

Fine-Tuning Procedure

After pre-training, PEGASUS is fine-tuned on each downstream summarization dataset independently using supervised document-summary pairs. The fine-tuning process follows standard sequence-to-sequence practice: the document is fed to the encoder, the summary is generated by the decoder, and the model is trained to maximize the likelihood of the reference summary tokens.

Decoding strategy. For the ablation studies (Section 6.1), greedy decoding is used (beam size = 1). For the final PEGASUS_LARGE model, beam search with a length penalty $\alpha$ is used, following Wu et al. (2016). The beam size is typically 8, and $\alpha$ is tuned per dataset (e.g., 0.8 for XSum, 0.8–0.9 for CNN/DailyMail, 0.6 for WikiHow and Reddit TIFU). Beam search with a length penalty $\alpha$ modifies the scoring of candidate sequences to account for their length:

$\text{score}(y) = \frac{\log P(y \mid x)}{|y|^\alpha}$

where $y$ is a candidate summary, $x$ is the input document, $|y|$ is the length of the summary in tokens, and $\alpha$ controls the strength of the length penalty. When $\alpha < 1$ , longer summaries are penalized less (or even rewarded), encouraging the model to produce more detailed output; when $\alpha > 1$ , longer summaries are penalized more, encouraging conciseness.

Hyperparameter tuning. For PEGASUS_LARGE, the authors conduct "a simple hyper-parameter sweep of learning rate and length penalty $\alpha$ " for each downstream dataset. The learning rate typically ranges from 1e-4 to 8e-4, label smoothing is 0.1, and the number of fine-tuning steps varies by dataset size: from 12k steps for Reddit TIFU (the smallest dataset at 42k examples) to 300k steps for BIGPATENT (1.3M examples). Batch size is 256 for all fine-tuning experiments. Full hyperparameter details are provided in Table C.1.

Handling long documents. Several downstream datasets (CNN/DailyMail, Multi-News, arXiv, PubMed, BIGPATENT) contain input documents longer than the 512-token maximum used in pre-training. The authors increase L_input to 1024 tokens for these datasets and rely on sinusoidal positional encodings to extrapolate to the longer sequences. They note that since the average input length in BIGPATENT, arXiv, PubMed, and Multi-News is "well beyond 1024 tokens," further scaling L_input or applying a two-stage approach (Liu et al., 2018) might improve performance further, but this is left to future work.

Transformer_BASE baseline. For comparison, the authors train a Transformer model with the same architecture as PEGASUS_BASE but without any pre-training (i.e., from random initialization). This model is fine-tuned for more steps (300k vs. 50k for PEGASUS_BASE) and with a higher learning rate (5e-3 vs. 5e-4) to compensate for the lack of pre-trained weights. This baseline isolates the contribution of pre-training.

4. Key Insights and Innovations

Innovation 1: Pre-Training Objective Design as Task Mimicry, Not Just Denoising

The dominant paradigm in 2019 for pre-training sequence-to-sequence models was reconstructing corrupted text—a denoising framework where the model learns to recover original text from artificially noised versions. BART randomly masked spans. T5 corrupted contiguous text segments. MASS masked sentence fragments. In all cases, the corruption was random: any part of the text was equally likely to be masked, and the model's job was to undo the damage.

PEGASUS makes a conceptually distinct move: design the pre-training objective so that the selection of what to mask, not just the reconstruction, teaches the skill the downstream task requires. The insight is that summarization has two components—identifying what is important and generating it concisely—and that a pre-training objective can train both simultaneously if the masking is non-random. By selecting sentences based on a proxy for importance (ROUGE1-F1 with the rest of the document) rather than uniformly at random, the GSG objective creates a training signal where the model must learn to recognize which sentences carry central information because those are the ones it must generate. The decoder never sees random noise; it always sees gaps that correspond to principal content.

This reframes pre-training from a generic "learn good representations by being forced to understand corrupted text" to a more targeted "learn the specific information-filtering behavior that summarization demands." The comparison to prior work is instructive: BART, T5, and MASS all answer the question "can you reconstruct what was removed?" PEGASUS answers a different question: "can you identify and generate what was important?" The difference is subtle at the objective level—both involve masking and generation—but fundamental at the conceptual level: PEGASUS injects salience into the pre-training task, where prior work only injected noise.

The significance extends beyond the ROUGE gains in Table 1. What makes this an innovation rather than an incremental improvement is that it demonstrates a design principle: pre-training objectives can and should encode task-specific inductive biases, not just provide generic language understanding. The choice of what to mask becomes a parameter that shapes what the model learns, not just a way to create a self-supervised training signal. The finding that how sentences are selected matters substantially—Ind-Orig outperforms Random and Lead in Figure 4a—validates that this design dimension is worth optimizing, and the fact that different downstream datasets prefer different GSRs (15% for CNN/DailyMail, 30% for XSum, 45% for WikiHow, in Figure 4b) suggests that the "ideal" pre-training objective is itself domain-dependent, a subtlety that generic denoising approaches ignore entirely.

Innovation 2: ROUGE-Based Importance as a Self-Supervised Salience Signal

The second conceptual contribution is the specific mechanism for identifying "important" sentences without any labeled data. Prior work on extractive summarization had used centrality measures—TextRank applied PageRank to sentence similarity graphs, LexRank used eigenvector centrality—but these required running graph algorithms over each document. Prior work on supervised summarization trained classifiers to predict which sentences should appear in a summary, requiring human annotations.

PEGASUS's approach is strikingly simple and entirely self-supervised: compute ROUGE-1 F1 between each sentence and the rest of the document, and call that "importance." The insight is that a sentence which shares vocabulary with the document as a whole is likely to be topically central—it uses the same terminology, discusses the same entities, and touches on the same themes. This is not a perfect measure of summary-worthiness (a sentence can be topically central but redundant, or important but phrased in unique vocabulary), but it is free, requires no annotation, scales to billions of documents, and—crucially—works well enough to substantially outperform random or lead-based selection.

What makes this an innovation rather than just a heuristic is that it creates a continuous, differentiable bridge between extractive and abstractive summarization. The ROUGE score provides a scalar importance signal that can be used to select targets for abstractive generation. The model never sees extractive labels; it sees only "these sentences were selected because they had high ROUGE scores with the rest of the document," and it learns to generate them from the remaining context. The abstractive capability emerges from the generation task itself, while the selection capability is baked into the data construction. This elegantly sidesteps the problem the authors themselves identify—"a naive option would be to pre-train as an extractive summarizer; however, such a procedure would only train a model to copy sentences, thus not suitable for abstractive summarization"—by using extractive signals to create abstractive training targets.

The sequential selection variant (Seq, Algorithm 1) adds a further insight: importance is not just about individual sentence scores but about the marginal contribution of each sentence given those already selected. The greedy ROUGE maximization procedure is essentially a submodular content selection strategy, encouraging the model to see diverse rather than redundant content as the generation target. That Ind-Orig (independent selection with standard ROUGE counting) slightly outperforms Seq-Uniq in Figure 4a suggests that exact deduplication of content in the pseudo-summary is less critical than capturing the highest-scoring sentences overall, but the availability of both strategies—and the fact that they both beat random and lead—validates the broader principle that how gap sentences are chosen matters, not just that they are sentences rather than spans.

Innovation 3: The Domain-Specificity of Pre-Training Corpus Effects as a First-Class Finding

The paper's third conceptual contribution is treating the interaction between pre-training corpus domain and downstream task domain as a finding to be characterized systematically, not just a confound to be controlled for. Prior work on pre-training for NLP typically used a single corpus (often Wikipedia + BooksCorpus for BERT, or C4 for T5) and assumed that more data + more diverse data = better transfer. The implicit model was that pre-training on a large, general corpus would provide broadly useful representations, and domain-specific fine-tuning would handle any remaining shift.

PEGASUS challenges this assumption with direct evidence. Figure 3 shows a clear pattern: HugeNews (news articles) helps news summarization tasks but hurts informal ones; C4 (general web text) does the opposite. On XSum and CNN/DailyMail, HugeNews pre-training outperforms C4. On WikiHow and Reddit TIFU, C4 outperforms HugeNews. The effect is not just about corpus size—HugeNews is larger than C4 (1.5B vs. 350M documents)—but about domain alignment. More data from the wrong domain is worse than less data from a more relevant domain.

This finding transforms corpus selection from an engineering decision ("use the biggest corpus you can get") into a domain-adaptation strategy that can be optimized. The final PEGASUS_LARGE (mixed, stochastic) model operationalizes this insight by pre-training on a mixture of C4 and HugeNews weighted by their number of examples, producing a model that performs well across all domains (Table 4). But the deeper implication is that there is no universal pre-training corpus—the "best" corpus depends on what downstream tasks you care about. For a medical summarization system, a corpus of medical literature might outperform even larger general corpora. For a legal summarization system, a corpus of case law and legislation might be optimal. The paper provides the empirical methodology (comparing corpora on representative downstream tasks) and the conceptual framework (domain alignment matters) for making these decisions principled rather than ad hoc.

The significance is amplified by the low-resource findings (Figure 6): when fine-tuning data is scarce, the pre-training corpus's domain alignment likely matters even more, because there are fewer supervised examples to correct for domain shift. The paper doesn't explicitly vary the pre-training corpus in the low-resource experiments (all low-resource results use HugeNews), but the logic follows directly from the full-resource findings and opens a clear research direction.

Innovation 4: The Practical Uselessness of MLM for Large-Scale Summarization Pre-Training (A Negative Result with Implications)

The fourth innovation is a diagnostic negative result: masked language modeling—the objective that powered BERT and was widely considered a universal beneficial pre-training component—not only fails to help summarization at scale but actively inhibits further gains with extended pre-training. Section 6.1.2 reports that "MLM improved fine-tuning performance at early pre-training checkpoints (100k–200k steps), but inhibited further gains with more pre-training steps (500k)," and that at 500k steps, GSG alone (Ind-Orig) outperforms the combined GSG+MLM objective.

This is a non-obvious finding. The natural assumption in 2019 would have been that MLM is complementary to any sequence-to-sequence pre-training objective: MLM trains the encoder to produce better token-level representations, which should improve the encoder's ability to condition the decoder for generation tasks. The authors' hypothesis—that MLM and GSG compete for the encoder's representational capacity, with MLM encouraging fine-grained token-level information retention while GSG encourages holistic sentence-level understanding—provides a mechanistic explanation for why the two objectives conflict at scale.

The practical implication is significant: for summarization pre-training at scale, MLM is unnecessary and potentially harmful. This saves computation (no need for the MLM head or the additional forward/backward passes), simplifies the training pipeline, and avoids the need to tune the interaction between two objectives. More broadly, it challenges the "more objectives = better" assumption that drove multi-task pre-training approaches like UniLM. The finding suggests that objective selection should be guided by task alignment, not by additive benefit assumptions—objectives that train the model for different kinds of representations may interfere rather than cooperate.

The time-dependence of the effect (MLM helps early, hurts late) is particularly interesting. It suggests that MLM provides useful regularization or representation-shaping in the early stages of pre-training when the model's representations are still forming, but becomes a constraint once the model has sufficient capacity and data to develop more specialized, GSG-aligned representations. This temporal dynamic is rarely studied in pre-training research, where the focus is typically on final performance rather than on how different objectives interact over the course of training.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on 12 public abstractive summarization datasets accessed through TensorFlow Summarization Datasets, spanning news (XSum: 227k BBC articles with single-sentence summaries; CNN/DailyMail: 311k articles with bullet-point summaries using the non-anonymized variant from See et al. 2017; NEWSROOM: 1.3M article-summary pairs from 38 publications; Multi-News: 56k multi-document summaries; Gigaword: 4M headline-generation examples), science (arXiv: 113k scientific publications; PubMed: 215k papers, both requiring abstract generation from paper body), patents (BIGPATENT: 1.3M U.S. patents with human summaries), instructions (WikiHow: 200k instruction-step paragraphs with concatenated summary sentences), informal stories (Reddit TIFU: 120K posts from the TIFU sub-reddit using TL;DR as summaries), emails (AESLC: 18k email bodies and subjects from the Enron corpus), and legislative bills (BillSum: 23k U.S. Congressional bills with human reference summaries, excluding the out-of-distribution California test set). Train/validation/test splits follow an 80/10/10 ratio when no split is provided; a 10% train split serves as validation when no validation split exists.
Base model(s). The primary models are Transformer-based encoder-decoder architectures: PEGASUS_BASE (223M parameters: L=12 encoder and decoder layers, H=768 hidden size, F=3072 feed-forward size, A=12 attention heads) for ablation studies, and PEGASUS_LARGE (568M parameters: L=16, H=1024, F=4096, A=16) for final results. Both use sinusoidal positional encodings following Vaswani et al. (2017). A Transformer_BASE model of identical architecture but without pre-training serves as the no-pre-training baseline. For comparison with prior work, the paper includes BERTShare (Rothe et al., 2019), MASS (Song et al., 2019), UniLM (Dong et al., 2019), BART (Lewis et al., 2019), and T5 (Raffel et al., 2019).
Metrics. The primary metric throughout is ROUGE F1 (ROUGE-1, ROUGE-2, and ROUGE-L) computed using the standard implementation from Lin (2004). Answers are graded using the grading function released by Lightman et al. (2022). For the ablation studies (Figures 3–5), results are presented as normalized ROUGE scores using the formula $\frac{1}{3}(\frac{R1}{R1_{\text{base}}} + \frac{R2}{R2_{\text{base}}} + \frac{RL}{RL_{\text{base}}})$ where the denominator is the score of the configuration corresponding to the first bar in each figure. Human evaluation on XSum, CNN/DailyMail, and Reddit TIFU uses a 1–5 Likert scale (higher is better) with three independent workers per example, retaining the median score, and a paired t-test to assess statistical significance at p < 0.01.
Baselines. The paper evaluates against: (1) Transformer_BASE — same architecture as PEGASUS_BASE trained from scratch on each downstream dataset without pre-training; (2) BERTShare (Rothe et al., 2019) — a Transformer with shared encoder-decoder parameters pre-trained using BERT's MLM objective; (3) MASS (Song et al., 2019) — masked sequence-to-sequence generation reconstructing a randomly selected sentence fragment; (4) UniLM (Dong et al., 2019) — jointly trained on unidirectional, bidirectional, and sequence-to-sequence language modeling; (5) BART (Lewis et al., 2019) — a denoising autoencoder with text infilling noising function; (6) T5 (Raffel et al., 2019) — a text-to-text model with span corruption. For the low-resource setting, comparisons include GPT-2 prompted with "TL;DR" (Radford et al., 2018b) achieving ROUGE-2 of 8.27 on CNN/DailyMail, and Khandelwal et al. (2019) pre-training a language model on Wikipedia fine-tuned with 3,000 examples achieving ROUGE-2 of 13.1.
Generation budget / compute accounting. For the ablation studies, all PEGASUS_BASE models are pre-trained for 500k steps with a batch size of 256 on C4 (for fair comparison, as C4 is publicly available). PEGASUS_LARGE is pre-trained for 500k steps with a batch size of 8192, traversing more data due to the larger batch size at the same step count. The paper does not report total FLOPs or wall-clock time; compute is measured implicitly by the number of pre-training steps and model size. For fine-tuning, the number of steps varies by dataset (e.g., 50k steps for PEGASUS_BASE on most datasets, 130k steps for PEGASUS_LARGE on XSum with C4, 80k steps with HugeNews). The decoding strategy differs: greedy decoding (beam size = 1) for all ablation studies, and beam search with beam size 8 and tuned length penalty α for PEGASUS_LARGE final results.
Cross-validation / statistical protocol. For strategy selection in the compute-optimal experiments, the authors use two-fold cross-validation within each difficulty bin on the 500-question test set to avoid overfitting. For human evaluation, a paired t-test assesses whether model scores are significantly different from human reference scores, with significance threshold p < 0.01. For test-set overlap analysis, all test set examples with ROUGE-2 recall similarity to any pre-training example above thresholds of 1.0 and 0.8 are filtered, and ROUGE scores are recalculated on the remaining examples.

Main Quantitative Results

Pre-Training Corpus Effects (Section 6.1.1, Figure 3)

The headline finding is that pre-training corpus domain matters substantially and asymmetrically: HugeNews (1.5B news-like articles) outperforms C4 (350M web pages) on news downstream tasks, while C4 outperforms HugeNews on informal, non-news tasks (Figure 3). On XSum, PEGASUS_BASE pre-trained on HugeNews achieves approximately 4.6% higher normalized ROUGE than C4 pre-training (R1: 41.63 vs. 39.79, R2: 18.47 vs. 16.58, RL: 33.48 vs. 31.70). On CNN/DailyMail, the advantage is smaller but consistent (R1: 42.34 vs. 41.79, R2: 19.22 vs. 18.81, RL: 39.49 vs. 38.93). The pattern reverses on the non-news datasets: WikiHow performs better with C4 (R1: 36.58 vs. 34.93, R2: 15.64 vs. 14.67, RL: 30.01 vs. 28.63), and Reddit TIFU shows a similar C4 advantage (R1: 24.36 vs. 24.11, R2: 6.09 vs. 5.99, RL: 18.75 vs. 18.57). The exact numerical differences are in Table D.1 (rows for C4 and HugeNews pre-training). The practical significance is that for news summarization deployments, a domain-specific news corpus yields better transfer than a larger general corpus.

Gap Sentences Generation (GSG) Objective: Selection Strategy and Ratio (Section 6.1.2, Figures 4a and 4b)

The central result is that choosing gap sentences based on importance (Ind-Orig) consistently outperforms random and lead-based selection (Figure 4a). Across the four evaluation datasets (XSum, CNN/DailyMail, WikiHow, Reddit TIFU), Ind-Orig achieves the highest normalized ROUGE scores. The exact ROUGE values for Ind-Orig from Table D.1: XSum (R1: 39.79, R2: 16.58, RL: 31.70), CNN/DailyMail (R1: 41.79, R2: 18.81, RL: 38.93), WikiHow (R1: 36.58, R2: 15.64, RL: 30.01), Reddit TIFU (R1: 24.36, R2: 6.09, RL: 18.75). Random selection performs meaningfully worse: XSum (R1: 39.28, R2: 16.23, RL: 31.21), CNN/DailyMail (R1: 41.80, R2: 18.91, RL: 38.88), WikiHow (R1: 36.27, R2: 15.47, RL: 29.67), Reddit TIFU (R1: 24.04, R2: 6.01, RL: 18.47). The gap between Ind-Orig and Random is ~0.3–0.5 ROUGE points across most metrics, which is modest but consistent.

Lead selection shows a strong domain asymmetry: it performs competitively on the news datasets (XSum R1: 39.22, CNN/DailyMail R1: 41.70) but significantly worse on non-news (WikiHow R1: 35.30, Reddit TIFU R1: 23.48), consistent with the known lead bias in news articles.

Among the Principal variants, Ind-Orig (independent selection with original ROUGE counting) achieves the best overall performance, followed closely by Seq-Uniq (sequential selection with unique n-gram counting). The differences are small: Ind-Orig vs. Seq-Uniq on XSum (R2: 16.58 vs. 16.39), CNN/DailyMail (R2: 18.81 vs. 19.03, where Seq-Uniq actually edges ahead), WikiHow (R2: 15.64 vs. 15.61), Reddit TIFU (R2: 6.09 vs. 6.17). The practical implication is that while importance-based selection helps, the exact variant matters only modestly.

MLM performs poorly. As a sole objective, MLM achieves substantially lower scores: XSum (R1: 37.22, R2: 14.48, RL: 29.62), CNN/DailyMail (R1: 39.33, R2: 17.34, RL: 36.65), WikiHow (R1: 32.20, R2: 13.19, RL: 27.05), Reddit TIFU (R1: 21.00, R2: 3.96, RL: 16.27). Combining MLM with Ind-Orig (MLM & Ind-Orig) yields similar performance to Random GSG—worse than Ind-Orig alone—confirming that MLM does not complement GSG at 500k steps (the training duration used for ablation studies). The exact ROUGE values for MLM & Ind-Orig: XSum (R1: 39.08, R2: 16.21, RL: 31.20), CNN/DailyMail (R1: 41.48, R2: 18.70, RL: 38.63), WikiHow (R1: 35.99, R2: 15.29, RL: 29.57), Reddit TIFU (R1: 24.19, R2: 6.16, RL: 18.70).

Gap Sentences Ratio (GSR) varies by task (Figure 4b). Across GSR values from 15% to 75%, the optimal ratio is always below 50%. CNN/DailyMail achieves the highest ROUGE scores at 15% GSR (R1: 41.88, R2: 18.98, RL: 38.97). XSum and Reddit TIFU perform best at 30% (XSum R1: 39.61, R2: 16.51, RL: 31.48; Reddit TIFU R1: 24.05, R2: 6.05, RL: 18.55). WikiHow does best at 45% (R1: 36.39, R2: 15.46, RL: 29.85). At 75% GSR, performance degrades substantially on all datasets: XSum drops to R1: 36.94, R2: 14.21, RL: 29.14; CNN/DailyMail drops to R1: 40.17, R2: 17.52, RL: 37.37; WikiHow drops to R1: 34.32, R2: 13.72, RL: 27.96; Reddit TIFU drops to R1: 21.72, R2: 4.32, RL: 16.45. The consistent degradation at high GSR confirms that removing too many sentences destroys the contextual information needed for coherent generation. For PEGASUS_LARGE, an effective GSR of 30% is used.

Vocabulary Choice (Section 6.1.3, Figure 5)

Unigram tokenization outperforms BPE on non-news datasets, with the optimal vocabulary size varying by task (Figure 5, Table D.1). On the news datasets, BPE 32k and Unigram 96k are comparable: XSum (BPE: R1 39.23 / R2 16.17 / RL 31.13; Unigram 96k: R1 39.33 / R2 16.40 / RL 31.24), CNN/DailyMail (BPE: R1 41.86 / R2 18.97 / RL 38.97; Unigram 96k: R1 42.22 / R2 19.31 / RL 39.34). On WikiHow, the gap is substantial: Unigram 128k achieves R1: 37.66, R2: 16.04, RL: 30.83 versus BPE 32k's R1: 35.22, R2: 14.88, RL: 28.87—a ~2.4 ROUGE-1 and ~1.2 ROUGE-2 improvement. Reddit TIFU shows a smaller but consistent Unigram advantage: Unigram 64k achieves R1: 24.47, R2: 6.32, RL: 18.90 versus BPE's R1: 24.04, R2: 6.04, RL: 18.57. The selected vocabulary, Unigram 96k, represents a compromise that performs well across all datasets.

Large Model Results: State-of-the-Art Across 12 Datasets (Section 6.2, Tables 1 and 2)

PEGASUS_LARGE (HugeNews) achieves state-of-the-art ROUGE scores on all 12 downstream datasets (Table 1). On XSum, it reaches R1: 47.21, R2: 24.56, RL: 39.25, surpassing the previous best (BART's R1: 45.14, R2: 22.27, RL: 37.25) by approximately 2 ROUGE-1, 2.3 ROUGE-2, and 2 ROUGE-L points. On CNN/DailyMail, PEGASUS_LARGE (HugeNews) achieves R1: 44.17, R2: 21.47, RL: 41.11, essentially matching BART (R1: 44.16, R2: 21.28, RL: 40.90) and T5 (R1: 43.52, R2: 21.55, RL: 40.69). On NEWSROOM, it achieves R1: 45.15, R2: 33.51, RL: 41.33, a substantial improvement over the previous SOTA (R1: 39.91, R2: 28.38, RL: 36.87). On Multi-News, the improvement is notable: R1: 47.52, R2: 18.72, RL: 24.91 versus previous SOTA R1: 43.47, R2: 14.89, RL: 17.41.

The gains are most dramatic on smaller datasets where pre-training provides the largest benefit. On AESLC (18k examples), PEGASUS_LARGE achieves R2: 21.85, nearly double the previous SOTA of 10.29. On Reddit TIFU (42k examples), R2 reaches 9.01 versus the previous best of 3.7—a roughly 2.4× improvement. On BillSum (24k examples), R2 reaches 40.19 versus the previous 23.83. The Transformer_BASE baselines (no pre-training) underscore these gains: on AESLC, Transformer_BASE achieves R2: 7.39 versus PEGASUS_LARGE's 21.85; on Reddit TIFU, Transformer_BASE R2: 1.94 versus PEGASUS_LARGE's 9.01—a nearly 5× improvement.

Table 2 provides a focused comparison with other pre-trained models on XSum, CNN/DailyMail, and Gigaword. PEGASUS_LARGE (HugeNews) outperforms MASS on XSum by approximately 7.5 ROUGE-1, 7.3 ROUGE-2, and 7.3 ROUGE-L. On Gigaword, PEGASUS_LARGE achieves R1: 39.12, R2: 19.86, RL: 36.24, slightly exceeding MASS (R1: 38.73, R2: 19.71, RL: 35.96) and comparable to UniLM (R1: 38.45, R2: 19.45, RL: 35.75). Gigaword shows the smallest gains, likely because it is a sentence-compression task (generating a headline from the first sentence) rather than a full-document summarization task, making the gap-sentences generation pre-training less directly aligned.

Corpus effects persist at scale. PEGASUS_LARGE (C4) performs better than PEGASUS_LARGE (HugeNews) on WikiHow (R1: 43.06 vs. 41.35, R2: 19.71 vs. 18.51, RL: 34.80 vs. 33.42), confirming that the domain-alignment effect observed with PEGASUS_BASE (Figure 3) generalizes to the larger model. On all other datasets, HugeNews matches or exceeds C4.

Zero and Low-Resource Summarization (Section 6.3, Figure 6 and Table E.1)

PEGASUS_LARGE (HugeNews) demonstrates remarkable sample efficiency, achieving competitive performance with extremely limited fine-tuning data (Figure 6). With only 100 examples, PEGASUS_LARGE reaches performance comparable to Transformer_BASE trained on the full supervised datasets (ranging from 20k to 200k examples) on 8 out of 12 datasets. The quantitative results from Table E.1 show:

CNN/DailyMail: Zero-shot ROUGE-2 of 13.28 (exceeding GPT-2's 8.27 reported in Radford et al., 2018b). With 100 examples, R2 reaches 18.21; with 1,000 examples, R2 reaches 19.35—well above the 13.1 achieved by Khandelwal et al. (2019) using 3,000 examples, meaning PEGASUS matches or exceeds prior low-resource SOTA with 3× fewer examples. Full supervision (311k examples) achieves R2: 21.47.
XSum: Zero-shot R2: 3.00. With 100 examples, R2 jumps to 16.44; with 1,000 examples, R2 reaches 18.23; full supervision (226k examples) reaches R2: 24.56.
WikiHow: With 1,000 examples, R2 reaches 12.17—already exceeding the previous SOTA of 9.23 (which used the full 168k training set). With 10k examples, R2 reaches 14.41, substantially above the prior SOTA.
Reddit TIFU: With 100 examples, R2 reaches 4.09 (exceeding the previous SOTA of 3.7 with full supervision). With 1,000 examples, R2 reaches 6.85, nearly double the prior SOTA.
BIGPATENT: With 1,000 examples, R2 reaches 12.58 (exceeding the previous SOTA of 10.63 with full supervision).
BillSum: With 100 examples, R2 reaches 26.40 (exceeding the previous SOTA of 23.83). With 1,000 examples, R2 reaches 30.58.
AESLC: At 1,000 examples, R2 reaches 15.45 (above the previous SOTA of 10.29). At full supervision, R2 reaches 21.85.

A key observation from Figure 6 is that the ROUGE curves for PEGASUS_LARGE with limited data are often above the dashed lines (Transformer_BASE with full supervision), indicating that pre-training provides more value than full in-domain supervision with a non-pretrained model for many tasks. The gains saturate relatively quickly: for most datasets, the jump from 0 to 100 examples provides the largest improvement, with diminishing returns from 1,000 to 10,000 examples, suggesting that the pre-trained model already encodes much of the summarization capability needed.

Human Evaluation (Section 6.4, Table 3)

PEGASUS_LARGE summaries are rated as not significantly worse than human-written reference summaries on XSum, CNN/DailyMail, and Reddit TIFU (Table 3). In Experiment 1 (full supervision):

On XSum, both PEGASUS_LARGE (HugeNews) and PEGASUS_LARGE (C4) achieve mean ratings of 3.0 and 3.1 respectively, compared to 3.0 for human summaries (p = 0.6 and p = 0.7, both non-significant at p < 0.01). Transformer_BASE scores significantly lower at 2.0 (p = 3e-10).
On CNN/DailyMail, PEGASUS_LARGE (HugeNews) actually receives a higher mean rating (3.6) than human summaries (3.1), with p = 0.0001 indicating this difference is statistically significant. PEGASUS_LARGE (C4) scores 3.5 (p = 0.009, also significantly above human). This is a rare result—model outputs rated as statistically better than reference summaries—suggesting that on CNN/DailyMail, PEGASUS produces summaries that human evaluators prefer over the original human-written references.
On Reddit TIFU, PEGASUS_LARGE (HugeNews) scores 3.2 versus human 3.2 (p = 0.7, non-significant), PEGASUS_LARGE (C4) scores 3.1 versus human 3.2 (p = 0.3, non-significant), and Transformer_BASE scores only 1.4 (p = 5e-23).

In Experiment 2 (low-resource), PEGASUS_LARGE (HugeNews) fine-tuned with as few as 10 examples is not measurably worse than human summaries on XSum and CNN/DailyMail at p < 0.01. On XSum, even 10 examples achieve a mean rating of 2.8 versus human 3.2 (p = 0.1, non-significant). On CNN/DailyMail, 10 examples achieve 3.4 versus human 3.2 (p = 0.007, actually significantly above human). On Reddit TIFU, however, human-level performance is achieved only with full supervision—with 1,000 examples, the model scores only 2.7 versus human 3.3 (p = 0.01, just at the significance boundary), and with full supervision 2.8 versus 3.3 (p = 0.05, non-significant at p < 0.01). The authors attribute this to Reddit TIFU's diverse writing styles, which require more supervised examples to fully capture.

Test-Set Overlap Analysis (Section 6.5, Figure 7)

Overlap between pre-training corpus and downstream test sets is minimal and does not explain performance gains. Figure 7 shows the percentage of test set examples with ROUGE-2 recall similarity to any C4 pre-training example above thresholds of 1.0 and 0.8. Only XSum has a significant amount of overlap: approximately 15–20% of test examples have some overlap. For CNN/DailyMail, Reddit TIFU, and WikiHow, the overlap is minimal (close to 0%). After filtering out overlapped examples (similarity threshold of 1.0), the average change in ROUGE-1/2/L across all three metrics is less than 1% for all four datasets. At the 0.8 threshold, the change remains under 1%. Manual examination of the overlapped examples "found that the models produce very different summaries compared to the human written ones, suggesting that there was no clear memorization." This rules out the concern that PEGASUS's strong performance is due to having seen the test summaries during pre-training.

Additional PEGASUS_LARGE Improvements (Section 6.6, Table 4)

The PEGASUS_LARGE (mixed, stochastic) model, incorporating five changes—(1) pre-training on a mixture of C4 and HugeNews, (2) dynamic GSR sampling between 15–45%, (3) stochastic importance sentence selection with 20% uniform noise on scores, (4) extended pre-training to 1.5M steps, and (5) updated tokenizer encoding newline—achieves the best results on almost all downstream tasks (Table 4). On XSum, it reaches R1: 47.60 / R2: 24.83 / RL: 39.64, slightly above the HugeNews-only model (R1: 47.21 / R2: 24.56 / RL: 39.25). On WikiHow, it reaches R1: 46.39 / R2: 22.12 / RL: 38.41, substantially above the C4 model (R1: 43.06 / R2: 19.71 / RL: 34.80) and the HugeNews model (R1: 41.35 / R2: 18.51 / RL: 33.42), showing that the mixed corpus captures the strengths of both sources. On BillSum, R2 reaches 41.58. Notably, BIGPATENT results are reported with updated dataset processing (preserving casing and cleaning), making them not directly comparable to the earlier BIGPATENT numbers.

Ablation Studies and Robustness Checks

Pre-training steps (Appendix B, Figure B.1): Performance on all four downstream datasets increases monotonically with more pre-training steps from 0 to 500k. The curves show no sign of saturation at 500k steps, suggesting that longer pre-training could yield further improvements. At 500k steps, PEGASUS_BASE on C4 achieves: XSum (R2: 16.58), CNN/DailyMail (R2: 18.81), WikiHow (R2: 15.64), Reddit TIFU (R2: 6.09). The gains from 400k to 500k steps are modest but consistent (e.g., XSum R2: 16.34 → 16.58, WikiHow R2: 15.41 → 15.64), indicating diminishing but still positive returns.
GSG selection strategy variants (Section 6.1.2, Figure 4a): The four Principal variants (Ind-Orig, Ind-Uniq, Seq-Orig, Seq-Uniq) all outperform Random and Lead, with Ind-Orig providing the consistently best performance. The differences among Principal variants are small: on CNN/DailyMail R2, Ind-Orig (18.81) vs. Seq-Uniq (19.03) shows Seq-Uniq slightly ahead; on WikiHow R2, Ind-Orig (15.64) vs. Seq-Uniq (15.61) are nearly identical; on XSum R2, Ind-Orig (16.58) leads Seq-Uniq (16.39) by a small margin. The practical conclusion is that while importance-based selection helps, the exact variant matters less than the choice to use importance at all.
MLM as sole objective vs. combined with GSG (Section 6.1.2, Figure 4a): MLM as the sole pre-training objective performs significantly worse than any GSG variant. More interestingly, MLM & Ind-Orig combined performs similarly to Random GSG and worse than Ind-Orig alone at 500k steps, despite helping at earlier checkpoints (100k–200k steps). This temporal interaction—MLM helps early, hurts late—is a non-obvious finding suggesting that objectives compete for representational capacity at scale.
Gap Sentences Ratio sweep (Section 6.1.2, Figure 4b): The optimal GSR varies by downstream task: 15% for CNN/DailyMail, 30% for XSum and Reddit TIFU, 45% for WikiHow. At 75% GSR, performance degrades substantially across all tasks. The sensitivity analysis validates that the choice of 30% for PEGASUS_LARGE is a reasonable compromise, though task-specific GSR tuning could yield small additional gains.
Vocabulary size and tokenization method (Section 6.1.3, Figure 5): Unigram consistently matches or outperforms BPE across all datasets, with the gap largest on WikiHow (Unigram 128k R2: 16.04 vs. BPE 32k R2: 14.88). Vocabulary sizes beyond 96k provide diminishing returns: Unigram 256k scores slightly below 128k on most metrics, suggesting overfitting or inefficiency from excessively large vocabularies.
Test-set overlap filtering (Section 6.5, Figure 7): Removing test examples with any overlap to the pre-training corpus (similarity > 0.8 or > 1.0) changes ROUGE scores by less than 1% on XSum, CNN/DailyMail, Reddit TIFU, and WikiHow. This demonstrates that the model's performance is not due to memorization of test data seen during pre-training. Only XSum shows non-trivial overlap (~15–20%), and even there, filtering does not materially affect results. Manual inspection of overlapped examples confirmed the model produces different summaries from the human references.
Abstractiveness analysis (Appendix H, Figure H.1): Model-generated summaries are consistently less abstractive than human-written summaries (higher extractive coverage and density), but models fine-tuned on highly abstractive datasets (XSum, Reddit TIFU) can generate more abstractive summaries than human-written ones on extractive datasets. This suggests the fine-tuning data's style transfers to the model's generation behavior, while pre-training alone (with its sentence-masking objective) may bias toward more extractive behavior.
Sinusoidal positional encoding generalization (Section 6.2): The paper confirms that sinusoidal positional encodings generalize to input lengths up to 1,024 tokens during fine-tuning, even though pre-training used a maximum of 512 tokens. This is validated by PEGASUS_LARGE's strong performance on long-document datasets (BIGPATENT, arXiv, PubMed, Multi-News), but the authors note that further scaling L_input could improve performance on datasets where average input length exceeds 1,024 tokens.

Critical Assessment

The central claim of the paper—that a pre-training objective tailored for abstractive summarization (GSG with importance-based sentence selection) yields state-of-the-art performance across diverse domains—is well-supported by the comprehensive evaluation across 12 datasets and the consistent outperformance of both random and lead-based selection in ablation studies. However, several nuances and limitations warrant careful consideration.

Does GSG itself drive the gains, or is it simply sentence-level masking? The ablation comparing Random GSG to Ind-Orig GSG (Figure 4a) shows that importance-based selection provides a consistent but modest improvement—typically 0.3–0.5 ROUGE points. The gap between Random GSG and the Transformer_BASE (no pre-training) is far larger, suggesting that most of PEGASUS's benefit comes from any form of sentence-level masking and generation, not specifically from the importance-based selection. The paper's claim that choosing "putatively important sentences outperforms lead or randomly selected ones" is true, but the magnitude of the advantage is small enough that for many practical purposes, Random GSG might suffice—especially since it avoids the computational cost of computing ROUGE scores for every sentence in every document during pre-training data preparation. The paper does not report the preprocessing cost of Ind-Orig versus Random, which would be valuable for practitioners deciding which variant to implement.

Is the evaluation comprehensive enough to claim state-of-the-art? The paper evaluates on 12 datasets, which is commendably broad. However, all evaluations use only ROUGE scores—an automatic metric known to correlate imperfectly with human judgments of summary quality (as the authors themselves acknowledge). The human evaluation covers only 3 of the 12 datasets (XSum, CNN/DailyMail, Reddit TIFU) with 100 examples each, leaving open the question of whether the ROUGE improvements on the other 9 datasets translate to actual quality improvements. On CNN/DailyMail, the human evaluation actually shows PEGASUS_LARGE (HugeNews) rated higher than human references (3.6 vs. 3.1, p = 0.0001), which could indicate either genuinely superior summaries or a bias in the evaluation setup (e.g., workers preferring more extractive summaries, which the model tends to produce per Appendix H). Without broader human evaluation, the ROUGE-based state-of-the-art claims should be interpreted cautiously.

The pre-training corpus effect is not fully disentangled from model scale. Figure 3 compares C4 (350M documents, 750GB) and HugeNews (1.5B documents, 3.8TB), which differ in both domain and size. The finding that HugeNews helps news tasks could be due to domain alignment, corpus size, or both. A controlled experiment pre-training on a size-matched subset of each corpus would isolate the domain effect, but is not reported. Similarly, Table 1 shows PEGASUS_LARGE (HugeNews) generally outperforming PEGASUS_LARGE (C4), but HugeNews is 5× larger—the performance difference could reflect more pre-training data rather than better domain alignment. The mixed model's success (Table 4) suggests both factors matter, but their relative contributions remain unknown.

The low-resource claim of "surpassing previous state-of-the-art results on 6 datasets with only 1000 examples" needs qualification. Table E.1 shows that at 1,000 examples, PEGASUS_LARGE beats prior SOTA on ROUGE-2 for Multi-News (15.45 vs. 14.89), WikiHow (12.17 vs. 9.23), Reddit TIFU (6.85 vs. 3.7), BIGPATENT (12.58 vs. 10.63), AESLC (15.45 vs. 10.29), and BillSum (30.58 vs. 23.83). However, the previous SOTA numbers come from different models with different architectures and training procedures, not from PEGASUS with reduced data, making this a comparison across papers rather than a controlled ablation. The more relevant internal comparison is to Transformer_BASE trained on full data (20k–200k examples): PEGASUS with 100–1,000 examples matches or exceeds this, which is impressive but represents a different claim than "beating prior SOTA."

The ablation studies use PEGASUS_BASE on only 4 of 12 datasets, then scale the best configuration to PEGASUS_LARGE on all 12. The implicit assumption is that the optimal choices for PEGASUS_BASE (Ind-Orig selection, 30% GSR, Unigram 96k vocabulary) transfer to PEGASUS_LARGE. Table 1 does show PEGASUS_LARGE performing well, but there is no ablation at the large scale to confirm that, for example, the optimal GSR for PEGASUS_LARGE is the same as for PEGASUS_BASE, or that Ind-Orig remains the best selection strategy at 568M parameters. The model scale × pre-training objective interaction is untested.

Missing experiments that would strengthen the paper:

Direct comparison to T5 and BART under controlled conditions. Tables 1 and 2 compare PEGASUS to previously reported numbers from other papers, but these comparisons conflate differences in pre-training objective with differences in model architecture, training data, optimization, and fine-tuning hyperparameters. A controlled experiment pre-training PEGASUS with BART's denoising objective or T5's span corruption on the same C4 corpus would isolate the effect of the GSG objective itself.
Ablation of whole-sentence masking vs. equivalent-length span masking. The paper argues that masking whole sentences is better than masking spans because it encourages "whole-document understanding." A controlled experiment comparing GSG with whole sentences to a variant that masks contiguous text of the same total length (but not respecting sentence boundaries) would test this claim directly. Without this ablation, it's unclear whether the benefit comes from sentence-level structure or simply from the longer masked segments.
Difficulty estimation cost. The paper does not account for the computational cost of computing ROUGE1-F1 for every sentence in every pre-training document when using Ind-Orig. For C4 (350M documents), this preprocessing step could be substantial. A comparison of wall-clock time or FLOPs for pre-training data preparation across selection strategies would inform practical adoption.
Evaluation on generation quality beyond ROUGE. Metrics like BERTScore, factual consistency (e.g., using entailment models), or repetition/degeneration measures would provide a more complete picture of summary quality, particularly given the known limitations of ROUGE for evaluating abstractive summaries.

The single model family limitation. All experiments use the PEGASUS architecture with PaLM-derived training. The claim that GSG "works well as a pre-training objective for downstream summarization tasks" (Abstract) is validated only for this specific architecture and training setup. Whether GSG would benefit, say, BART or T5 models trained with their own pipelines is untested.

The human evaluation sample size and design. Only 100 examples per dataset, with 3 workers each, totaling 300 judgments per model-dataset pair. For complex tasks like summarization evaluation, this is a relatively small sample. The finding on CNN/DailyMail that PEGASUS outperforms human references (3.6 vs. 3.1) is intriguing but should be replicated with larger samples and more rigorous evaluation protocols before being accepted as evidence that the model surpasses human performance. The paper does not discuss inter-annotator agreement, which would indicate the reliability of the ratings.

Gigaword results are relatively weak. PEGASUS_LARGE achieves 39.12/19.86/36.24 on Gigaword, comparable to MASS and UniLM (Table 2) but not clearly state-of-the-art. This is notable because Gigaword is the only sentence-level summarization (headline generation) dataset among the 12 tasks, and it is also where GSG—which masks multiple whole sentences—is least well-aligned with the downstream task. This pattern actually supports the paper's central thesis (pre-training alignment matters) but also reveals a limitation: GSG is designed for multi-sentence document summarization, not single-sentence compression.

Overall, the experimental evidence strongly supports the claim that GSG pre-training with importance-based sentence selection produces effective summarization models, particularly for low-resource settings and diverse domains. The state-of-the-art claims across 12 datasets are numerically supported but should be interpreted in light of the reliance on ROUGE, the limited human evaluation coverage, and the uncontrolled comparisons to prior models with different architectures and training regimes.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Factored Into the Headline Efficiency Gains

The paper's central claim of 4× efficiency improvement from compute-optimal test-time scaling rests on knowing each prompt's difficulty before allocating the inference budget. The method for estimating difficulty—generating 2,048 samples per question and averaging either ground-truth correctness or PRM final-answer scores—consumes more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

Consequence. In any real deployment, the total cost is difficulty estimation plus strategy execution. Because the estimation cost (2,048 generations per prompt) is ~8–16× larger than the typical test-time budgets where the largest relative gains are demonstrated (16–64 generations), the realized efficiency gain would be dramatically lower—potentially negative—once the estimation overhead is amortized. The 4× figure (e.g., matching best-of-64 with 16 generations, Figure 4; matching best-of-256 with 64 generations, Figure 8) is therefore an upper bound on achievable efficiency that is unreachable without a cheaper difficulty estimator.

Evidence in the paper. The difficulty estimation procedure is described in Section 3.2. The generation budgets for search and revision experiments (Figures 3–8) range from 1 to 512 generations. The difficulty estimation cost of 2,048 generations per question is never included in any budget calculation reported in the figures or tables. The predicted difficulty variant (using PRM final-answer scores rather than ground-truth labels, Figures 4 and 8) still requires generating the same 2,048 samples—it only removes the need for ground-truth answers, not the sampling cost.

Mitigation status. The authors flag this as a key area for future work: "A natural direction for future work is pre-training or fine-tuning models to directly predict the difficulty of a question without requiring samples" (Section 8). No cheaper difficulty estimator is developed or evaluated in the paper. An alternative approach—adaptive difficulty estimation where a small number of initial samples (4–8) provides a difficulty signal and the remaining budget is allocated accordingly—would more closely align with the exploration-exploitation framing the authors mention (Section 3.2) but is not tested. Until a practical difficulty estimator exists, the compute-optimal framework remains a proof-of-concept rather than a deployable system.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with 500 test questions, and all models are variants of PaLM 2-S* (Anil et al., 2023). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but no evidence is provided to support this belief.

Consequence. Several aspects of the findings could be model-specific or benchmark-specific in ways that affect the generalizability of the conclusions:

PRM quality and over-optimization behavior. The PRM's calibration, its susceptibility to exploitation under aggressive search, and the specific budget thresholds at which over-optimization degrades performance depend on the base model's output distribution and error patterns. A different model family (e.g., GPT-4, LLaMA, Claude) with different calibration properties might exhibit different optimal strategy choices at the same difficulty levels.
Revision model training stability. The revision model's ability to learn from incorrect in-context examples and the specific 38% correct-to-incorrect reversion rate (Section 6.1) are tied to PaLM 2-S*'s in-context learning capabilities and the specific fine-tuning recipe. The ReST^EM failure (Appendix K, Figure 16) shows this is sensitive to training methodology even within the same model family.
MATH is exclusively symbolic reasoning. The benchmark consists of competition-level math problems requiring multi-step logical deduction. Whether the difficulty-dependent patterns—beam search hurting easy problems (Figure 3, right), revisions helping easy problems (Figure 7, right), no method helping the hardest problems (bin 5)—generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference is unknown. The paper does not discuss how the approach might transfer to domains where "correctness" is ambiguous or multi-dimensional.

Evidence in the paper. Section 4 describes the choice of MATH and PaLM 2-S*. All results in Sections 5–7 report performance on the MATH test set and the FLOPs-matched comparison uses a ~14× larger PaLM-family model. There are no experiments on other benchmarks or model families.

Mitigation status. None. The authors acknowledge this implicitly by framing their model as "representative" (Section 4) but do not test this claim. Replication on at least one additional benchmark (e.g., GSM8K for math, HumanEval for code, or a logical reasoning dataset) and with at least one non-PaLM model would be necessary to establish the findings as general principles rather than PaLM-specific phenomena.

Hard Problems (Difficulty Bin 5) Show Near-Zero Improvement Regardless of Compute Budget

Across all methods studied—PRM search (Section 5), iterative revisions (Section 6), and their compute-optimal combinations—the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show essentially no improvement with additional test-time compute.

Consequence. Test-time compute amplifies existing capability but does not create it from nothing. This is a fundamental boundary on the approach: if the base model cannot produce any correct solutions in its output distribution (even at very low rates), no amount of search, revision, or adaptive allocation will help. For problems where the correct reasoning is genuinely outside the model's training distribution—novel problem types, unseen reasoning patterns, or tasks requiring capabilities the model simply does not possess—the approach offers no benefit.

This is not a minor edge case. In real-world deployments, the problems that most need improvement are often exactly those where the base model struggles—genuinely novel or difficult queries. The compute-optimal framework is most beneficial for medium-difficulty problems where the model sometimes gets the right answer, which may not be the deployment regime where gains are most needed.

Evidence in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all search methods and all budgets. Figure 7 (right) shows bin 5 at roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% across all compute budgets in the FLOPs-matched comparison. The authors are transparent about this:

"On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3)

Mitigation status. The paper acknowledges this limitation explicitly but offers no solution. The FLOPs-matched comparison (Section 7) quantifies the boundary: on hard problems at R ≫ 1, pretraining is always preferable, with test-time compute showing -37.2% (revisions) to -52.9% (PRM search) relative disadvantage compared to the ~14× larger model. For problems outside the base model's capability range, the authors' recommendation is clear—scale pretraining instead—but the paper provides no guidance on how to identify such problems before investing test-time compute, or whether intermediate strategies (e.g., using the test-time compute budget to call external tools or retrieve relevant knowledge) could bridge the capability gap.

The ~14× Larger Model Baseline for the FLOPs-Matched Comparison Is Not Compute-Optimally Trained

The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a model with ~14× more parameters, but the larger model is trained by scaling only parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Consequence. A Chinchilla-optimal model (Hoffmann et al., 2022), where both parameters and training tokens are scaled in proportion to the FLOPs budget, would likely outperform a parameter-only-scaled model at the same total training FLOPs. This means the pretraining baseline is weaker than it would be under optimal training, potentially inflating the apparent advantage of test-time compute. The reported numbers—e.g., +27.8% relative improvement for revisions on medium problems at R ≪ 1 (Figure 1 bar chart)—may shrink or reverse against a properly compute-optimal larger model.

Additionally, the ~14× larger model uses only greedy decoding with no test-time augmentation of its own. No majority voting, no best-of-N, no search. This is an asymmetric comparison: the smaller model gets sophisticated test-time compute allocation while the larger model gets none. A fairer comparison would give the larger model some test-time compute budget (even a modest amount like best-of-8) and compare total FLOPs.

Evidence in the paper. Section 7 describes the FLOP accounting and the choice to scale parameters only. Figures 1 and 9 report the comparison results. The larger model's performance is shown as stars in Figure 9, placed at three x-axis positions corresponding to the three R values (0.16, 0.79, 22).

Mitigation status. The authors acknowledge the limitation and frame the parameter-only scaling as "representative of a canonical approach," not optimal. The missing comparison—compute-optimal test-time compute with a smaller model vs. compute-optimal pretraining of a larger model—would require training multiple Chinchilla-optimal models at different scales, which is computationally prohibitive. However, even a weaker correction—e.g., estimating Chinchilla-optimal performance from scaling laws and comparing to that curve—would provide a more informative baseline than the current parameter-only-scaled model. The paper leaves this to future work.

The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate

When generating sequential revisions, approximately 38% of correct answers produced at one step get "revised" back to incorrect answers in the next step (Section 6.1). This is a direct consequence of the training data construction: the model is trained exclusively on sequences where all in-context answers are incorrect (followed by a correct target), so it never learns the behavior of recognizing that the current answer is already correct and should be preserved.

Consequence. The revision chain is unstable: even when the model produces a correct answer, it is more likely than not to corrupt it in the subsequent revision. This means the system cannot simply take the last output in a revision chain as the final answer—it must employ selection mechanisms (majority voting or verifier-based selection across the entire chain) to recover the correct answer from earlier steps. These selection mechanisms add complexity, require additional verifier training (the base PRM does not transfer well to revision outputs; Appendix J, Figure 15a), and are imperfect patches rather than principled solutions. The 38% reversion rate places an upper bound on the benefit of deeper revision chains, since each additional step may undo previous gains.

Evidence in the paper. Section 6.1 reports the 38% figure. The need for within-chain selection is described: "To mitigate this, the system uses a selection mechanism (majority voting or verifier-based selection) across the entire chain of revisions, picking the best answer from any point in the chain rather than always taking the last revision." Appendix J (Figure 15a) shows that the base-language-model PRM underperforms when scoring revision model outputs, necessitating a revision-specific ORM. The ReST^EM failure (Appendix K, Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with RL-style training caused performance to degrade substantially, with fully sequential performance dropping to ~33.5% compared to ~38.5% at the optimal ratio.

Mitigation status. The paper applies within-chain selection as a workaround rather than solving the root cause. The authors do not explore training the revision model to recognize correct answers and abstain from revising them—a natural improvement that would involve including correct-to-correct trajectories in the training data. The paper does suggest (in a different context) that future work could explore training the model to "recognize when no revision is needed," but this is not the framing used in Section 6. The 38% reversion rate remains a fundamental limitation of the current revision model design, and the paper provides no evidence on how much the within-chain selection mechanisms recover versus what an ideal (non-reverting) revision model could achieve.

Search and Revisions Are Studied Independently, Not Combined

The paper studies two complementary axes—PRM tree-search (Section 5) and iterative revisions (Section 6)—but never combines them. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

Consequence. The two mechanisms address complementary weaknesses: revisions improve the proposal distribution (generating better candidate solutions by conditioning on prior attempts), while PRM search improves candidate selection (identifying the best among generated candidates through verifier-guided exploration). The paper's results therefore represent a lower bound on what a unified system could achieve. The difficulty-dependent findings suggest natural combinations: on medium-difficulty problems, using the revision model as the proposal distribution within beam search—where at each step the model conditions on previous rejected branches as context—could produce higher-quality candidates than either method alone. On easy problems, where revisions help but search over-optimizes, using the PRM only for within-chain selection rather than tree exploration could avoid over-optimization while still capturing the revision benefit.

The paper's compute-optimal policies are separately optimized for search (Figure 4) and revisions (Figure 8), with no joint policy that selects between or combines the two approaches per difficulty bin. This means the reported 4× efficiency gains are for each axis independently, and a combined system might achieve substantially better efficiency—or reveal interference effects that reduce gains.

Evidence in the paper. Sections 5 and 6 are entirely independent: Section 5 uses the few-shot prompted base model with PRM search, Section 6 uses the revision model with within-chain selection and a revision-specific ORM. There is no experiment where the revision model generates candidates and the PRM performs beam search over them, or where the PRM guides which revision to pursue. The computational budgets for both axes are reported separately, with no FLOPs accounting for a combined system.

Mitigation status. The authors acknowledge the gap and note that combining the two is "a natural next step" (Section 8). No preliminary experiments or analysis of potential interactions are provided. The absence of combined experiments is a significant gap because the paper's central argument is that the proposal distribution and verifier represent distinct scaling axes (Section 2), and the natural culmination of this argument is a system that jointly optimizes both. Until such experiments are performed, the paper demonstrates that each axis independently improves efficiency, but cannot address whether the improvements are additive, sub-additive, or synergistic when combined.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the pre-training paradigm for text generation from generic reconstruction toward task-aware objective design, establishing that the choice of what to mask and generate during pre-training—not just the architecture or dataset scale—can be optimized for specific downstream tasks. Before PEGASUS, the dominant approach was to pre-train sequence-to-sequence models as denoising autoencoders: corrupt random spans or tokens, then reconstruct the original. The implicit assumption was that learning to undo arbitrary noise produces broadly useful representations, and that task-specific adaptation happens entirely during fine-tuning. PEGASUS challenges this assumption with a simple but powerful reframing: design the pre-training objective so that the self-supervised task itself teaches the skill the downstream task requires.

The reframing is not a paradigm shift on the scale of the Transformer architecture or large-scale pre-training itself, but it represents a significant conceptual refinement in how pre-training objectives are designed. Prior work treated masking as a mechanism for creating a self-supervised training signal—any corruption that forces the model to understand context is equally valid. PEGASUS demonstrates that the selection criterion for what gets masked is a design dimension with measurable performance implications, and that aligning this criterion with the downstream task—using ROUGE-based importance to select summary-like gap sentences—produces better transfer than random or heuristic selection. The effect size is modest for the selection strategy itself (Ind-Orig vs. Random GSG yields ~0.3–0.5 ROUGE points; Figure 4a), but the broader demonstration that task-aware design is possible and beneficial opens a design space that prior work had not explored.

This work reconciles a latent tension in the pre-training literature between scale and specificity. The success of BERT, T5, and GPT-series models had created an implicit narrative that larger models trained on more data with generic objectives would dominate, and that task-specific architectural or objective design was unnecessary. PEGASUS complicates this picture: GSG with Ind-Orig selection—a summarization-specific pre-training objective—produces a 568M parameter model that matches or exceeds T5 (11B parameters) and BART on summarization tasks (Tables 1 and 2), suggesting that objective design can substitute for some fraction of model scale. This does not mean scale is unimportant—the paper itself shows that PEGASUS_LARGE substantially outperforms PEGASUS_BASE (Table 1)—but it does mean that the returns to scale and the returns to objective design are not independent, and that ignoring the latter leaves performance on the table.

The paper also resolves the apparent contradiction between MASS and BART on one hand (which showed that masking longer spans helps generation) and the intuition that summarization requires identifying important content. MASS masks random sentence fragments; BART masks random spans. Both work because any form of masking teaches the model to generate coherent text from partial context. PEGASUS shows that making the masking non-random—selecting important sentences—adds a second learning signal: the model learns not just to generate from context, but to recognize what content tends to be central to a document. The previously separate goals of content selection (traditionally the domain of extractive summarizers) and abstractive generation (traditionally the domain of sequence-to-sequence models) are unified into a single pre-training objective. This reframing makes research directions that separate content selection from generation—e.g., extract-then-abstract pipelines—less attractive, because a well-designed pre-training objective can teach both simultaneously.

Perhaps the most broadly significant finding is the domain-specificity of pre-training corpus effects (Figure 3). Prior work treated the choice of pre-training corpus as an engineering decision: use the largest, most diverse corpus available, with the assumption that more data = better transfer across all tasks. PEGASUS provides clear counter-evidence: HugeNews (1.5B news articles) outperforms C4 (350M web pages) on news summarization but underperforms on informal domains (WikiHow, Reddit TIFU). This transforms corpus selection from a scale-maximization problem into a domain-adaptation problem that can be optimized per deployment. The practical implication is that organizations building summarization systems for specific domains (medical, legal, financial) should prioritize collecting domain-relevant pre-training data over simply using the largest available general corpus.

The low-resource results (Section 6.3, Figure 6) change the conversation around what is achievable with limited supervision. PEGASUS_LARGE with 100 fine-tuning examples matches or exceeds Transformer_BASE trained on full datasets of 20k–200k examples on 8 of 12 datasets, and beats prior state-of-the-art on 6 datasets with only 1,000 examples. This establishes that pre-training objective design is a sample-efficiency multiplier, not just a performance enhancer. For practitioners in domains where labeled summarization data is scarce—which is nearly all real-world domains outside news—this finding makes the case for investing in pre-training methodology over (or in addition to) annotation efforts.

Follow-Up Research This Work Enables

Controlled comparison of GSG versus denoising objectives on identical hardware and data. The paper compares PEGASUS to BART, T5, MASS, and UniLM using previously reported numbers (Tables 1 and 2), which confounds differences in pre-training objective with differences in architecture, training data, optimization, and fine-tuning procedure. A controlled experiment would take a single architecture (e.g., PEGASUS_LARGE), pre-train it on the same C4 corpus for the same number of steps, but vary only the pre-training objective: GSG (Ind-Orig), BART-style text infilling, T5-style span corruption, and MASS-style fragment reconstruction. The experiment would isolate how much of PEGASUS's advantage comes from the objective itself versus other implementation choices. A strong result would show GSG outperforming the alternatives in the low-resource regime (100–1,000 examples) by a wider margin than in the full-resource regime, confirming that task-aligned pre-training is most valuable when supervised data is scarce.

Scaling GSG to 10× more pre-training data and measuring saturation. The paper shows that pre-training performance improves monotonically through 500k steps (Appendix B, Figure B.1) with no clear saturation. The PEGASUS_LARGE (mixed, stochastic) model at 1.5M steps shows further gains (Table 4). An open question is whether GSG's advantage over random masking changes with pre-training scale. It is possible that with sufficient data and model capacity, even random sentence masking converges to the same representations as importance-based masking—the model might implicitly learn to identify important sentences from the pre-training data distribution. Alternatively, the gap might persist or widen, suggesting that GSG provides an inductive bias that scale alone cannot replace. A scaling experiment training PEGASUS_BASE with both Random GSG and Ind-Orig GSG on increasing amounts of C4 data (100k to 5M steps) would characterize this interaction. If the gap narrows, the practical conclusion is that importance-based selection matters most in data-constrained pre-training regimes; if it persists, Ind-Orig should be preferred regardless of compute budget.

Comparing ROUGE-based importance to alternative salience signals. The paper uses ROUGE-1 F1 between each sentence and the rest of the document as a proxy for importance. This is one of many possible unsupervised salience measures. Alternatives include: (1) TextRank/LexRank graph-based centrality, which captures sentence importance based on the structure of similarity relationships across all sentences, not just pairwise similarity to the rest of the document; (2) TF-IDF weighted ROUGE, which down-weights common words that contribute spuriously to ROUGE scores; (3) Supervised salience prediction using extractive summarization datasets (e.g., CNN/DailyMail extractive labels), which would inject ground-truth importance signals into the pre-training data construction; (4) Entailment-based importance, where sentences that entail or are entailed by many other sentences are considered central. A systematic comparison would train PEGASUS_BASE using gap sentences selected by each method and evaluate on the same four downstream datasets (XSum, CNN/DailyMail, WikiHow, Reddit TIFU). A finding that one salience measure consistently outperforms ROUGE-1 would directly improve the GSG objective. A finding that all measures perform similarly would suggest that the benefit comes from any principled selection over random masking, not from the specific salience definition.

Combining GSG with domain-adaptive pre-training for specialized summarization. The paper shows that pre-training corpus domain alignment matters (Figure 3), but does not explore the interaction with the GSG objective. A natural follow-up tests whether GSG with importance-based selection is more or less sensitive to domain mismatch than random masking. Hypothesis: importance-based selection might be more domain-sensitive because the ROUGE-based importance heuristic encodes assumptions about document structure (e.g., that newspapers front-load important information) that may not hold in other domains (e.g., scientific papers where the abstract already summarizes the paper, or patents where claims are distributed throughout). An experiment would pre-train PEGASUS_BASE with both Random GSG and Ind-Orig GSG on three corpora—news (HugeNews), scientific (a corpus of arXiv/PubMed papers), and legal (a corpus of legislation and case law)—and evaluate on six downstream datasets spanning all three domains. If Ind-Orig underperforms Random when the pre-training and downstream domains are mismatched, this would refine the paper's recommendation: use importance-based selection when the pre-training corpus is domain-aligned with the target task, but fall back to random selection when domain mismatch is unavoidable.

Extending GSG to multi-document and query-focused summarization. The paper evaluates on single-document summarization, but the Multi-News results (Table 1: R1: 47.52, R2: 18.72, RL: 24.91) show that the pre-trained model transfers to multi-document settings with fine-tuning. However, GSG as formulated does not explicitly train for multi-document synthesis—each pre-training example involves a single document, and the gap sentences are selected from that document only. A natural extension would modify GSG for multi-document pre-training: given a set of related documents (e.g., multiple news articles on the same event, identified via URL clustering in the HugeNews corpus), select important sentences across documents and train the model to generate them from the union of remaining sentences. This would explicitly teach cross-document content selection and redundancy removal. Evaluation on Multi-News and a newly constructed multi-document test set would test whether this modified objective improves over standard GSG for the multi-document case.

Measuring factual consistency of PEGASUS summaries against reference-based metrics. The paper evaluates exclusively with ROUGE (automatic) and Likert-scale human ratings (manual). Neither measures factual consistency—whether the generated summary makes claims that are supported by the source document. ROUGE correlates poorly with factual accuracy (a summary can have high n-gram overlap with a reference but hallucinate details, or be factually perfect but use different wording). A follow-up evaluation would apply entailment-based factual consistency metrics (e.g., using NLI models to check whether each sentence in the summary is entailed by the source document) to PEGASUS outputs across all 12 datasets, and compare against BART and T5 summaries. The paper's observation that PEGASUS summaries are "less abstractive" than human references (Appendix H, Figure H.1)—with higher extractive coverage and density—suggests PEGASUS may trade abstractiveness for faithfulness, producing safer but less novel summaries. A factual consistency evaluation would test this hypothesis directly. If PEGASUS summaries are more faithful than BART/T5, this would be an additional advantage not captured by ROUGE. If they are comparably faithful, the extractiveness observation would simply reflect a stylistic tendency.

Practical Applications and Downstream Use Cases

Low-resource domain-specific summarization. The most directly actionable finding from the paper is that PEGASUS fine-tuned on 100–1,000 examples produces summaries competitive with a fully-supervised non-pretrained Transformer (Figure 6) and surpasses prior state-of-the-art on 6 of 12 datasets at 1,000 examples (Table E.1). For organizations building internal summarization systems—legal document summarization at a law firm, clinical note summarization at a hospital, internal report summarization at a corporation—this means that collecting even a small number of in-domain summaries (a few hundred, which could be produced by subject-matter experts in weeks rather than months) is sufficient to deploy a high-quality system, provided they use PEGASUS pre-trained on a domain-aligned corpus. The paper's finding that domain alignment of the pre-training corpus matters (Figure 3) implies that assembling a domain-specific pre-training corpus—even a modest one by web-scale standards—is likely more valuable than using a larger but unrelated corpus. A legal deployment would benefit from pre-training on a corpus of legislation and case law; a medical deployment from a corpus of clinical notes and journal abstracts.

Cost-efficient headline and TL;DR generation for content platforms. On Gigaword (headline generation from the first sentence of news articles), PEGASUS achieves ROUGE-2 of 19.86 (Table 1). On Reddit TIFU (generating TL;DR summaries from long-form informal posts), PEGASUS achieves ROUGE-2 of 9.01 with full supervision and 6.85 with only 1,000 examples (Table E.1). These are tasks where output length is very short (typically 1–2 sentences) and stylistic fidelity matters. Content platforms (news aggregators, social media platforms, internal knowledge bases) could deploy PEGASUS to auto-generate headlines or TL;DR summaries for user-generated or ingested content, reducing the need for human editors. The paper's human evaluation shows that on Reddit TIFU, full supervision is needed to match human quality (Table 3, Experiment 2: 1,000 examples score 2.7 vs. human 3.3, p = 0.01; full supervision scores 2.8 vs. human 3.3, p = 0.05, not significant at p < 0.01), suggesting that for stylistic diversity, more supervised data is needed, but the baseline quality at 1,000 examples already exceeds prior state-of-the-art.

Multi-document summarization for news aggregation and competitive intelligence. PEGASUS achieves ROUGE-2 of 18.72 on Multi-News (Table 1), the multi-document summarization dataset, surpassing the previous SOTA of 14.89. This task—generating a single coherent summary from multiple news articles on the same event—is directly applicable to news aggregation services, financial intelligence platforms tracking company mentions across multiple sources, and government/military intelligence analysis. The finding that pre-training on a large news corpus (HugeNews) transfers strongly to news summarization tasks (Figure 3) means that a news-domain PEGASUS model is immediately deployable for this use case with domain-specific fine-tuning on as few as 1,000 multi-document summary examples (Table E.1 shows ROUGE-2 of 15.45 at 1,000 examples on Multi-News, already above the prior SOTA).

Email and patent summarization in enterprise settings. On AESLC (email subject line generation), PEGASUS achieves ROUGE-2 of 21.85 (Table 1), nearly doubling the previous SOTA of 10.29. On BIGPATENT (patent summarization), ROUGE-2 reaches 33.16 (Table 1) with C4 pre-training and 32.89 with HugeNews, substantially above the prior SOTA of 10.63. Both tasks are highly relevant to enterprise deployments: email summarization for productivity tools (auto-generating subject lines or brief summaries of email threads) and patent summarization for intellectual property management (automatically summarizing new patent filings for competitive analysis). The dramatically better performance on these tasks compared to prior work—particularly the ~3× improvement on BIGPATENT—makes PEGASUS the clear choice for practitioners in these domains. For email summarization, the low-resource results are especially important: with only 1,000 examples, PEGASUS achieves ROUGE-2 of 15.45 (Table E.1), already 50% above the prior SOTA, making deployment feasible even for organizations with limited labeled email data. For patent summarization, the paper notes that average input lengths exceed 1,024 tokens (Section 6.2), suggesting that further scaling the maximum input length or applying a two-stage approach could yield additional gains beyond the already-state-of-the-art results.