BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

ArXiv: 1910.13461

🎯 Pitch

BART smashes the wall between understanding and generation models: a single pretrained denoising autoencoder matches RoBERTa on comprehension tasks while absolutely crushing prior art on text generation—for instance, a 6 ROUGE leap on abstractive summarization. The secret is training a standard Transformer to reconstruct text that was corrupted by both sentence shuffling and a novel span-infilling noising scheme, forcing the model to learn high-level reasoning about overall document structure.

1. Executive Summary

This paper introduces BART, a denoising autoencoder for pretraining sequence-to-sequence models that learns to reconstruct original text from corrupted versions using an arbitrary noising function. The model uses a standard Transformer architecture—a bidirectional encoder paired with an autoregressive decoder—making it a generalization of both BERT's bidirectional encoding and GPT's left-to-right generation within a single unified framework. BART is trained by corrupting documents with a combination of text infilling (replacing arbitrary-length spans with a single mask token) and sentence permutation (shuffling sentence order), then optimizing the reconstruction cross-entropy between the decoder's output and the uncorrupted original. It matches RoBERTa's performance on GLUE and SQuAD discriminative benchmarks, achieves new state-of-the-art results on abstractive generation tasks—including gains of up to 6 ROUGE on the XSum summarization dataset and a 1.1 BLEU improvement on WMT Romanian-English machine translation—establishing that a single pretrained sequence-to-sequence model can be simultaneously effective for both understanding and generation, with the noising flexibility being the key mechanism that enables this dual capability.

2. Context and Motivation

The Core Problem: A Fractured Pretraining Landscape

In 2019, the field of NLP pretraining was simultaneously flourishing and fractured. Self-supervised pretraining had become the dominant paradigm, but different architectural choices led to models that excelled at either understanding tasks or generation tasks—rarely both at the same time. The paper addresses this fundamental gap: how can we design a single pretrained model that achieves state-of-the-art performance on both discriminative tasks (like classification and question answering) and generative tasks (like summarization and dialogue)?

This is not merely an aesthetic preference for unification. The practical consequence of the split is that practitioners must choose a model architecture based on their end task, pretraining separate models for understanding and generation, or accept suboptimal performance on one side of the divide. A model that natively handles both categories with high performance would simplify deployment pipelines, reduce the engineering overhead of maintaining multiple pretrained models, and open up applications where understanding and generation must be tightly coupled (e.g., reading a document and then summarizing it, or comprehending a dialogue context and generating an appropriate response).

The problem is architectural at its core. As of late 2019, the two dominant pretraining paradigms had fundamental structural limitations:

BERT (Devlin et al., 2019) uses a bidirectional Transformer encoder that can attend to both left and right context at every layer. This enables deep bidirectional understanding, making it exceptionally strong on tasks like classification, extractive question answering, and natural language inference. But BERT is not a generative model—its masked language modeling objective predicts missing tokens independently (conditionally independent given the context), not autoregressively. There is no mechanism for generating coherent multi-token sequences left-to-right. To use BERT for generation, you would need to either (a) repeatedly mask and predict tokens in some iterative scheme, or (b) use it only as an encoder and attach a separately trained decoder—neither of which is a natural fit for the way the model was pretrained.
GPT (Radford et al., 2018) uses a left-to-right autoregressive Transformer decoder. It is a natural generative model—samples are produced token by token, each conditioned on all previously generated tokens. But GPT's architecture is unidirectional: every token can only attend to tokens that precede it in the sequence. This means the model cannot learn bidirectional interactions during pretraining. While GPT demonstrated impressive generation capabilities, particularly at scale (Radford et al., 2019), its unidirectional nature makes it inherently weaker on tasks that require integrating information from both sides of a token, such as span extraction (SQuAD) or natural language inference where understanding the full relationship between two sentences matters.

The authors frame this problem visually in Figure 1 of the paper, which juxtaposes BERT's bidirectional encoder (independent masked token prediction), GPT's autoregressive decoder (left-to-right generation but no bidirectional context), and BART's combined architecture (bidirectional encoder over corrupted input + autoregressive decoder generating the reconstruction). BART's design is explicitly positioned as unifying these two paradigms.

Why This Matters: The Generation Gap

The paper is motivated by a specific observation that was becoming increasingly apparent in the 2018–2019 NLP literature: denoising autoencoder pretraining (particularly BERT-style masked language modeling) was dramatically improving understanding tasks, but generation tasks were not seeing commensurate gains from the same approaches.

This "generation gap" had real practical significance. Tasks like abstractive summarization, dialogue response generation, and long-form question answering require models not just to classify or extract spans, but to produce fluent, coherent, contextually appropriate text that may differ substantially from the input. These tasks were seeing improvements from encoder-decoder architectures (See et al., 2017), but those models typically relied on task-specific architectures and training procedures rather than benefiting from the massive pretraining that was revolutionizing classification benchmarks.

Conversely, the models that did benefit from large-scale pretraining (BERT and its variants) were architecturally incapable of generation without significant modifications. There was an asymmetry: pretraining was solving understanding problems, but generation remained largely in the realm of task-specific supervised training. The authors saw this as an architectural problem rather than a fundamental limitation of pretraining—what was needed was a pretraining objective and architecture that matched the generation setting from the start.

The numbers in the paper quantify this gap. In Table 1, the Masked Language Model (a BERT-style objective reimplemented in a controlled setting) achieves strong SQuAD F1 (90.0) and MNLI accuracy (83.5), but relatively poor perplexities on generation tasks: 7.87 on XSum and 12.59 on ConvAI2. Meanwhile, a pure Language Model (GPT-style) achieves much better generation perplexities (7.00 on XSum, 11.51 on ConvAI2) but terrible SQuAD performance (76.7 F1). No approach in the existing literature was simultaneously competitive on both types of tasks.

Prior Approaches and Where They Fell Short

The paper surveys several approaches that had been proposed before BART, each of which partially addressed the unification problem but had specific limitations:

BERT's Bidirectional Limitation. BERT (Devlin et al., 2019) trains a bidirectional encoder to predict randomly masked tokens. The key limitation for generation is architectural: predictions for different masked positions are made independently given the bidirectional context. There is no autoregressive chain of conditioning. As the paper states (Section 1, Figure 1a caption): "Missing tokens are predicted independently, so BERT cannot easily be used for generation." To generate text with BERT requires iterative masked prediction or separate decoder training, both of which create a mismatch between pretraining and generation conditions.

GPT's Unidirectional Limitation. GPT (Radford et al., 2018) is a pure left-to-right language model. While naturally suited for generation, it suffers from a fundamental asymmetry: "words can only condition on leftward context, so it cannot learn bidirectional interactions" (Figure 1b caption). This limits its effectiveness on tasks requiring integration of full context, particularly extractive question answering (SQuAD) where knowing what follows a candidate answer span is as important as knowing what precedes it. Table 1 shows this empirically: the Language Model baseline achieves only 76.7 F1 on SQuAD, dramatically worse than the bidirectional approaches.

XLNet's Partial Solution with Added Complexity. XLNet (Yang et al., 2019) attacked the unidirectional limitation by pretraining with permuted language modeling—tokens are predicted autoregressively, but the order in which they are predicted is a random permutation of the original sequence. This cleverly allows each predicted token to condition on both left and right context tokens (since some appear earlier in the permuted order), while maintaining an autoregressive factorization. However, this introduces significant complexity: the model requires two-stream self-attention (content and query streams) and relative positional embeddings. More importantly from the perspective of generation, the pretraining setting (predicting in random order) does not match the generation setting (predicting left-to-right), creating a pretraining-finetuning mismatch for generative tasks. The paper notes (Section 4.3): "The Permuted Language Model and the Masked Language Model perform less well than others on generation, and are the only models we consider that do not include left-to-right auto-regressive language modelling during pre-training."

UniLM's Ensemble-of-Masks Approach. UniLM (Dong et al., 2019) took a different approach to unification: fine-tune BERT with an ensemble of different self-attention masks (left-to-right, right-to-left, bidirectional, and a hybrid prefix mask). This allows a single set of parameters to serve multiple pretraining objectives. The authors acknowledge UniLM as a significant step toward unification, but identify a key architectural difference: "UniLM predictions are conditionally independent, whereas BART's are autoregressive" (Section 7). The conditional independence means that during generation tasks, UniLM still cannot model the sequential dependencies between generated tokens as naturally as a true autoregressive decoder.

MASS and Masked Sequence-to-Sequence. MASS (Song et al., 2019) is perhaps the most direct predecessor to BART. It pretrains a sequence-to-sequence model by masking a contiguous span of 50% of tokens in the encoder input and training the decoder to generate only those masked tokens. This introduces autoregressive generation into pretraining and uses a bidirectional encoder. However, MASS has a specific limitation that the paper identifies: it is "less effective for discriminative tasks, because disjoint sets of tokens are fed into the encoder and decoder" (Section 7). The encoder sees only unmasked tokens, and the decoder generates only masked tokens—the two processing streams never handle the same content. This creates a mismatch when fine-tuning for tasks like classification where the same input should be fed to both encoder and decoder.

SpanBERT and Improved Masking Schemes. SpanBERT (Joshi et al., 2019) improved on BERT by masking contiguous spans of tokens rather than individual random tokens, forcing the model to learn span-level representations. BART adopts a related idea (text infilling) but generalizes it: in BART, spans are replaced with a single MASK token regardless of span length, forcing the model to predict not just which tokens are missing, but how many. This is a strictly harder task that the authors argue trains better generative capabilities.

The Measurement Problem: Unfair Comparisons. An important secondary motivation the paper identifies is that comparisons between pretraining methods had been confounded by uncontrolled variables. As the paper states in Section 4: "While many pre-training objectives have been proposed, fair comparisons between these have been difficult to perform, at least in part due to differences in training data, training resources, architectural differences between models, and fine-tuning procedures." Liu et al. (2019) (RoBERTa) had shown that optimization details, data scale, and training duration could matter as much as the choice of pretraining objective. Without controlling for these factors, it was unclear whether reported differences between pretraining methods reflected genuine architectural advantages or implementation details. BART's ablation study in Section 4 addresses this directly by reimplementing multiple pretraining objectives within the same codebase, same data, and same optimization budget.

How BART Positions Itself

BART positions itself not as an incremental improvement over any single predecessor, but as a synthesis framework that unifies the most effective elements of prior work while generalizing beyond them:

1. Architectural unification rather than novel architecture. The authors emphasize that BART uses a "standard Transformer-based neural machine translation architecture" (Section 1)—there is nothing novel about the architecture itself. The innovation is in recognizing that this standard architecture, when paired with the right noising scheme, can simultaneously achieve the benefits of bidirectional encoding (like BERT), autoregressive decoding (like GPT), and span-level reasoning (like SpanBERT). Figure 1 explicitly diagrams this relationship, showing BART as the architectural union of the BERT and GPT approaches.

2. Noising as the key design axis. Rather than proposing a new self-attention mechanism or training objective, BART treats the noising function as the primary design choice. The paper states: "A key advantage of this setup is the noising flexibility; arbitrary transformations can be applied to the original text, including changing its length" (Section 1). This flexibility is what makes BART a general framework rather than a specific recipe. The authors experiment with token masking, token deletion, text infilling, sentence permutation, and document rotation—each represents a different hypothesis about what kind of corruption teaches the most useful representations. The finding that text infilling + sentence permutation works best is an empirical result, not a foregone conclusion, and the authors explicitly note that "there is a significant potential for development of other new alternatives" (Section 2.2).

3. Empirical breadth as validation. BART's claim to generality is backed by evaluating on an unusually wide range of tasks spanning both discriminative (SQuAD, MNLI, GLUE) and generative (XSum, CNN/DM, ConvAI2, ELI5, WMT translation) benchmarks. This breadth is intentional—the paper argues that a truly general pretraining method should work well across the full spectrum, and the controlled ablation in Section 4 demonstrates that individual prior methods show much more variance across tasks than BART does. The authors summarize this finding explicitly: "BART exhibits the most consistently strong performance across the full range of tasks we consider" (Section 1).

4. New ways of using pretrained models. Beyond matching existing benchmarks, BART also enables novel fine-tuning strategies. The machine translation approach described in Section 3.4—using the entire BART model as a pretrained target-side decoder by adding a new source encoder on top—was not possible with BERT or GPT alone. This demonstrates that the sequence-to-sequence architecture is not merely a convenience for pretraining but enables transfer learning paradigms that other architectures preclude. The authors explicitly connect this to prior work's limitations: "Previous work...has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in decoders have been limited" (Section 3.4). BART's architecture allows it to serve as a pretrained decoder, not just encoder.

5. Controlled ablation as scientific contribution. A significant part of how BART positions itself is through its ablation methodology. Rather than simply reporting a single model's performance, Section 4 reimplements Language Model, Permuted Language Model, Masked Language Model, Multitask Masked Language Model, and Masked Seq-to-Seq—all within the same codebase, same data, and same training budget. This allows the paper to make claims about which aspects of the pretraining objective matter most, controlling for confounds that had plagued prior comparisons. The results confirm that left-to-right autoregressive pretraining helps generation (the models without it perform worse on generation tasks), bidirectional encoders are crucial for SQuAD (one-directional models fail here), and token-level masking/deletion is essential (sentence permutation or document rotation in isolation perform poorly). These controlled comparisons give the paper's conclusions more scientific weight than typical single-model benchmark papers.

3. Technical Approach

3.1 Reader Orientation

BART is a method for pretraining a single neural network so that after pretraining, it can be fine-tuned to perform well on both text understanding tasks (like classifying documents or answering questions about a passage) and text generation tasks (like summarizing articles or producing dialogue responses). The core idea is simple: teach the model to reconstruct original text from corrupted versions, using a standard encoder-decoder architecture where the encoder reads the corrupted text bidirectionally and the decoder generates the reconstruction autoregressively, which forces the model to learn both deep bidirectional understanding and fluent left-to-right generation in a single unified training process.

3.2 Big-Picture Architecture (Diagram in Words)

BART has three major components that work together during pretraining and then get reused during fine-tuning:

The Noising Function — an arbitrary text corruption procedure that takes a clean document and transforms it into a corrupted version. This is the primary design axis the paper explores. During pretraining, the noising function can apply any combination of token masking, token deletion, span infilling, sentence permutation, or document rotation. The choice of noising function determines what kind of knowledge the model must learn to recover.
The Bidirectional Encoder — a standard Transformer encoder that reads the corrupted document and produces contextualized representations for every position. Because it uses bidirectional self-attention, each token's representation can incorporate information from both left and right context, enabling deep understanding of the corrupted input. This is architecturally identical to BERT's encoder.
The Autoregressive Decoder — a standard Transformer decoder that generates the original uncorrupted document one token at a time, left to right. At each generation step, the decoder attends to (a) all previously generated tokens via causal self-attention, and (b) all encoder output representations via cross-attention. This gives the decoder access to the full bidirectional context of the corrupted input while maintaining the autoregressive generation capability needed for text generation tasks.

Information flows as follows during pretraining: a clean document enters the noising function → the corrupted document is fed to the bidirectional encoder → the encoder produces hidden representations → the autoregressive decoder generates the reconstruction token by token → the cross-entropy loss between the decoder's output distribution and the original clean document is computed and backpropagated through both encoder and decoder.

During fine-tuning, the noising function is removed. For discriminative tasks, the same clean input is fed to both encoder and decoder, and the decoder's final hidden state for an end-of-sequence token is used for classification. For generative tasks, the encoder reads the input sequence and the decoder generates the output sequence autoregressively, exactly matching the pretraining setup.

3.3 Roadmap for the Deep Dive

First, the sequence-to-sequence architecture and its exact configuration, because understanding how the encoder and decoder interact is prerequisite to understanding why certain noising schemes work and how fine-tuning operates.
Second, the core pretraining objective (negative log-likelihood of the original document) and the formal reconstruction loss, since this is the mathematical optimization that drives all learning.
Third, each noising transformation in detail—token masking, token deletion, text infilling, sentence permutation, and document rotation—because the noising function is the primary design choice and the paper's key empirical finding is that text infilling + sentence permutation works best.
Fourth, the fine-tuning procedures for each task category (sequence classification, token classification, sequence generation, machine translation), since the architecture's flexibility enables fundamentally different fine-tuning strategies for different task types.
Fifth, the controlled ablation methodology that allows fair comparison of pretraining objectives, because this is a significant methodological contribution that gives the paper's conclusions scientific weight beyond a single benchmark result.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core technical contribution is the BART pretraining framework: a sequence-to-sequence denoising autoencoder where the noising function is treated as a flexible design parameter, and the same architecture is used for both pretraining and fine-tuning across a wide range of discriminative and generative tasks. The architecture itself is not novel—it is the standard Transformer from Vaswani et al. (2017). The innovation lies in recognizing that this specific combination of bidirectional encoding and autoregressive decoding, when trained with the right noising scheme, yields a model that generalizes BERT, GPT, SpanBERT, and other approaches within a single framework while achieving state-of-the-art on both understanding and generation.

3.4.1 Sequence-to-Sequence Transformer Architecture

BART uses the standard Transformer sequence-to-sequence architecture from Vaswani et al. (2017) with two modifications inspired by GPT (Radford et al., 2018): ReLU activation functions are replaced with GeLU activations (Hendrycks and Gimpel, 2016), and all parameters are initialized from a normal distribution $\mathcal{N}(0, 0.02)$ .

The architecture consists of two stacks of Transformer layers:

Encoder stack. The encoder is a standard bidirectional Transformer encoder. Each layer consists of multi-head self-attention followed by a position-wise feed-forward network, with residual connections and layer normalization around each sublayer. Because the self-attention is bidirectional (unmasked), every position can attend to every other position in the input. The encoder takes the corrupted document as input and produces a sequence of hidden representations $\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_T$ , where $T$ is the length of the corrupted input. These representations capture the full bidirectional context of the corrupted text.

Decoder stack. The decoder is a standard autoregressive Transformer decoder. Each layer consists of three sublayers: (1) masked (causal) multi-head self-attention over the previously generated output tokens, where each position can only attend to positions before it; (2) multi-head cross-attention over the encoder's output representations, where each decoder position can attend to all encoder positions; and (3) a position-wise feed-forward network. Each sublayer has residual connections and layer normalization. The decoder generates the original document autoregressively, producing tokens one at a time from left to right.

Model sizes. The paper uses two configurations:

Base model: 6 encoder layers and 6 decoder layers, with hidden size 768. Used for the ablation experiments in Section 4.
Large model: 12 encoder layers and 12 decoder layers, with hidden size 1024. Used for the large-scale experiments in Section 5.

Both configurations use 16 attention heads and a feed-forward network dimension of 3072 (base) or 4096 (large), following the standard Transformer architecture sizes.

Differences from BERT. The authors note two specific architectural differences from BERT (Devlin et al., 2019):

Each decoder layer includes cross-attention over the encoder's final hidden layer—this is standard in sequence-to-sequence models but absent from BERT, which has no decoder.
BERT uses an additional feed-forward network (a linear layer + softmax) before word prediction from the final hidden states, while BART does not—it predicts tokens directly from the decoder's output embeddings. This makes BART slightly simpler at the output layer.

The authors note in Section 2.1 that "BART contains roughly 10% more parameters than the equivalently sized BERT model." This is due to the decoder layers (the encoder alone is comparable to BERT's encoder, but BART adds a full decoder stack of equal depth, roughly doubling the layer count while sharing some embedding parameters).

Tokenization and vocabulary. For the large-scale experiments, documents are tokenized using the same byte-pair encoding (BPE) as GPT-2 (Radford et al., 2019). The BPE vocabulary is shared between the encoder and decoder. This is standard practice for sequence-to-sequence models and ensures that the encoder and decoder operate over the same subword units.

3.4.2 The Core Pretraining Objective

BART is trained as a denoising autoencoder. Formally, given a clean document $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ , the noising function $f_\text{noise}$ produces a corrupted version $\tilde{\mathbf{x}} = f_\text{noise}(\mathbf{x})$ . The model is trained to maximize the probability of reconstructing $\mathbf{x}$ given $\tilde{\mathbf{x}}$ , factorized autoregressively:

$P(\mathbf{x} \mid \tilde{\mathbf{x}}) = \prod_{t=1}^{n} P(x_t \mid x_{<t}, \tilde{\mathbf{x}})$

where $x_t$ is the $t$ -th token of the original document, $x_{<t}$ represents all preceding tokens in the original document (which the decoder has generated so far), and $\tilde{\mathbf{x}}$ is the corrupted document (encoded bidirectionally by the encoder).

What this equation states: The probability of reconstructing the entire original document is the product of the probabilities of each token given all previous correctly-generated tokens and the full corrupted input. This is exactly the standard autoregressive factorization of sequence-to-sequence models.

The training loss is the negative log-likelihood of the original document:

$\mathcal{L} = -\sum_{t=1}^{n} \log P(x_t \mid x_{<t}, \tilde{\mathbf{x}})$

where the sum is over all token positions in the original document.

What it computes: For each position $t$ in the original document, the model predicts a probability distribution over the vocabulary given the previous ground-truth tokens and the corrupted input. The loss is the negative log probability assigned to the correct token, summed over all positions. This is standard teacher-forced cross-entropy training.

Why this form: The autoregressive factorization is essential for two reasons. First, it exactly matches how the model will be used during generation tasks at fine-tuning time (token-by-token left-to-right generation), so there is no pretraining-finetuning mismatch for generative tasks. Second, it allows the model to learn the dependencies between consecutive tokens in natural language—something that conditionally independent prediction objectives (like BERT's masked language modeling) cannot do. The cross-entropy objective is the maximum likelihood estimator for categorical distributions, which is the standard choice for language modeling and sequence generation.

Teacher forcing. During training, the decoder receives the ground-truth previous tokens as input at each step (not the model's own predictions). This is standard teacher forcing: the model always conditions on correct history, which stabilizes training, but creates a train-test mismatch that is mitigated by the scale of pretraining data.

A crucial property: The corrupted input $\tilde{\mathbf{x}}$ does not need to have the same length as the original $\mathbf{x}$ . The noising function can delete tokens (making $\tilde{\mathbf{x}}$ shorter), insert mask tokens (making it longer, if 0-length spans are infilled with MASK tokens), or rearrange the order. The decoder always generates the original sequence in the correct order with the correct length. This length flexibility is a key advantage that the paper emphasizes: it forces the model to learn not just which tokens to generate, but how many tokens to generate at each position, which is critical for tasks like summarization where the output length differs from the input length.

3.4.3 Noising Transformations

The paper treats the noising function as a design parameter and experiments with five transformations (illustrated in Figure 2), which can be composed arbitrarily. Each transformation corrupts the document in a different way, teaching the model different aspects of language structure.

Token Masking

Random tokens are sampled and replaced with a special [MASK] token. This is exactly the corruption used in BERT (Devlin et al., 2019). The model must predict the original identity of each masked token based on the surrounding context.

What the model learns: Local lexical semantics and the ability to use bidirectional context to infer missing words. Since masked tokens are replaced with a distinct symbol, the model knows where information is missing and must fill it in.

Why this is included as a baseline: Token masking is the most studied noising scheme, and including it allows direct comparison with BERT-style pretraining within the BART framework. It serves as a controlled baseline for the ablation study.

Token Deletion

Random tokens are deleted from the input entirely. Unlike token masking, the model receives no explicit indication of where tokens were deleted—it must infer which positions are missing inputs and what those missing tokens should be. For example, if the original sentence is "A B C . D E .", token deletion might produce "A . C . D E", where the reader (and the model) must figure out that tokens were removed between "A" and "." and between "C" and "D".

What the model learns: This is a strictly harder task than token masking because the model must simultaneously (a) determine which positions have missing tokens, and (b) predict what those tokens were. The paper notes (Section 2.2): "In contrast to token masking, the model must decide which positions are missing inputs." This teaches the model to reason about sentence structure and typical token sequences without explicit position markers for deletions.

Why this helps generation: Generation tasks (especially abstractive summarization) often require the model to produce output that is not a simple copy of the input—it must delete irrelevant information. Token deletion pretraining teaches the model that it is acceptable and necessary to omit information during reconstruction, which is a better inductive bias for generative compression tasks.

Text Infilling

This is one of BART's novel contributions and a key component of the final model. The procedure works as follows:

A number of text spans are sampled from the document.
The length of each span is drawn from a Poisson distribution with $\lambda = 3$ . This means most spans are 0–3 tokens long, with an average of 3, and longer spans become increasingly rare but are possible.
Each sampled span—regardless of its length—is replaced with a single [MASK] token.
Spans of length 0 correspond to the insertion of a [MASK] token at a position where no text was removed—the model must learn to predict that no token should be generated there.

The critical design choice is that a span of any length is replaced by exactly one [MASK] token. This means the model sees one mask token but must generate a variable number of tokens (potentially zero) at that position. For example, if the original text is "The quick brown fox jumps over the lazy dog" and the span "quick brown fox" is infilled, the input becomes "The [MASK] jumps over the lazy dog". The model must learn that this single mask token corresponds to three words in the original.

What the model learns: Text infilling teaches three skills simultaneously:

Span length prediction: The model must infer from the bidirectional context how many tokens are missing. A short gap might need one word; a long gap might need a phrase.
Content prediction: The model must predict the actual tokens that fill the gap.
Zero-length span handling: When a 0-length span is infilled, a mask token is inserted but the model must learn to generate nothing, effectively teaching it to ignore certain mask insertions.

Distinction from SpanBERT: The paper explicitly contrasts text infilling with SpanBERT (Joshi et al., 2019). SpanBERT also masks spans rather than individual tokens, but it (a) samples span lengths from a clamped geometric distribution rather than a Poisson distribution, and (b) replaces a span of length $k$ with exactly $k$ [MASK] tokens—one per masked token. This means SpanBERT preserves length information: the model sees $k$ masks and knows it must predict $k$ tokens. BART's text infilling deliberately destroys length information by replacing any span with exactly one mask token, regardless of span length. The authors argue this "teaches the model to predict how many tokens are missing from a span" (Section 2.2), which is a harder and more useful skill for generation.

Why Poisson distribution: The Poisson with $\lambda = 3$ produces mostly short spans (0–5 tokens) with a long tail of occasionally longer spans. This distribution matches the observation that most missing information in language involves short phrases or single words, while occasionally longer gaps require more substantial reconstruction. The Poisson was chosen (rather than, say, a uniform distribution) because it naturally models counts of rare events and produces a distribution that is bounded below by zero (span lengths can't be negative) and has a natural scale parameter ( $\lambda$ ).

Sentence Permutation

A document is split into sentences based on full stops (period characters), and these sentences are randomly shuffled. For example, "A B C. D E. F G." might become "F G. A B C. D E." The model must reconstruct the original order.

What the model learns: Discourse-level coherence and the ability to reason about which sentences precede or follow each other based on their content. This requires understanding narrative structure, causal relationships, temporal ordering, and topic flow—all skills that are important for tasks like summarization and dialogue where the model must organize information coherently.

Why this works: Sentence permutation forces the model to consider long-range dependencies that span entire sentences. In the reconstruction, the decoder must generate sentences in the correct order, which requires the encoder to capture enough information about each sentence's role in the discourse to guide the decoder's ordering decisions. This is a much longer-range dependency than token-level masking.

Limitation in isolation: The paper finds (Table 1) that sentence permutation alone performs poorly—85.4 SQuAD F1 and 81.5 MNLI accuracy, substantially below token-level noising approaches. This is because the model receives no information about why the sentences are out of order; it only knows that the original order was correct. The transformation is beneficial as a supplement to token-level noising but insufficient as the sole pretraining task.

Document Rotation

A token is chosen uniformly at random, and the document is rotated so that it begins with that token. Everything before the chosen token is moved to the end. For example, "A B C. D E." rotated to start at "C" becomes "C. D E. A B".

What the model learns: This task trains the model to identify the start of a document. Because the model must reconstruct the original document starting from the true beginning, it must learn to recognize discourse-level cues that indicate document openings (introductions, topic setting, etc.) versus mid-document content.

Why it performs poorly: The paper finds (Table 1) that document rotation performs extremely poorly across all tasks—it is the worst noising scheme by a large margin (77.2 SQuAD F1, 75.3 MNLI accuracy, dramatically worse perplexities on all generation tasks). The authors do not elaborate on why, but the likely reason is that the task is simultaneously too easy (there are clear linguistic signals of document beginnings, like capitalization, that the model can exploit without deep understanding) and too destructive (the rotation corrupts all local coherence, making token-level prediction extremely difficult). The model essentially learns to detect beginnings rather than to understand language.

Practical role: Document rotation is included in the ablation to demonstrate the range of possible noising functions and to show that not all corruptions are equally useful. It serves as a negative result that highlights the importance of choosing the right noising scheme.

Composition of Transformations

The final BART model uses a composition of two transformations applied to each document during pretraining (Section 5.1):

Text infilling: 30% of tokens in each document are masked using the text infilling procedure (Poisson-distributed spans, each replaced with a single [MASK] token).
Sentence permutation: All sentences in the document are randomly permuted.

The composition means that the model receives a document where (a) the sentence order is randomly shuffled, and (b) within each sentence, some spans are replaced with single mask tokens. The model must simultaneously recover the correct sentence order and the missing spans.

Why 30% masking: The 30% figure is higher than BERT's 15% masking rate. The authors do not provide a specific justification for this number, but it is likely motivated by the observation that text infilling with a single mask token per span already destroys some information about token count, so a higher masking rate is needed to create enough reconstruction difficulty. Additionally, text infilling is inherently less destructive per-mask-token than token masking (because one mask can hide multiple tokens), so a higher rate is needed to achieve comparable difficulty.

Why sentence permutation at scale: In the base-model ablation (Section 4, Table 1), text infilling alone achieves 90.8 SQuAD F1 and 84.0 MNLI accuracy, while text infilling + sentence permutation achieves 90.8 SQuAD F1 and 83.8 MNLI accuracy—essentially identical performance on discriminative tasks. However, on generation tasks, the combination improves CNN/DM perplexity from 5.83 to 5.41 (a meaningful reduction). The authors hypothesized (Section 5.1) that "larger pre-trained models may be better able to learn from this task," which is why sentence permutation is included in the large-scale model despite showing marginal benefits in the base-size ablation. The intuition is that discourse-level reasoning requires more model capacity to learn effectively, and the large model (12 layers, 1024 hidden size) has sufficient capacity to benefit.

3.4.4 Fine-Tuning Procedures

One of BART's key advantages is that the same architecture can be fine-tuned for different tasks by changing how information flows through the encoder and decoder. The paper describes four fine-tuning strategies, each matched to a different task category.

Sequence Classification Tasks (Section 3.1)

For tasks like MNLI (sentence pair classification) and sentiment analysis, BART is used as follows:

The same input sequence is fed to both the encoder and the decoder.
Unlike BERT, which uses a special [CLS] token at the beginning of the input, BART adds an end-of-sequence (EOS) token at the end of the decoder input.
The decoder processes the input autoregressively (left to right), and the final hidden state corresponding to the EOS token is used as the sequence representation.
This representation is fed into a new multi-class linear classifier (a single linear layer + softmax) that is randomly initialized and trained during fine-tuning.

Why EOS at the end rather than CLS at the beginning: The authors explain (Section 3.1) that this allows the decoder's representation of the EOS token to "attend to decoder states from the complete input." Because the decoder is autoregressive, the EOS token (being the last token) can attend to all preceding tokens in the decoder's self-attention. If a CLS token were at the beginning, it could only attend to itself (since causal masking prevents attending forward). By placing the classification token at the end, BART ensures it has seen the entire sequence before making the classification decision. This is architecturally equivalent to how encoder-only models use a CLS token (since the encoder is bidirectional, the CLS token can attend to everything regardless of position), but adapted for the autoregressive decoder.

Why feed the same input to both encoder and decoder: This leverages the pretrained architecture without modification. The encoder processes the input bidirectionally, capturing rich contextual representations. The decoder processes the same input autoregressively, building representations that capture left-to-right dependencies. The final EOS representation integrates information from both paths via cross-attention, giving the classifier access to both bidirectional and autoregressive features.

Token Classification Tasks (Section 3.2)

For tasks like SQuAD (extractive question answering, where the model must predict the start and end positions of an answer span), BART is used as follows:

The complete document (concatenated question and context) is fed to both the encoder and decoder, same as for sequence classification.
The top hidden state of the decoder at each position is used as the representation for the corresponding input token.
Separate start and end classifiers (linear layers) take each token's representation and predict a start score and an end score.
During inference, the answer span is the contiguous region with the highest start and end scores (with the constraint that the end position follows the start position).

Why decoder states rather than encoder states: The paper does not explicitly justify this choice, but it is consistent with BART's design philosophy of using the decoder for all predictions. Using decoder states means the model leverages both the bidirectional encoder context (via cross-attention) and the left-to-right decoder processing. Additionally, this unifies the fine-tuning approach: classification, token labeling, and generation all use the decoder as the prediction module.

Comparison with BERT's SQuAD approach: BERT uses its final encoder hidden states for token classification, since it has no decoder. BART's approach is architecturally different (the predictions come from the decoder, which processes the input autoregressively), but the paper shows it achieves comparable performance (88.8 EM / 94.6 F1 for BART vs. 88.9 EM / 94.6 F1 for RoBERTa on SQuAD 1.1, per Table 2). This suggests that autoregressive processing in the decoder does not hurt span prediction, likely because the cross-attention mechanism gives each decoder position access to the full bidirectional context from the encoder, even though the decoder's self-attention is causal.

Sequence Generation Tasks (Section 3.3)

For tasks like summarization (XSum, CNN/DailyMail), dialogue response generation (ConvAI2), and abstractive question answering (ELI5), BART is fine-tuned as a standard sequence-to-sequence model:

The encoder receives the input sequence (the article to summarize, the dialogue context, or the question + supporting documents).
The decoder generates the output sequence (the summary, the dialogue response, or the answer) autoregressively, exactly matching the pretraining setup where the decoder generated the original document from corrupted input.
The model is trained with teacher forcing: at each training step, the decoder receives the ground-truth output tokens as input, and the loss is the cross-entropy between the predicted distribution and the ground-truth next token.
During inference, the decoder generates tokens one at a time, each conditioned on the encoder output and all previously generated tokens.

Why this setup matches pretraining: This is BART's key advantage for generation. During pretraining, the encoder sees corrupted text and the decoder generates clean text. During fine-tuning, the encoder sees the input and the decoder generates the output. The input-output relationship in fine-tuning is analogous to the corrupted-clean relationship in pretraining: the model learns to take one form of text and transform it into another. This is particularly natural for abstractive summarization, where the summary is a compressed, reworded version of the article—exactly the kind of transformation the model learned during denoising pretraining.

Fine-tuning hyperparameters: For generation tasks, the paper uses:

Label smoothed cross-entropy loss (Pereyra et al., 2017) with a smoothing parameter of 0.1. Label smoothing replaces the hard one-hot target distribution with a smoothed distribution that assigns probability $(1 - \epsilon)$ to the correct token and $\epsilon / (V - 1)$ to all other tokens (where $V$ is the vocabulary size and $\epsilon = 0.1$ ). This regularizes the model by preventing it from becoming overconfident.
During inference (decoding): beam search with beam size 5, duplicated trigram removal (if a trigram appears twice in the beam, subsequent tokens are blocked to prevent repetition), and tuned minimum length, maximum length, and length penalty hyperparameters on the validation set. The length penalty (Fan et al., 2017) biases the beam search toward longer or shorter outputs by dividing the log-probability by a length-dependent term.

Machine Translation (Section 3.4)

The machine translation fine-tuning approach is novel and architecturally creative. BART is used as a pretrained target-side language model for translating into English. The procedure works as follows:

Replace the encoder embedding layer: BART's encoder word embedding layer is replaced with a new, randomly initialized encoder. This new encoder has its own vocabulary (potentially different from BART's English vocabulary) for the source language (Romanian in the paper's experiments).
The new encoder maps source language words into representations that BART can process: The new encoder transforms the source language tokens into hidden states. These hidden states feed into BART's pretrained encoder layers, which process them through the bidirectional self-attention stack. The decoder then generates the English translation autoregressively.
The model is trained end-to-end: The cross-entropy loss from the decoder's output (the English translation) is backpropagated through the entire model, including the new source encoder and all of BART's pretrained parameters.

Two-step training procedure: The paper uses a two-stage training process to avoid catastrophic forgetting of BART's pretrained knowledge:

Step 1 (Fixed BART): Most of BART's parameters are frozen. Only the randomly initialized source encoder, the BART positional embeddings, and the self-attention input projection matrix of BART's encoder first layer are updated. This allows the new encoder to learn to map source language representations into the space that BART's frozen encoder expects, without disturbing BART's pretrained weights.
Step 2 (Tuned BART): All model parameters are unfrozen and trained for a small number of iterations. This fine-tunes BART's English language understanding to the specific task of translating from Romanian, adapting the pretrained representations to the translation domain.

Why this works: The approach can be understood as treating BART as a pretrained "denoiser" for English. The new encoder learns to produce representations that, when processed by BART's encoder and decoder, result in fluent English output. Crucially, BART's decoder is pretrained to generate coherent English text given corrupted English input—so the new encoder's job is to map Romanian text into a representation that looks, to BART, like corrupted English. The decoder then "denoises" this into fluent English translation. This is conceptually elegant: translation is framed as a denoising problem where the "noise" is the foreign language.

The back-translation connection: The paper notes (Section 5.4) that "preliminary results suggested that our approach was less effective without back-translation data, and prone to overfitting." Back-translation (Sennrich et al., 2016) augments the training data by translating target-language monolingual data into source language using a reverse translation model, then training on these synthetic source-target pairs. The BART-based approach benefits from back-translation because it increases the amount of parallel data, reducing overfitting on the limited parallel corpus (WMT16 Romanian-English is relatively small).

Architectural innovation: The paper frames this as addressing a limitation of prior work (Section 3.4): "Previous work Edunov et al. (2019) has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in decoders have been limited." By using the entire BART model (encoder + decoder) as a pretrained decoder, this approach achieves gains where prior methods could not. The key insight is that BART's encoder, when kept in the pipeline, helps the decoder by providing bidirectional context over the "denoised" intermediate representation, even though the ultimate source is a different language.

3.4.5 Controlled Ablation Methodology (Section 4)

A significant technical contribution is the methodology used to compare pretraining objectives. The paper reimplements several previously proposed pretraining approaches within the same codebase, with the same data, same training budget (1M steps), and comparable model sizes. This controls for implementation details that had confounded prior comparisons.

The reimplemented objectives are:

Language Model (GPT-style): A left-to-right Transformer language model, equivalent to BART's decoder without cross-attention. The model predicts each token given all preceding tokens. For fine-tuning on discriminative tasks, the input is fed as a prefix to the decoder, and the loss is computed only on the target portion of the sequence. This approach is implemented using a diagonal self-attention mask (causal masking) in the decoder.

Permuted Language Model (XLNet-style): Based on XLNet (Yang et al., 2019), the model samples 1/6 of the tokens and generates them autoregressively in a random order. For consistency with other models in the comparison, the paper does not implement XLNet's relative positional embeddings or segment-level recurrence. This simplification is noted as a likely reason why the reimplementation underperforms published XLNet results, but it ensures architectural comparability.

Masked Language Model (BERT-style): The standard BERT objective: 15% of tokens are replaced with [MASK] symbols, and the model is trained to independently predict the original tokens at those positions. The predictions are conditionally independent (not autoregressive), matching BERT's approach.

Multitask Masked Language Model (UniLM-style): As in UniLM (Dong et al., 2019), the model is trained with different self-attention masks applied to different portions of the data. The self-attention masks are chosen randomly with the following proportions:

1/6 left-to-right (causal mask, like GPT)
1/6 right-to-left (reverse causal mask)
1/3 unmasked (bidirectional, like BERT)
1/3 with the first 50% of tokens unmasked and a left-to-right mask for the remaining 50% (a hybrid prefix mask)

This trains the model to handle multiple attention patterns with shared parameters.

Masked Seq-to-Seq (MASS-style): A span containing 50% of tokens is masked, and a sequence-to-sequence model is trained to predict only the masked tokens. The encoder sees the unmasked tokens, and the decoder generates the masked ones. This is the most architecturally similar baseline to BART, since it uses an encoder-decoder architecture, but with a restricted noising scheme (one contiguous span).

Technical adaptation for fair comparison: For the Permuted LM, Masked LM, and Multitask Masked LM, the paper uses two-stream attention (Yang et al., 2019) to efficiently compute likelihoods. Two-stream attention separates the content representation (what the model knows about the input) from the query representation (what position the model is predicting), which allows autoregressive prediction with bidirectional context. The paper also experiments with two ways of structuring the sequence-to-sequence problem:

Encoder-decoder format: The corrupted input goes to the encoder, and the target goes to the decoder.
Decoder-only format: The source is prefixed to the target in the decoder, with the loss computed only on the target portion.

The authors find that "the former works better for BART models, and the latter for other models" (Section 4.1). This is because BART's architecture naturally supports encoder-decoder processing, while the other models are designed as decoder-only architectures and perform better when used that way. This finding underscores the importance of matching the architectural assumptions when comparing pretraining objectives.

Perplexity as a unified metric. For generation tasks in the ablation, the paper reports perplexity (Table 1), which measures how well the model predicts the human-written reference text. Lower perplexity indicates better language modeling quality. Perplexity is defined as the exponentiated average negative log-likelihood per token: $\text{PPL} = \exp(-\frac{1}{N}\sum_{t=1}^N \log P(x_t | x_{<t}, \text{context}))$ . This metric allows direct comparison of generative quality across different pretraining objectives, since all models are fine-tuned as generative models on the same tasks.

Design Choices and Their Justifications

Choice 1: Standard Transformer architecture over novel architecture. BART uses the unmodified Transformer from Vaswani et al. (2017) with only activation function and initialization changes adopted from GPT. The justification is that architectural novelty is not the goal—the goal is to demonstrate that the right pretraining objective can make a standard architecture simultaneously strong on understanding and generation. Using a standard architecture also makes the comparison with BERT, GPT, and other models cleaner, since architectural differences are minimized.

Choice 2: Autoregressive decoder for generation despite the efficiency cost. Autoregressive generation is slower than non-autoregressive alternatives (like BERT's independent predictions) because tokens must be generated sequentially. The paper accepts this cost because autoregressive pretraining exactly matches the generation setting during fine-tuning, eliminating pretraining-finetuning mismatch. The ablation results (Table 1) empirically validate this: models without left-to-right autoregressive pretraining (Masked LM, Permuted LM) consistently underperform on generation tasks.

Choice 3: Text infilling with single MASK token per span. The choice to replace any-length span with exactly one [MASK] token is deliberate: it forces the model to learn span length prediction, which is critical for abstractive generation where output length differs from input length. The alternative (replacing a span of length $k$ with $k$ masks, as in SpanBERT) would be easier but would not teach the model to predict how much text to generate at each position.

Choice 4: Sentence permutation as a supplement, not a standalone task. The paper keeps token-level noising (text infilling) as the primary task and adds sentence permutation on top, rather than using sentence permutation alone. The ablation justifies this: sentence permutation in isolation performs poorly (Table 1), but it adds value when combined with text infilling, particularly on summarization tasks that require discourse-level organization.

Choice 5: Decoder hidden states for all fine-tuning predictions. Whether for classification (using the EOS token), token labeling (using per-token decoder states), or generation (autoregressive decoding), all predictions come from the decoder. This unified approach means the decoder is always the prediction module, which reduces the architectural complexity of fine-tuning and ensures the pretrained decoder receives gradient signal for all task types.

Choice 6: Two-step training for machine translation. Freezing most of BART during the first training step prevents catastrophic forgetting of the pretrained English language model while the new encoder adapts. The second step (full fine-tuning) then adapts BART's internal representations to the translation task. This two-step approach is a form of gradual unfreezing (Howard and Ruder, 2018) adapted to the specific challenge of grafting a new encoder onto a pretrained sequence-to-sequence model.

4. Key Insights and Innovations

Innovation 1: Treating Noising Flexibility as the Primary Design Axis Rather Than Architecture

The dominant paradigm in pretraining prior to BART was to innovate on architecture or training objectives in tandem—BERT introduced masked language modeling with a bidirectional encoder, GPT used autoregressive language modeling with a left-to-right decoder, XLNet invented permuted language modeling with two-stream attention, and UniLM designed an ensemble of attention masks. In every case, the noising scheme was tightly coupled to the architecture: BERT's masking was designed for its bidirectional encoder, GPT's language modeling matched its causal decoder, and XLNet's permutation required its specialized attention mechanism.

BART makes a subtle but fundamental conceptual move: decouple the noising function from the architecture entirely. By using a standard sequence-to-sequence Transformer—an architecture that had been available since Vaswani et al. (2017)—BART treats the choice of what to corrupt and how as the independent variable and demonstrates that architectural generality plus noising flexibility yields a model that subsumes prior approaches without needing architectural novelty. This is a reframing of the pretraining problem: rather than asking "what architecture enables a particular pretraining objective?", BART asks "what noising scheme, when applied to a general encoder-decoder architecture, teaches the most useful representations?"

The evidence for this as a genuine conceptual shift comes from the ablation study in Table 1. When implemented in the same architecture with the same data and training budget, the previously proposed objectives (Masked LM, Permuted LM, Language Model, Multitask Masked LM, Masked Seq-to-Seq) show dramatically task-dependent performance—some excel at SQuAD but fail at generation, others do the reverse. BART's text infilling, by contrast, achieves the most consistently strong performance across all tasks. This demonstrates that the noising scheme, not the architecture, is the primary driver of cross-task generalization. The architecture provides the capacity for both bidirectional understanding and autoregressive generation; the noising scheme determines whether that capacity is fully utilized.

The practical consequence is that future work on pretraining can focus on designing better noising functions (the paper explicitly notes "there is a significant potential for development of other new alternatives" in Section 2.2) without needing to redesign the architecture. This represents a maturing of the field: architecture design and corruption design become separate research axes, enabling more rapid iteration on each.

Innovation 2: Demonstrating That a Single Pretrained Model Can Be SOTA-Competitive on Both Understanding and Generation

This sounds like a benchmark claim, but it represents a deeper intellectual contribution: it disproves the implicit assumption that understanding and generation require fundamentally different pretraining paradigms. The field had largely accepted a tradeoff—bidirectional context improves understanding but breaks autoregressive generation; autoregressive pretraining enables generation but sacrifices deep bidirectional reasoning. Different papers optimized different sides of this tradeoff, and the result was a landscape where no single model appeared in the top ranks of both GLUE/SQuAD leaderboards and summarization/dialogue benchmarks.

BART's results in Tables 2–5 directly challenge this assumed tradeoff. On discriminative tasks (Table 2), BART achieves 88.8 EM / 94.6 F1 on SQuAD 1.1 and 89.9/90.1 on MNLI, matching RoBERTa within fractions of a point despite having a unidirectional decoder. On generation tasks, it achieves unprecedented performance: +6 ROUGE on XSum over the prior state-of-the-art (Table 3), best results on CNN/DailyMail summarization, ConvAI2 dialogue (Table 4), and ELI5 abstractive QA (Table 5). No prior model appeared simultaneously competitive across this range.

What makes this intellectually significant rather than merely a leaderboard sweep is that it provides existence proof for a unified pretraining paradigm. Before BART, one could reasonably argue that the understanding-generation gap was fundamental—that a model architecture optimized for one would inevitably sacrifice the other. BART demonstrates that this is false: the right pretraining setup can produce representations that serve both purposes. The key is the encoder-decoder architecture where the encoder handles bidirectional context (for understanding) and the decoder handles autoregressive generation (for production), but both are trained jointly with the same objective.

A subtle point that strengthens this argument: BART's decoder is autoregressive and unidirectional during pretraining (it generates the original document left-to-right), yet the model still achieves RoBERTa-level performance on SQuAD. This means the decoder learned to leverage the encoder's bidirectional representations via cross-attention even though its own self-attention is causal. The practical implication is that bidirectional context doesn't need to be computed in the same module that makes predictions—it can be computed in an encoder and accessed on-demand via cross-attention, preserving autoregressive generation capability without sacrificing understanding quality.

Innovation 3: Identifying Text Infilling with Length Ambiguity as the Critical Generative Pretraining Mechanism

The ablation study in Table 1 produces a striking pattern: BART with text infilling (replacing arbitrary-length spans with a single [MASK] token) achieves the most consistently strong performance across all task categories, outperforming both token masking (which preserves position information) and token deletion (which removes tokens without a marker). This is not merely a hyperparameter finding—it identifies a specific mechanistic principle that explains why certain pretraining objectives transfer better to generation tasks.

The principle is length ambiguity: forcing the model to predict not just which tokens to generate but how many tokens to generate at each position. Standard token masking (BERT-style) tells the model exactly where information is missing—each [MASK] token corresponds to exactly one word to predict. Text deletion removes tokens without markers, requiring the model to infer missing positions, but still one deleted token maps to one generated token. Text infilling with a single mask token per span creates genuine length ambiguity: one input token might require zero, one, five, or more output tokens.

This matters for generation tasks because they inherently involve length mismatch between input and output. Summarization compresses; dialogue expansion elaborates; question answering produces answers of variable length. A model pretrained with length-ambiguous reconstruction has already learned to produce variable-length output from fixed-length input markers, which directly transfers to these tasks. The models without this property (Masked LM, Permuted LM) struggle on generation because they learned in a setting where input and output positions have a fixed correspondence.

The comparison with SpanBERT is instructive here. SpanBERT masks spans but preserves length information (a span of length k is replaced with k masks). BART's text infilling deliberately destroys this information. The fact that BART outperforms the SpanBERT-inspired approach (Masked Seq-to-Seq in Table 1, which also uses contiguous span masking but with length preservation) suggests that length ambiguity specifically, not span-level reasoning in general, is the active ingredient. This is a diagnostic finding: it tells us which aspect of span-based pretraining matters most.

Innovation 4: Reframing Machine Translation as a Denoising Problem from a Target-Language Perspective

The machine translation approach in Section 3.4—using the entire pretrained BART as a target-side decoder by grafting on a new source encoder—represents a conceptual shift in how to leverage pretrained models for translation. Prior work (Edunov et al., 2019; Lample and Conneau, 2019) had shown that pretrained encoders could improve translation quality, but gains from pretrained decoders had been limited. The standard framing was: pretrain a language model or encoder on monolingual data, then use it as initialization for parts of a translation model.

BART inverts this perspective. Instead of using pretrained components to initialize a translation model, the approach treats the pretrained BART as a complete English denoising engine and adds a minimal new component (the source encoder) whose job is to learn to produce representations that BART can denoise into English. Translation is reframed not as mapping between languages but as: source language → corrupted English → clean English. The new encoder learns to produce "corrupted English" representations that BART's pretrained denoising can recover.

This is a fundamentally different way of thinking about transfer learning for MT. Rather than pretraining on the source language, target language, or both, and then fine-tuning the entire model as a translator, BART's approach pretrains only on the target language (English) and learns to treat the source language as a form of noise that can be removed. The conceptual elegance is that it aligns perfectly with BART's pretraining: the model was trained to denoise corrupted English; now the source language is simply a new, systematic form of corruption.

The empirical result—a 1.1 BLEU improvement over a strong back-translation baseline on WMT16 Romanian-English (Table 6)—is modest compared to the generation task gains, but the conceptual contribution is more significant than the metric. It opens up a new way of using pretrained sequence-to-sequence models: as universal target-language decoders that can be paired with lightweight source-language encoders for any translation direction. This is particularly valuable for low-resource translation directions where pretraining on the source language is impractical due to data limitations, but target-language (often English) pretraining data is abundant. The paper also acknowledges limitations (the approach is "less effective without back-translation data, and prone to overfitting"), making it a proof of concept rather than a fully solved problem, but the conceptual reframing is what constitutes the innovation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a broad suite: (1) SQuAD 1.1 and 2.0 (Rajpurkar et al., 2016) for extractive question answering—Wikipedia paragraphs with crowd-sourced questions, using the standard train/dev splits; (2) the GLUE benchmark (Wang et al., 2018), a collection of nine natural language understanding tasks including MNLI, SST-2, QQP, QNLI, STS-B, RTE, MRPC, and CoLA, each with standard train/dev splits; (3) CNN/DailyMail (Hermann et al., 2015) for extractive-oriented summarization; (4) XSum (Narayan et al., 2018) for highly abstractive summarization; (5) ConvAI2 (Dinan et al., 2019) for dialogue response generation conditioned on persona and context; (6) ELI5 (Fan et al., 2019) for long-form abstractive question answering; (7) WMT16 Romanian-English (Sennrich et al., 2016) for machine translation, augmented with back-translation data. The ablations in Section 4 use a representative subset (SQuAD, MNLI, ELI5, XSum, ConvAI2, CNN/DM), while large-scale experiments in Section 5 cover all tasks.
Base model(s). Two configurations of the BART architecture are used: a base model with 6 encoder and 6 decoder layers (hidden size 768, 16 attention heads) for the ablation study in Section 4, and a large model with 12 encoder and 12 decoder layers (hidden size 1024, 16 attention heads) for the large-scale experiments in Section 5. Both configurations are built on the standard Transformer sequence-to-sequence architecture (Vaswani et al., 2017) with GeLU activations and $\mathcal{N}(0, 0.02)$ initialization. The base model is chosen to enable efficient comparison of multiple pretraining objectives at controlled scale; the large model is chosen to match the resource scale of RoBERTa (Liu et al., 2019) for fair comparison. All ablation baselines (Language Model, Permuted Language Model, Masked Language Model, Multitask Masked Language Model, Masked Seq-to-Seq) are implemented at the base model scale within the same codebase and trained on identical data.
Metrics. Task-specific metrics are used throughout. For SQuAD 1.1: Exact Match (EM) and token-level F1 score. For SQuAD 2.0: EM and F1 (the dataset includes unanswerable questions). For GLUE: accuracy for MNLI, SST-2, QQP, QNLI, RTE, and MRPC; Pearson correlation for STS-B; Matthews correlation for CoLA. For summarization (CNN/DailyMail, XSum): ROUGE-1, ROUGE-2, and ROUGE-L F1 scores. For ConvAI2 dialogue: validation F1 and validation perplexity (renormalized based on the official ConvAI2 tokenizer). For ELI5: ROUGE-1, ROUGE-2, and ROUGE-L. For machine translation (WMT16 RO-EN): BLEU score. In the ablation study (Table 1), perplexity (PPL) is reported as a unified generative quality metric, defined as the exponentiated average negative log-likelihood per token of the human reference text under the fine-tuned model.
Baselines. The paper compares against a wide range of prior systems. For discriminative tasks (Table 2): BERT (Devlin et al., 2019), UniLM (Dong et al., 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019)—RoBERTa being the most directly comparable since it uses the same pretraining data and scale as BART's large model. For summarization (Table 3): Lead-3 (taking the first three sentences of the article), Pointer-Generator networks with and without coverage (PTGEN / PTGEN+COV; See et al., 2017), UniLM, and BERTSUMABS / BERTSUMEXTABS (Liu and Lapata, 2019). For ConvAI2 (Table 4): Seq2Seq with attention and the best system from the ConvAI2 competition. For ELI5 (Table 5): Best extractive model, Language Model, Seq2Seq, and Seq2Seq Multitask—all from Fan et al. (2019). For machine translation (Table 6): a Transformer-large baseline (Vaswani et al., 2017) trained on the same WMT16 RO-EN data augmented with back-translation.
Generation budget / compute accounting. The ablation study in Section 4 controls for training budget by training all models (base size) for exactly 1 million steps on the same combination of books and Wikipedia data, using identical optimization infrastructure. This is a critical control—differences in performance cannot be attributed to differences in training duration or data volume. The large-scale experiments in Section 5 use a batch size of 8000 and train for 500,000 steps, matching RoBERTa's training setup. For fine-tuning, generation tasks use beam search with beam size 5, duplicated trigram removal, and tuned minimum/maximum length and length penalty on the validation set. The machine translation approach uses a beam width of 5 and length penalty $\alpha = 1$ . In the two-step MT training, the "fixed BART" step trains only the new source encoder and a few BART projection matrices, while the "tuned BART" step trains all parameters "for a small number of iterations" (exact count not specified).
Cross-validation / statistical protocol. The paper does not report formal cross-validation or statistical significance testing. For the large-scale experiments, hyperparameters (learning rate, dropout, etc.) are presumably tuned on validation sets, but no explicit cross-validation protocol is described. The ablation study in Section 4 notes that "minor changes to the learning rate and usage of layer normalisation" were tuned separately for each objective, but these are not further quantified. The absence of confidence intervals or significance tests is a methodological limitation.

Main Quantitative Results

Controlled Ablation of Pretraining Objectives (Section 4, Table 1)

The ablation study at base model scale compares seven BART noising variants and five reimplemented prior objectives, all trained on identical data for 1M steps. The headline finding is that BART with text infilling achieves the most consistently strong performance across all tasks, with text infilling + sentence permutation offering the best generation results.

On SQuAD 1.1 (F1): BART with text infilling achieves 90.8, matching BART with token masking (90.4) and token deletion (90.4), while substantially outperforming Language Model (76.7), Masked Seq-to-Seq (87.0), Permuted Language Model (89.1), and Multitask Masked LM (89.2). The published BERT Base achieves 88.5 F1, meaning all BART token-level noising variants surpass BERT on SQuAD despite having a unidirectional decoder.

On MNLI (accuracy): BART with token masking and token deletion both achieve 84.1, slightly above BERT Base (84.3, though this difference is within typical variance). Text infilling achieves 84.0. Language Model achieves the worst at 80.1, confirming that bidirectional context is crucial for natural language inference.

On generation tasks (measured by perplexity; lower is better), the pattern shifts dramatically:

XSum: Text infilling achieves 6.61 PPL, outperforming token masking (7.08), token deletion (6.90), Language Model (7.00), Permuted LM (7.69), Multitask Masked LM (7.50), and Masked Seq-to-Seq (6.80). Text infilling + sentence permutation further reduces this to 6.62.
ConvAI2: Text infilling achieves 11.05 PPL, the best among BART variants (token masking: 11.73, token deletion: 11.46). Language Model achieves 11.51 and Masked Seq-to-Seq achieves 11.43—both competitive but slightly worse. Text infilling + sentence permutation achieves 11.12.
CNN/DailyMail: Text infilling + sentence permutation achieves the best PPL at 5.41, surpassing text infilling alone (5.83), token deletion (5.87), token masking (6.10), and all non-BART objectives (Language Model: 6.56, Masked Seq-to-Seq: 6.19, Permuted LM: 6.96).
ELI5: This is the outlier. Language Model achieves the best perplexity at 21.40, followed by Masked Seq-to-Seq (23.40) and Multitask Masked LM (23.73). BART with text infilling achieves 24.26 and text infilling + sentence permutation achieves 24.17—both worse than several baselines. The authors note this explicitly: "The ELI5 dataset is an outlier... A pure language model performs best, suggesting that BART is less effective when the output is only loosely constrained by the input."

A critical pattern across all generation tasks: token deletion outperforms token masking (e.g., XSum: 6.90 vs. 7.08; CNN/DM: 5.87 vs. 6.10; ConvAI2: 11.46 vs. 11.73). The authors interpret this as evidence that deletion-based pretraining better matches generation tasks where information must be removed—"Deletion appears to outperform masking on generation tasks" (Section 4.3).

The document rotation and sentence permutation objectives perform catastrophically in isolation. Document rotation achieves 77.2 SQuAD F1 and 75.3 MNLI accuracy—far worse than any other method. Sentence permutation achieves 85.4 SQuAD F1 and 81.5 MNLI accuracy—better than rotation but still substantially below token-level noising. The authors conclude: "Pre-training objectives based on rotating documents or permuting sentences perform poorly in isolation. The successful methods either use token deletion or masking, or self-attention masks" (Section 4.3).

Large-Scale Discriminative Tasks (Section 5.2, Table 2)

BART (large, 12 encoder + 12 decoder layers, trained for 500K steps on 160GB of data) is compared against BERT, UniLM, XLNet, and RoBERTa on SQuAD and GLUE. The headline: BART matches RoBERTa within fractions of a point on nearly all discriminative tasks, demonstrating that the unidirectional decoder does not hurt understanding performance.

On SQuAD 1.1: BART achieves 88.8 EM / 94.6 F1, compared to RoBERTa's 88.9 EM / 94.6 F1—essentially tied. XLNet achieves 89.0 EM / 94.5 F1 (also tied within rounding). BERT achieves 84.1 EM / 90.9 F1, substantially lower.

On SQuAD 2.0: BART achieves 86.1 EM / 89.2 F1, slightly behind RoBERTa (86.5 EM / 89.4 F1) and XLNet (86.1 EM / 88.8 F1), but well ahead of BERT (79.0 EM / 81.8 F1) and UniLM (80.5 EM / 83.4 F1).

On GLUE tasks (reporting the metric appropriate to each task):

MNLI: BART achieves 89.9 matched / 90.1 mismatched accuracy, compared to RoBERTa's 90.2/90.2. XLNet achieves 89.8/- (mismatched not reported in this comparison). BERT achieves 86.6/-.
SST-2: BART achieves 96.6, slightly ahead of RoBERTa (96.4), XLNet (95.6), and BERT (93.2).
QQP: BART achieves 92.5, ahead of RoBERTa (92.2), XLNet (91.8), and BERT (91.3).
QNLI: BART achieves 94.9, slightly ahead of RoBERTa (94.7), XLNet (93.9), and BERT (92.3).
STS-B: BART achieves 91.2, behind RoBERTa (92.4), XLNet (91.8), and BERT (90.0).
RTE: BART achieves 87.0, slightly ahead of RoBERTa (86.6), XLNet (83.8), and BERT (70.4).
MRPC: BART achieves 90.4, behind RoBERTa (90.9), slightly ahead of XLNet (89.2) and BERT (88.0).
CoLA: BART achieves 62.8 Matthews correlation, behind RoBERTa (68.0), XLNet (63.6), and UniLM (61.1), but slightly ahead of BERT (60.6).

The overall pattern is clear: BART and RoBERTa trade small advantages across tasks but are statistically indistinguishable in aggregate. The authors state this explicitly: "BART performs comparably to RoBERTa and XLNet, suggesting that BART's uni-directional decoder layers do not reduce performance on discriminative tasks" (Table 2 caption).

Large-Scale Generation Tasks (Section 5.3)

Summarization (Table 3). BART achieves new state-of-the-art results on both CNN/DailyMail and XSum, with particularly dramatic gains on the more abstractive dataset.

On CNN/DailyMail: BART achieves 44.16 ROUGE-1 / 21.28 ROUGE-2 / 40.90 ROUGE-L. This surpasses the previous best (UniLM: 43.33/20.21/40.51) by 0.83 ROUGE-1, 1.07 ROUGE-2, and 0.39 ROUGE-L. It also outperforms BERTSUMEXTABS (42.13/19.60/39.18), BERTSUMABS (41.72/19.39/38.76), and the Lead-3 baseline (40.42/17.62/36.67). The gains on CNN/DM are modest but consistent—unsurprising given that the dataset favors extractive approaches and the baseline performance is already high.

On XSum: BART achieves 45.14 ROUGE-1 / 22.27 ROUGE-2 / 37.25 ROUGE-L. This represents a dramatic improvement of approximately 6 ROUGE points across all metrics over the prior state-of-the-art (BERTSUMEXTABS: 38.81/16.50/31.27). The gap is even larger relative to Pointer-Generator networks (PTGEN: 29.70/9.21/23.24) and the Lead-3 extractive baseline (16.30/1.60/11.95). The authors highlight: "BART outperforms the best previous work, which leverages BERT, by roughly 6.0 points on all ROUGE metrics—representing a significant advance in performance on this problem" (Section 5.3).

The contrast between CNN/DM and XSum gains is instructive: CNN/DM summaries closely track source sentences (extractive models do well), while XSum requires abstraction and rewriting. BART's larger gains on XSum confirm that the denoising pretraining is particularly well-suited to abstractive generation where the output differs substantially from the input.

Dialogue (Table 4). On ConvAI2, BART achieves 20.72 validation F1 and 11.85 validation perplexity (renormalized). This outperforms the best system from the ConvAI2 competition (19.09 F1, 17.51 PPL) and a Seq2Seq + Attention baseline (16.02 F1, 35.07 PPL). The improvement in both F1 (generation quality) and perplexity (language modeling quality) indicates that BART produces more fluent and more appropriate dialogue responses.

Abstractive QA (Table 5). On ELI5, BART achieves 30.6 ROUGE-1 / 6.2 ROUGE-2 / 24.3 ROUGE-L. This improves over the previous best (Seq2Seq Multitask: 28.9/5.4/23.1) by 1.7 ROUGE-1, 0.8 ROUGE-2, and 1.2 ROUGE-L. The gains are real but modest compared to summarization. The authors note that "the dataset remains challenging, because answers are only weakly specified by the question" (Section 5.3). Interestingly, the Language Model baseline (27.8/4.7/23.1) is competitive with Seq2Seq models on this task, consistent with the ablation finding that ELI5 is an outlier where output is "only loosely constrained by the input."

Machine Translation (Section 5.4, Table 6)

On WMT16 Romanian-English augmented with back-translation data, the baseline Transformer-large achieves 36.80 BLEU. BART with fixed pretrained parameters (only training the new source encoder, positional embeddings, and first-layer self-attention projection) achieves 36.29 BLEU—slightly worse than the baseline. However, BART with full fine-tuning (tuned BART) achieves 37.96 BLEU, a 1.16 BLEU improvement over the baseline and a 1.67 BLEU improvement over the fixed BART variant.

The degradation in the fixed setting suggests that simply grafting a new encoder onto frozen BART is insufficient—the pretrained encoder expects representations in a specific distribution, and the new source encoder cannot fully adapt without also updating BART's internal layers. The substantial gain from full fine-tuning demonstrates that the pretrained BART benefits translation when it is allowed to adapt, but the improvement is more modest than on purely monolingual generation tasks. The authors acknowledge limitations: "Preliminary results suggested that our approach was less effective without back-translation data, and prone to overfitting—future work should explore additional regularization techniques" (Section 5.4).

Ablation Studies and Robustness Checks

Token masking vs. token deletion vs. text infilling (Table 1, bottom block): On generation tasks, token deletion consistently outperforms token masking (XSum PPL: 6.90 vs. 7.08; CNN/DM: 5.87 vs. 6.10; ConvAI2: 11.46 vs. 11.73), and text infilling further outperforms both (XSum: 6.61; CNN/DM: 5.83; ConvAI2: 11.05). On discriminative tasks (SQuAD, MNLI), all three are essentially tied. This demonstrates that the choice of noising scheme matters specifically for generation, with the ranking deletion > masking > infilling (where infilling means single-mask-token-per-span). However, text infilling + sentence permutation achieves the best CNN/DM perplexity (5.41), suggesting that sentence-level noising adds value primarily for tasks requiring discourse-level organization.
Sentence permutation in isolation vs. combined with text infilling (Table 1): Sentence permutation alone achieves only 85.4 SQuAD F1 and 81.5 MNLI accuracy, with generation perplexities far worse than token-level noising variants (XSum: 10.93 vs. 6.61 for text infilling; CNN/DM: 7.89 vs. 5.83). However, when combined with text infilling, it improves CNN/DM perplexity from 5.83 to 5.41 while maintaining comparable discriminative performance (90.8 → 90.8 SQuAD F1, 84.0 → 83.8 MNLI accuracy). This ablation confirms that sentence permutation is not viable as a standalone pretraining task but provides complementary benefits for certain generation tasks when paired with token-level corruption.
Document rotation (Table 1): Document rotation performs catastrophically across all metrics—77.2 SQuAD F1, 75.3 MNLI accuracy, 53.69 ELI5 PPL (more than double any other variant), 17.14 XSum PPL, 19.87 ConvAI2 PPL, 10.59 CNN/DM PPL. This strongly negative result validates that identifying the document start is not a sufficiently rich pretraining task, and that destroying all local token order destroys learnable structure. The authors do not ablate rotation combined with other noising schemes.
Presence vs. absence of left-to-right autoregressive pretraining (Table 1): The Masked Language Model and Permuted Language Model are the only objectives that do not include left-to-right autoregressive language modeling during pretraining. On generation tasks, these models consistently underperform: on XSum, MLM achieves 7.87 PPL and Permuted LM achieves 7.69 PPL, versus 6.61 for text infilling and 7.00 for Language Model. On ConvAI2, MLM achieves 12.59 PPL and Permuted LM achieves 12.23 PPL, versus 11.05 for text infilling. On CNN/DM, MLM achieves 7.06 PPL and Permuted LM achieves 6.96 PPL, versus 5.83 for text infilling. The authors conclude: "Left-to-right pre-training improves generation" (Section 4.3). This is an important mechanistic finding: autoregressive pretraining provides a measurable advantage for generation tasks that non-autoregressive objectives cannot fully compensate for, even when both use bidirectional encoders.
Effect of sentence permutation at scale (Section 5.1, Table 1 vs. Table 3): In the base-size ablation, text infilling + sentence permutation shows only marginal gains over text infilling alone (CNN/DM PPL: 5.41 vs. 5.83; other tasks nearly identical). The authors hypothesized that "larger pre-trained models may be better able to learn from this task." The large model results do not directly ablate this (the large BART uses text infilling + sentence permutation without a text-infilling-only large baseline for comparison), so it remains unclear whether sentence permutation provides outsized benefits at scale or whether the large model's strong results are primarily due to the text infilling component. This is a missing ablation that would strengthen the paper's claims about the importance of sentence permutation.
Encoder-decoder format vs. decoder-only format for non-BART models (Section 4.1): For the reimplemented baselines (Permuted LM, Masked LM, Multitask Masked LM), the paper experiments with two architectural formats: (1) standard encoder-decoder where the corrupted input goes to the encoder and the target to the decoder, or (2) decoder-only where the source is prefixed to the target in the decoder with loss only on the target portion. The finding that "the former works better for BART models, and the latter for other models" confirms that architectural assumptions affect objective performance. The reported results in Table 1 use the better-performing format for each model, making the comparison fair to each objective.
Label smoothed cross-entropy and beam search tuning (Section 5.3): For generation fine-tuning, BART uses label smoothing (0.1) and beam search (size 5) with tuned minimum length, maximum length, and length penalty on the validation set. No ablation is reported for label smoothing versus standard cross-entropy. No ablation is reported for beam size, length penalty sensitivity, or the effect of duplicated trigram removal. These hyperparameters are important for generation quality, and their contribution relative to the pretraining objective is unknown. Given that Liu et al. (2019) showed optimization details can matter as much as pretraining objectives for BERT, this is a relevant omission.
Two-step vs. single-step MT fine-tuning (Table 6): The fixed BART variant (only training the new encoder) achieves 36.29 BLEU, slightly below the 36.80 BLEU baseline and substantially below the 37.96 BLEU of fully tuned BART. This demonstrates that freezing most of BART prevents the model from adapting its internal representations to the source language, and that the full fine-tuning step is necessary to realize gains. The two-step procedure can be viewed as an ablation where the first step confirms that BART's English representations are not immediately usable for Romanian input, and the second step demonstrates that adaptation is possible.

Critical Assessment

Does BART genuinely unify understanding and generation in a single model, or does it simply achieve good numbers on both by being a large model trained on lots of data? The evidence in Table 2 shows that BART matches RoBERTa on GLUE and SQuAD within fractions of a point. But RoBERTa was explicitly optimized for discriminative tasks and cannot do generation at all—it has no decoder. So BART achieves the same discriminative performance while also achieving state-of-the-art generation results (Tables 3–5). This is genuinely a unification: the same weights, same architecture, same pretraining run produces both capabilities simultaneously. The controlled ablation in Table 1 strengthens this claim by showing that the noising scheme, not just scale, is responsible—different objectives within the same architecture produce dramatically different task-specific performance profiles, and BART's text infilling is the only one that is strong across all categories. The claim of unification is well-supported, with the caveat that the comparison is at a single scale (large model, 500K steps, 160GB data) and may not hold at all scales or data regimes.

Does the 6 ROUGE improvement on XSum represent genuine progress, or could it be partly attributable to training data scale, optimization, or decoding strategies not available to prior work? The 6 ROUGE gain on XSum is the most dramatic number in the paper and requires scrutiny. Several prior systems in Table 3 did not have access to the same scale of pretraining data (160GB of diverse text) or the extensive hyperparameter tuning that RoBERTa-style training enables. BERTSUMEXTABS builds on BERT, which was pretrained on 16GB of books + Wikipedia—an order of magnitude less data. It is possible that some fraction of the 6-point gain comes from data scale and optimization rather than the BART architecture or noising scheme. The paper partially addresses this by matching RoBERTa's training setup (same data, same batch size, same number of steps) for its large model, which controls for the data and optimization factors relative to RoBERTa, but RoBERTa itself cannot do generation, so there is no direct "RoBERTa-with-generation-capability" baseline to isolate the architectural contribution. A controlled experiment that trained a BART-style encoder-decoder model with BERT's masking objective (rather than text infilling) at the same data scale would help isolate the contribution of the noising scheme, but this is not reported. The 6-point gain is likely a combination of architecture, noising scheme, data scale, and optimization improvements rather than purely attributable to BART's design choices.

Is BART genuinely better than prior sequence-to-sequence pretraining (MASS), or are the gains due to training scale and controlled comparison methodology? MASS (Song et al., 2019) is the most architecturally similar prior work. The Masked Seq-to-Seq baseline in Table 1 serves as a MASS-inspired reimplementation. At base scale with controlled training, text infilling outperforms Masked Seq-to-Seq on SQuAD (90.8 vs. 87.0 F1), MNLI (84.0 vs. 82.1), and generation tasks (XSum: 6.61 vs. 6.80 PPL; CNN/DM: 5.83 vs. 6.19; ConvAI2: 11.05 vs. 11.43). The controlled comparison is convincing: text infilling genuinely improves over contiguous span masking when architecture, data, and training budget are held constant. However, the comparison is BART's text infilling against a simplified MASS (masking 50% of tokens in one contiguous span), not against the full MASS model with MASS's exact training recipe. The gaps on discriminative tasks are notably larger than on generation, suggesting BART's advantage may be strongest for transfer to understanding tasks where MASS's disjoint encoder/decoder inputs create a mismatch.

Does BART's competitive discriminative performance truly demonstrate that the unidirectional decoder imposes no penalty, or are there tasks where the penalty would be more visible? On the GLUE and SQuAD tasks evaluated, BART matches RoBERTa. These tasks primarily require sentence-level or token-level understanding. However, there are discriminative tasks that require autoregressive prediction (e.g., language modeling as a classification task, or tasks requiring sequential reasoning over long contexts) where BART's decoder might have an architectural advantage over encoder-only models—and conversely, tasks requiring only bidirectional encoding might be slightly penalized by the additional decoder parameters that contribute nothing to the prediction. Table 2 shows BART slightly behind RoBERTa on STS-B (91.2 vs. 92.4) and CoLA (62.8 vs. 68.0), but ahead on SST-2 and QQP. These differences are within typical task variance, but the pattern is not uniformly "BART = RoBERTa"—there is task-specific variation that the paper does not analyze in detail.

Are the ablation results at base scale (Table 1) predictive of large-scale performance? The paper uses base-scale ablations to justify design choices (text infilling + sentence permutation) that are then applied to the large model. However, the base model uses only 6 encoder + 6 decoder layers on books + Wikipedia data, while the large model uses 12+12 layers on 160GB of diverse data. The finding that sentence permutation adds only marginal benefit at base scale but is hypothesized to matter more at large scale is plausible but untested—there is no large-scale ablation of text infilling alone vs. text infilling + sentence permutation. The base-scale ablation also does not sweep masking rates (30% for large model vs. BERT's 15%), leaving open the question of whether the masking rate or the noising scheme drives the large model's performance. These are genuine gaps between the ablation analysis and the final model.

Missing ablation: effect of Poisson distribution parameter $\lambda$ for text infilling span lengths. The paper specifies $\lambda = 3$ for the Poisson distribution governing span lengths but provides no ablation over $\lambda$ values. Different $\lambda$ values would produce different distributions of span lengths (more short spans with lower $\lambda$ , more long spans with higher $\lambda$ ), which could substantially affect what the model learns. This is a notable omission given that the paper emphasizes the importance of text infilling's length ambiguity—the $\lambda$ parameter directly controls how much length ambiguity the model encounters during training.

Missing baseline: BART architecture trained with vanilla BERT masking. The paper compares BART variants against a reimplemented Masked Language Model objective, but the MLM baseline uses a decoder-only format (Section 4.1: "the latter [decoder-only format] for other models"). There is no baseline where BART's full encoder-decoder architecture is trained with standard BERT masking (15% individual tokens replaced with MASK, predicting them conditionally independently from the decoder). This would isolate whether the architectural unification (encoder-decoder) or the noising scheme (text infilling) is the primary driver of performance. The existence of such a baseline would directly answer the question: "Is it BART's architecture or BART's noising that matters?" Since it's absent, the architecture and noising contributions remain partially confounded.

The ELI5 exception is important and underexamined. The paper notes that ELI5 is "the only generation task where other models outperform BART" (Section 4.3) and attributes this to the output being "only loosely constrained by the input." This explanation is speculative—the paper does not analyze what specific properties of ELI5 make BART's pretraining less suitable. ELI5 answers are long, often multiple paragraphs, and require integrating information from supporting documents. BART's maximum generation length may be a limitation, or the denoising pretraining may not adequately train the model to handle very long outputs. Understanding why ELI5 is different would strengthen the paper's claims about BART's generality, but this analysis is absent.

The machine translation experiment is a proof of concept with acknowledged limitations. The 1.1 BLEU gain on WMT16 RO-EN is over a back-translation-augmented baseline, and the authors note that the approach is "less effective without back-translation data, and prone to overfitting" (Section 5.4). This limits the generality of the translation approach—it works in a data-rich setting with synthetic parallel data but may not transfer to truly low-resource translation without back-translation. The two-step training procedure also introduces additional hyperparameters (how many steps to train in each phase) that are not ablated. The translation results are best viewed as demonstrating architectural flexibility (BART can be used for MT) rather than a practical translation method.

6. Limitations and Trade-offs

Limitation 1: The Ablation Experiments Use Base-Scale Models Only, While the Strongest Claims Rely on Large-Scale Training

The assumption or constraint. All controlled comparisons of pretraining objectives (Section 4) are conducted at base model scale (6 encoder + 6 decoder layers, hidden size 768) trained for 1M steps on books + Wikipedia data. The large model (12 + 12 layers, hidden size 1024) trained for 500K steps on 160GB of diverse data—which produces the headline state-of-the-art results in Section 5—is not included in the ablation comparison. The authors explicitly acknowledge a gap between these scales when discussing sentence permutation: "Although sentence permutation only shows significant additive gains on the CNN/DM summarization dataset, we hypothesised that larger pre-trained models may be better able to learn from this task" (Section 5.1). This hypothesis is never tested—there is no large-scale BART trained with text infilling alone for comparison.

The consequence. The paper's central design claim—that text infilling + sentence permutation is the optimal noising scheme—is based on base-scale ablations but applied to a large-scale model without verification that the ranking of noising schemes is preserved across scales. Several factors could cause reversal: (a) the large model uses different data (160GB of diverse text vs. books + Wikipedia), so the optimal noising scheme may differ; (b) the large model uses a 30% masking rate, while the base model masking rate is unspecified and may differ; (c) larger models may benefit more from certain noising schemes if they have capacity to learn the more complex patterns, or less if the noising scheme becomes too easy at scale. The finding that sentence permutation adds marginal benefit at base scale but is assumed to matter more at large scale is an untested extrapolation. A practitioner cannot know whether the 6 ROUGE gain on XSum (Table 3) is primarily attributable to the specific noising scheme or to the scale of data and model—the two are confounded in the absence of a large-scale text-infilling-only ablation.

What evidence exists in the paper. The entire evidence base for the noising scheme selection is Table 1, which uses base models. The large-scale results in Tables 2–6 are for a single BART configuration only (text infilling + sentence permutation). There is no large-scale ablation varying the noising scheme. The authors' hypothesis about sentence permutation benefiting more at scale is stated (Section 5.1) but not tested. This is a genuine gap in the experimental design: the controlled comparison methodology that distinguishes the paper's approach (Section 4) is not carried through to the large-scale setting.

Mitigation status. The paper does not address this gap. The authors present the base-scale ablation as sufficient justification for the large-scale design choices, but they do not acknowledge the extrapolation as a limitation or suggest that future work should verify the ranking of noising schemes at scale. This is a missed opportunity given the paper's own emphasis on the importance of controlled comparisons (Section 4 introduction: "fair comparisons between these have been difficult to perform, at least in part due to differences in training data, training resources, architectural differences between models, and fine-tuning procedures"). The same logic should apply to comparisons across model scales, but it is not applied.

Limitation 2: The ELI5 Outlier Is Unexplained and Reveals a Boundary Condition on BART's Effectiveness for Generation

The assumption or constraint. BART is presented as a general-purpose pretraining method for both understanding and generation. The abstract claims BART "achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks." However, the results reveal a clear exception: on the ELI5 long-form abstractive question answering dataset, BART is not the best method at base scale, and its advantage at large scale is modest relative to other generation tasks.

The consequence. The ELI5 results point to a regime where BART's pretraining is less effective, but the paper does not characterize this regime precisely. In the base-scale ablation (Table 1), BART with text infilling achieves 24.26 PPL on ELI5, worse than the Language Model (21.40), Masked Seq-to-Seq (23.40), and Multitask Masked LM (23.73). At large scale (Table 5), BART achieves 30.6 ROUGE-1 / 6.2 ROUGE-2 / 24.3 ROUGE-L, improving over prior work by 1.7 ROUGE-1, but the gains are an order of magnitude smaller than on XSum (~6 ROUGE across all metrics). The authors attribute this to ELI5 outputs being "only loosely constrained by the input" (Section 4.3) and note that "the dataset remains challenging" (Section 5.3). But this explanation is underspecified: why does loose input-output coupling favor language models over denoising autoencoders? Is it the length of answers (ELI5 answers are multi-paragraph, while XSum summaries are 1-2 sentences)? The degree of abstraction? The need for external knowledge beyond what is in the input documents? Without diagnosing the cause, a practitioner cannot predict whether BART will underperform on their loosely-constrained generation task or whether their task is "loosely constrained" in the relevant sense.

What evidence exists in the paper. Table 1 shows that at base scale, the Language Model achieves 21.40 ELI5 PPL vs. 24.26 for text infilling—a 13% relative degradation. This is the largest gap favoring any non-BART method on any task in the ablation. Table 5 shows the large-scale BART achieves state-of-the-art on ELI5 but with modest absolute gains (1.2 ROUGE-L over prior work, compared to 6.0 ROUGE-L on XSum). The paper acknowledges ELI5 as an outlier (Section 4.3: "The ELI5 dataset is an outlier, with much higher perplexities than other tasks, and is the only generation task where other models outperform BART") but provides no mechanistic analysis.

Mitigation status. The paper does not attempt to diagnose or address the ELI5 weakness. The authors note it as an observation, not as a research question. No experiments vary answer length, input-output coupling strength, or knowledge requirements to isolate the factor driving BART's relative underperformance. This leaves a significant gap in understanding the boundaries of BART's effectiveness. The paper suggests in the conclusion (Section 8) that "future work should explore new methods for corrupting documents for pre-training, perhaps tailoring them to specific end tasks," which could address this limitation indirectly, but does not propose specific diagnostics for the ELI5 case.

Limitation 3: Machine Translation Gains Are Modest, Depend on Back-Translation Data, and the Method Overfits Without It

The assumption or constraint. The paper presents BART's machine translation approach (Section 3.4) as a novel way to use pretrained sequence-to-sequence models for translation—treating the entire BART as a target-side decoder with a new source encoder grafted on. The claim is that this "provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining" (abstract).

The consequence. The translation approach has two practical limitations that make it less general than the paper's framing suggests. First, the 1.1 BLEU gain is over a baseline that already uses back-translation—synthetic parallel data generated by translating target-language monolingual data into the source language. The authors explicitly state: "Preliminary results suggested that our approach was less effective without back-translation data, and prone to overfitting—future work should explore additional regularization techniques" (Section 5.4). This means the method does not work well with only the original parallel corpus; it requires augmented data. For truly low-resource language pairs where back-translation is difficult (because no good reverse translation model exists), BART's approach may fail entirely.

Second, the two-step training procedure (fixed BART → tuned BART) introduces additional hyperparameters and training phases that are not ablated. The fixed BART step (Table 6) actually underperforms the baseline Transformer (36.29 vs. 36.80 BLEU), meaning the pretrained BART is not immediately useful—it requires careful unfreezing and fine-tuning to realize gains. The full fine-tuning step recovers and surpasses the baseline (37.96 BLEU), but the 1.1 BLEU gain, while statistically meaningful, is modest compared to the generation task improvements (6 ROUGE on XSum). A practitioner considering using BART for translation would need to weigh this modest gain against the additional complexity of the two-step training procedure and the requirement for back-translation data.

What evidence exists in the paper. Table 6 reports all three data points: Baseline Transformer-large (36.80 BLEU), fixed BART (36.29 BLEU, worse than baseline), and tuned BART (37.96 BLEU, 1.16 BLEU improvement). The back-translation dependency is acknowledged in the text (Section 5.4). The translation experiment is evaluated on only one language pair (WMT16 Romanian-English), so the generalization to other language pairs—particularly those with different typological distance from English or different data availability—is unknown. The two-step hyperparameters (number of steps in each phase) are not specified or ablated.

Mitigation status. The paper acknowledges the back-translation dependency and overfitting as limitations and suggests "future work should explore additional regularization techniques" (Section 5.4). However, it does not propose specific regularization methods, test on additional language pairs, or evaluate in genuinely low-resource settings without back-translation. The translation experiment is best understood as a proof of concept demonstrating architectural flexibility rather than a practical translation method ready for deployment, but the paper does not make this distinction explicit.

Limitation 4: The Base-Scale Ablation Is Performed on a Narrow Data Distribution (Books + Wikipedia), Limiting Generalization of the Noising Scheme Rankings

The assumption or constraint. The controlled ablation study in Section 4—which is the paper's primary evidence for the superiority of text infilling over alternative noising schemes—trains all models on "a combination of books and Wikipedia data" for 1M steps (Section 4.1). This is the same data used by BERT (Devlin et al., 2019) and represents a relatively clean, formal, well-edited text distribution. The large-scale model, by contrast, is trained on the RoBERTa data mixture: "160Gb of news, books, stories, and web text" (Section 5.1)—a substantially more diverse and noisy distribution.

The consequence. The optimal noising scheme may depend on the pretraining data distribution. Text infilling, which requires the model to predict missing spans of text, may be particularly effective on well-edited text where the surrounding context provides strong signals about the missing content. On noisier web text—which contains typos, informal language, inconsistent formatting, and non-sequiturs—the same noising scheme might teach different (potentially less useful) patterns. For example, if the pretraining data contains incoherent text, learning to reconstruct missing spans from unreliable context could actually harm the model's language understanding. The paper's claim that text infilling + sentence permutation is generally the best noising scheme is based on books + Wikipedia data and extrapolated to the diverse 160GB mixture without verification. A practitioner pretraining on a different data distribution (e.g., domain-specific text, social media, code) cannot assume the ranking of noising schemes from Table 1 will transfer.

Additionally, the base-scale ablation uses models with only 6 encoder and 6 decoder layers. The optimal noising scheme may interact with model depth: deeper models may be better able to handle more destructive noising (because they have more layers to reconstruct the lost information), or they may benefit more from noising schemes that require longer-range reasoning. The paper's hypothesis that sentence permutation benefits more at larger scale (Section 5.1) is an example of this potential interaction, but it is tested only for one noising scheme and only by assumption, not experiment.

What evidence exists in the paper. Table 1 shows clear rankings among noising schemes on books + Wikipedia data at base scale. Section 5.1 states the large model uses different data (RoBERTa's 160GB mixture) without re-running the ablation on that data. The paper provides no evidence that the text infilling advantage persists on more diverse or noisier text distributions. Section 4 does not include any experiments varying the pretraining data while holding the noising scheme constant.

Mitigation status. The paper does not address this limitation. The ablation methodology is presented as a contribution (controlling for architecture, training budget, and fine-tuning procedure), but the data distribution is another critical control variable that is not varied. The authors do not discuss the potential for data-dependent noising scheme rankings or suggest that future work should verify the ablation findings on more diverse data distributions.

Limitation 5: Generation Quality Is Evaluated Exclusively with Automated Metrics; No Human Evaluation Is Reported

The assumption or constraint. All generation task results in the paper are evaluated using automated metrics: ROUGE for summarization (CNN/DailyMail, XSum, ELI5), validation F1 and perplexity for dialogue (ConvAI2), and BLEU for machine translation. While the paper includes a qualitative analysis section (Section 6) with example summaries, there is no systematic human evaluation of output quality. The qualitative analysis acknowledges factual errors: in the first example summary, "the claim that the work was published in Science is not supported by the source" (Section 6, Table 7 discussion). This is reported as an observation, not as part of a systematic evaluation.

The consequence. Automated metrics—particularly ROUGE and BLEU—are known to correlate imperfectly with human judgments of summary quality, factual accuracy, and fluency. A 6 ROUGE improvement on XSum (Table 3) is statistically dramatic, but without human evaluation, a practitioner cannot know whether this translates to summaries that human readers would actually prefer. The gap could arise from BART producing more extractive summaries that happen to share n-grams with reference summaries (ROUGE rewards n-gram overlap) without improving informativeness or factual reliability. The qualitative examples (Table 7) show fluent, abstractive summaries that integrate information across the article and draw on background knowledge, which is suggestive but not systematic.

Factual accuracy is a particularly important dimension that automated metrics do not capture well. The qualitative analysis reveals at least one factual error (the Science journal claim), and a practitioner deploying BART for summarization would need to know the rate and severity of such errors. Without human evaluation, the paper's generation results are impressive on leaderboard metrics but incomplete as an assessment of practical utility. This is especially relevant for the XSum dataset, where summaries are highly abstractive and factual faithfulness is a known challenge.

What evidence exists in the paper. Section 6 provides 5 example summaries with qualitative commentary. The authors note that "model output is fluent and grammatical English" and "highly abstractive, with few phrases copied from the input," and that "output is also generally factually accurate" (emphasis on "generally"). They then immediately note a specific factual error. Table 7 is the full extent of the qualitative analysis. No quantitative human evaluation metrics (e.g., human preference ratings, factual accuracy judgments, fluency scores) are reported.

Mitigation status. The paper includes qualitative examples (Section 6) as a partial supplement to automated metrics, but this is not a substitute for systematic human evaluation. The authors acknowledge the factual error in the example but do not quantify the frequency of such errors across the full test set. No inter-annotator agreement, sample size, or evaluation protocol is described. The paper does not frame the absence of human evaluation as a limitation—it presents the qualitative analysis as confirming the automated metric gains, which is a weaker claim than the evidence supports.

Limitation 6: BART Contains ~10% More Parameters Than Equivalently Sized BERT, but Computational Cost Comparisons at Equal Model Size Are Absent

The assumption or constraint. The paper notes in Section 2.1 that "BART contains roughly 10% more parameters than the equivalently sized BERT model" because the decoder stack adds additional layers. For the base model, this means BART has 6 encoder + 6 decoder layers vs. BERT's 12 encoder layers—both have 12 total Transformer layers, but the decoder's cross-attention sublayers add parameters. For the large model comparison with RoBERTa (Table 2), BART has 12 encoder + 12 decoder layers (24 Transformer layers total with cross-attention) while RoBERTa has 24 encoder layers. The total parameter counts are not provided, but the architectures are not parameter-matched.

The consequence. The comparison with RoBERTa in Table 2—which the paper frames as showing that BART "performs comparably to RoBERTa and XLNet" on discriminative tasks—is not at equal parameter count or computational cost. BART has both more parameters (from the decoder's cross-attention) and a different layer configuration (half the layers are encoder, half decoder, vs. all encoder for RoBERTa). A practitioner choosing between BART and RoBERTa for a purely discriminative task would need to know: does BART match RoBERTa because of its architectural advantages (bidirectional encoder + autoregressive decoder), or simply because it has more parameters? If a RoBERTa model with BART's exact parameter count were trained, would it outperform BART on discriminative tasks? The paper cannot answer this question because it does not provide parameter-matched comparisons.

This matters for deployment decisions. If a practitioner only needs discriminative capabilities, RoBERTa (or another encoder-only model) with the same parameter count as BART would be both cheaper to train (fewer FLOPs per step due to no decoder cross-attention) and cheaper to run inference (no autoregressive decoder). The paper's claim that BART achieves "similar performance to RoBERTa on discriminative tasks" (Section 8) is accurate at the model size level (both are "large" models), but not at the parameter count level. The practical cost-performance tradeoff is not characterized.

What evidence exists in the paper. The parameter disparity is acknowledged in Section 2.1: "BART contains roughly 10% more parameters than the equivalently sized BERT model." However, this statement is not revisited in the experimental sections. Table 2 compares BART large against RoBERTa large, XLNet large, and BERT large—all of which have different architectures and parameter counts. No experiment controls for parameter count (e.g., by training a smaller BART and comparing against BERT at equal parameters, or by reporting FLOP-matched rather than layer-matched comparisons). The inference cost of the decoder during discriminative fine-tuning (where the decoder processes the input autoregressively, adding sequential computation) is not discussed or quantified.

Mitigation status. The paper does not address this as a limitation. The authors present the ~10% parameter difference as an architectural detail (Section 2.1) rather than as a confounding factor in the discriminative task comparisons. No parameter-matched or FLOP-matched baselines are proposed. A practitioner reading Table 2 might reasonably conclude that BART and RoBERTa are equivalent on understanding tasks, but this equivalence is achieved with different computational budgets, which the paper does not help the reader to account for.

7. Implications and Future Directions

How This Work Changes the Landscape

BART is not a paradigm shift in the sense of inventing a new architecture or training objective—it uses a standard Transformer sequence-to-sequence model and a conceptually straightforward denoising objective. Its impact is instead a reframing of the pretraining problem: it demonstrates that architectural unification (bidirectional encoder + autoregressive decoder) paired with flexible noising is sufficient to subsume the capabilities that previously required separate model families, and it provides a controlled experimental methodology for comparing pretraining objectives that had been lacking.

The most significant conceptual shift is the decoupling of noising function from architecture. Before BART, pretraining objectives were tightly bound to specific architectures: BERT's masked language modeling required a bidirectional encoder, GPT's language modeling required a causal decoder, XLNet's permutation required two-stream attention. BART shows that a single architecture—the standard encoder-decoder Transformer—can accommodate all of these objectives and more, simply by changing what gets corrupted and how the reconstruction is set up. This is not a claim about novel architecture; it is a claim about architectural generality enabling objective flexibility. The implication is that future pretraining research can focus on designing better corruption schemes without simultaneously needing to engineer new architectures to support them. This separation of concerns was not obvious before BART, and it makes the design space for pretraining substantially more modular.

BART also reconciles contradictory findings in the 2018–2019 pretraining literature. The field had accumulated evidence that (a) bidirectional context is crucial for understanding tasks (Devlin et al., 2019), (b) autoregressive generation is crucial for generation tasks (Radford et al., 2018), (c) span-level masking improves over token-level masking (Joshi et al., 2019), and (d) the pretraining objective matters as much as architecture for downstream performance (Liu et al., 2019). But these findings appeared to point in incompatible directions: how can a model be simultaneously bidirectional for understanding and autoregressive for generation? BART's answer—use a bidirectional encoder for the corrupted input and an autoregressive decoder for the reconstruction—is simple in retrospect but was not the field's default. XLNet had attempted to unify bidirectional context and autoregressive generation within a single stack using permutation, which introduced significant complexity. BART shows that the simpler encoder-decoder separation works at least as well, and does so without requiring specialized attention mechanisms. This shifted the field's intuition: rather than trying to get one stack to do everything, use two stacks that each do what they're good at.

The controlled ablation in Table 1 represents a methodological contribution independent of the model's performance. By reimplementing Language Model, Masked Language Model, Permuted Language Model, Multitask Masked Language Model, and Masked Seq-to-Seq within the same codebase, data, and training budget, BART established a template for how to compare pretraining objectives fairly. Before this, comparisons between BERT, GPT, XLNet, and UniLM conflated differences in architecture, data, optimization, and objective. Liu et al. (2019) (RoBERTa) had shown that optimization and data scale matter enormously—BART's ablation controls for these factors to isolate the effect of the objective. The finding that text infilling outperforms all prior objectives on most tasks, and that the ranking is task-dependent (language models win on ELI5, token-level noising is essential for most tasks, sentence permutation in isolation fails), is credible precisely because the confounds are controlled. This methodological standard influenced subsequent pretraining research, where controlled comparisons of objectives within a shared framework became more common.

The paper also redirects research attention from architecture design toward corruption design. The finding that document rotation and sentence permutation in isolation perform catastrophically (Table 1), while text infilling with single-mask-per-span representations performs best, is a diagnostic: it tells the field which kinds of corruption teach useful representations. Token-level corruption that preserves some local structure (masking, deletion, infilling) works; corruption that destroys global order without teaching token-level reconstruction (rotation, permutation alone) fails. This narrows the space of promising corruption schemes considerably. The paper explicitly invites exploration of this space: "there is a significant potential for development of other new alternatives" (Section 2.2). This framing—treating noising as the primary research frontier rather than architecture—was influential in subsequent work on denoising pretraining objectives.

Finally, BART establishes that pretrained sequence-to-sequence models can serve as complete target-language decoders for machine translation, reframing translation as a denoising problem. This is a conceptual shift from the dominant paradigm of pretraining encoders for source languages or pretraining on both source and target. While the translation results are modest (1.1 BLEU) and depend on back-translation data, the architectural flexibility BART demonstrates—grafting a new encoder onto a frozen pretrained decoder—opens a new way of thinking about transfer learning for MT that does not require pretraining on the source language.

Follow-Up Research This Work Enables

1. Scaling the controlled ablation to large model sizes and diverse data distributions. The paper's ranking of noising schemes (Table 1) comes from base-scale models trained on books + Wikipedia data. The large-scale BART uses text infilling + sentence permutation on 160GB of diverse data, but there is no large-scale ablation confirming that this ranking holds. A strong follow-up would train multiple large-scale BART variants—text infilling alone, text infilling + sentence permutation, token deletion, token masking—on the same 160GB data mixture and compare them on the full task suite (GLUE, SQuAD, XSum, CNN/DM, ConvAI2, ELI5). This is expensive but necessary: if the ranking changes at scale (e.g., sentence permutation provides zero benefit at large scale, or token deletion surpasses text infilling), then the paper's central design recommendation is scale-dependent. A negative result here—finding that the base-scale ranking does not generalize—would be as valuable as a positive one, because it would reveal that noising scheme selection must be done at target scale. The paper's hypothesis that "larger pre-trained models may be better able to learn from" sentence permutation (Section 5.1) is testable with this experiment.

2. Characterizing the ELI5 boundary: when does denoising pretraining underperform language modeling? The ELI5 dataset is the single clear case where BART's denoising pretraining is outperformed by a simple language model at base scale (Table 1: 24.26 PPL for text infilling vs. 21.40 for LM), and where large-scale gains are modest (Table 5: +1.2 ROUGE-L over prior work vs. +6 on XSum). The paper attributes this to ELI5 outputs being "only loosely constrained by the input" (Section 4.3), but this explanation is vague and untested. A diagnostic follow-up would systematically vary the degree of input-output coupling in a controlled generation task—for example, by taking a summarization dataset and progressively increasing the abstractiveness (from extractive to highly abstractive) or decreasing the input-output overlap (by subsampling input sentences). If BART's advantage over language models degrades smoothly as input-output coupling weakens, that would confirm the "loose constraint" hypothesis and provide a quantitative boundary. If the degradation is abrupt at some threshold, it would suggest a specific architectural limitation (e.g., the cross-attention mechanism fails when input-output alignment is too diffuse). This experiment would help practitioners predict when BART-style pretraining is worth the cost vs. a simpler language model.

3. Training a difficulty estimator or data-dependent noising selector. The paper treats the noising scheme as fixed: text infilling + sentence permutation for the large model. But Table 1 shows that different noising schemes excel on different tasks—language models outperform BART on ELI5, token deletion outperforms masking on generation, and text infilling is the best all-rounder. This suggests that the optimal noising scheme may be task-dependent or even instance-dependent. A natural extension is to train BART with a mixture of noising schemes during pretraining, and then learn a lightweight selector that, given a downstream task's data distribution (or even individual examples), chooses which noising scheme to emphasize during fine-tuning. Concretely: pretrain with text infilling 50% of the time, token deletion 30%, and language modeling 20%, then at fine-tuning time, run a small hyperparameter sweep per task to select the optimal mixture ratio. If the optimal mixture varies systematically by task (e.g., more language modeling for ELI5-like tasks, more infilling for summarization), this would establish noising-scheme selection as a standard part of the fine-tuning pipeline. The paper's existing results provide the motivation; the experiment would add a practical tool.

4. Verifying factual accuracy degradation in BART's abstractive summaries. The qualitative analysis (Section 6, Table 7) reveals at least one factual error: "the claim that the work was published in Science is not supported by the source." This is presented anecdotally, not systematically. A rigorous follow-up would conduct a human evaluation of BART's XSum summaries, measuring factual accuracy (do all claims in the summary appear in or follow from the source article?), faithfulness (does the summary contradict the source?), and hallucination rate (what fraction of summaries contain unsupported claims?). The 6 ROUGE gain on XSum is the paper's most dramatic result; human evaluation would determine whether it comes at the cost of factual reliability. If BART's summaries are significantly less factual than extractive baselines, that would be an important caveat to the ROUGE gains. If they are comparably factual, that would strengthen the paper's claims. This experiment is straightforward to design: sample 100-200 XSum articles, collect summaries from BART and from BERTSUMEXTABS (the prior best), and have human annotators (blind to model identity) rate each summary on factual accuracy and overall quality. The paper makes this experiment possible by establishing the model and providing qualitative evidence that the question is worth asking.

5. Stress-testing the translation approach on low-resource language pairs without back-translation. The paper acknowledges that BART's MT approach is "less effective without back-translation data, and prone to overfitting" (Section 5.4). This is a significant boundary condition that limits the method's practical value for truly low-resource translation, where back-translation is difficult because no good reverse translation model exists. A stress-test would evaluate the approach on a genuinely low-resource language pair (e.g., Nepali-English or Kazakh-English from the FLoRes dataset, with only the available parallel data and no synthetic augmentation). The experiment would compare: (a) the BART-based approach (new encoder + frozen BART → fully tuned BART) against (b) a standard Transformer trained from scratch on the same parallel data, and (c) a Transformer initialized with a pretrained target-side language model (GPT-2 or similar). If BART fails entirely without back-translation (BLEU near random), that would confirm the method requires large parallel corpora and is not a low-resource solution. If it provides even modest gains (0.5-1 BLEU) without back-translation, that would substantially strengthen the method's generality. The experiment would also test whether the two-step training procedure can be replaced with a single step (joint training from scratch) to simplify the approach, and whether the overfitting problem can be addressed with standard regularization (dropout, label smoothing, early stopping).

6. Teasing apart the contributions of architecture vs. noising scheme at scale. The paper's claim that BART unifies understanding and generation rests on two design choices: (1) the encoder-decoder architecture, and (2) the text infilling + sentence permutation noising scheme. These are confounded in the large-scale experiments. A clean ablation would train a BART-architecture model (encoder-decoder) with standard BERT masking (15% individual tokens, predict independently from decoder) at large scale on the same 160GB data. If this "BERT-masking BART" matches RoBERTa on discriminative tasks but significantly underperforms text-infill BART on generation, that isolates the noising scheme as the driver of generation quality. If it matches text-infill BART on generation as well, then the architecture alone (encoder-decoder) is sufficient and the noising scheme matters less than the paper claims. A third condition—training a standard encoder-only RoBERTa and adding a randomly initialized decoder at fine-tuning time for generation tasks—would test whether the decoder needs to be pretrained or can be trained from scratch during fine-tuning. This three-way comparison (BART architecture + BART noising, BART architecture + BERT noising, BERT architecture + decoder from scratch) would decompose the contributions of architecture and noising, answering the central question the paper raises but does not fully resolve.

Practical Applications and Downstream Use Cases

1. Unified NLP pipelines that handle both understanding and generation with a single model. The most direct practical application is deploying BART in settings where a system needs to both comprehend and produce text. For example, a customer support system might need to classify incoming messages (understanding: sentiment analysis, intent classification) and generate responses (generation: dialogue). Before BART, such a system would typically use separate models—a BERT-based classifier and a GPT-based generator—requiring two model deployments, two serving infrastructures, and twice the memory footprint. BART's matched performance on GLUE/SQuAD (Table 2: 88.8 EM / 94.6 F1 on SQuAD 1.1, 89.9/90.1 on MNLI) and state-of-the-art generation (Table 4: 20.72 F1 on ConvAI2 dialogue) means a single model can serve both roles. The practical benefit is reduced operational complexity: one model to maintain, one set of weights to update, one serving endpoint to scale. The memory savings are roughly 50% (one model instead of two of comparable size), which matters for edge deployment or multi-tenant serving. The tradeoff is that BART's decoder adds inference latency for classification tasks (the input must be processed autoregressively), which pure encoder models avoid. For applications where classification throughput is the bottleneck, an encoder-only model may still be preferable despite the operational complexity of maintaining separate models.

2. Abstractive summarization at production quality levels previously requiring extractive methods. The 6 ROUGE improvement on XSum (Table 3: 45.14 ROUGE-1 / 22.27 ROUGE-2 / 37.25 ROUGE-L) represents a step change in abstractive summarization quality. Before BART, extractive methods (selecting and concatenating source sentences) were competitive with abstractive methods on many datasets, and the best abstractive systems (BERTSUMABS, UniLM) produced summaries that were still substantially extractive in nature. BART's XSum results suggest that abstractive summarization can now outperform extractive approaches by a wide margin on tasks requiring genuine rewriting. A news aggregation service could fine-tune BART on XSum to produce one-sentence article summaries that are more informative than the first sentence of the article (the Lead-3 baseline achieves only 16.30 ROUGE-1 vs. BART's 45.14), directly improving user experience in news feeds. The beam search generation time (beam size 5, with trigram deduplication) would need to be accounted for in a latency budget, but for asynchronous summarization (batch processing of articles), the generation cost is acceptable. A practitioner would need to validate factual accuracy on their specific domain (the paper acknowledges at least one factual error in the qualitative examples) before deployment, but the ROUGE gains provide strong evidence that the summaries are at least more informative and better-written by automated measures.

3. Conversational AI with improved persona-consistent dialogue generation. BART's ConvAI2 results (Table 4: 20.72 F1, 11.85 PPL) demonstrate that the model can generate dialogue responses conditioned on both conversation history and a textual persona description, outperforming the best systems from the ConvAI2 competition. This has direct application in chatbots and virtual assistants where consistency with a defined persona matters. The key practical advantage over prior systems is the combination of low perplexity (fluency) and high F1 (relevance), suggesting that BART produces responses that are both natural-sounding and appropriate to the context and persona. A deployment would fine-tune BART on domain-specific dialogue data with persona descriptions, using the same label-smoothed cross-entropy and beam search setup described in Section 5.3. The primary deployment consideration is generation latency: beam search with beam size 5 over the full vocabulary is slower than greedy decoding, and dialogue applications are latency-sensitive. A practitioner might experiment with smaller beam sizes or nucleus sampling at inference time to trade off quality for speed, an ablation the paper does not provide.

4. Target-side language model for improving machine translation into English. The MT approach in Section 3.4—using BART as a pretrained English decoder with a new source encoder—provides a 1.1 BLEU improvement on WMT16 Romanian-English (Table 6: 37.96 vs. 36.80 BLEU). While modest, this gain requires no source-language pretraining data, only target-language (English) monolingual data, which is abundant. For translation directions into English from low-resource languages where source-language pretraining is infeasible, this approach offers a way to leverage large English corpora to improve translation quality. The practical workflow would be: (1) pretrain BART on English monolingual data (or use the released BART checkpoint), (2) train a new 6-layer source encoder to map the source language into BART's expected input space using available parallel data, following the two-step procedure (freeze BART, train encoder; then unfreeze and fine-tune). A practitioner would need parallel data of at least modest size—the paper used WMT16 RO-EN augmented with back-translation—and should expect overfitting without data augmentation. The approach is not suitable for truly low-resource settings (sub-100K parallel sentences) without additional regularization, as the paper's own caveats indicate. For medium-resource translation into English, it provides a simple way to incorporate target-side pretraining without the complexity of pretraining on both source and target languages.

When to Prefer This Method

The paper explicitly positions BART against several named alternatives and provides sufficient evidence to articulate decision rules:

Prefer BART over BERT (or RoBERTa) when you need both understanding and generation from a single model. BERT cannot generate; BART can. If your application requires classification, extraction, and text production, BART's matched discriminative performance (Table 2) and state-of-the-art generation (Tables 3–5) make it the clear single-model choice. The cost is ~10% more parameters and slower inference on pure classification tasks due to the autoregressive decoder.
Prefer BART over GPT (or left-to-right language models) when bidirectional context matters for your task. The ablation (Table 1) shows language models achieving 76.7 SQuAD F1 vs. 90.8 for BART with text infilling—a catastrophic gap on extractive QA. If your task requires integrating information from both sides of a token (span extraction, natural language inference, sentence-pair tasks), BART's bidirectional encoder is essential.
Prefer BART over XLNet when you need strong generative capabilities with a simpler architecture. Both models handle bidirectional context and generation, but BART's standard encoder-decoder architecture is simpler to implement and fine-tune than XLNet's two-stream attention and permutation mechanism. The paper's ablation shows the reimplemented Permuted LM underperforming BART on generation (Table 1: XSum PPL 7.69 vs. 6.61), though the reimplementation is simplified (no relative positional embeddings). For practitioners who prioritize implementation simplicity and generation quality, BART is the more straightforward choice.
Prefer BART over MASS (or Masked Seq-to-Seq) when you need strong performance on both discriminative and generative tasks. The Masked Seq-to-Seq baseline (Table 1) underperforms BART on SQuAD (87.0 vs. 90.8 F1) and MNLI (82.1 vs. 84.0), confirming the paper's claim that MASS is "less effective for discriminative tasks, because disjoint sets of tokens are fed into the encoder and decoder" (Section 7). BART's unified input (same text to both encoder and decoder during fine-tuning) removes this mismatch.
Prefer a pure language model when input-output coupling is very weak. The ELI5 result (Table 1: Language Model achieves 21.40 PPL vs. BART's 24.26) provides a specific boundary condition. When the generation task is only loosely constrained by the input—long-form open-ended generation, creative writing, or tasks where the input serves as a prompt rather than a specification—a left-to-right language model may produce better quality output than a denoising autoencoder. The paper's explanation ("output is only loosely constrained by the input") provides a qualitative heuristic but not a quantitative threshold; a practitioner should validate on their specific task.