SpanBERT: Improving Pre-training by Representing and Predicting Spans

ArXiv: 1907.10529

🎯 Pitch

SpanBERT makes a simple but powerful change to BERT's masking: instead of hiding random words, it hides entire spans like "an American football game" and forces the model to predict them using only the boundary words. This alone slashes question-answering errors by up to 27% and shatters the coreference resolution record by 6.6 points, proving that what you mask matters more than how much data you have.

1. Executive Summary

This paper introduces SpanBERT, a pre-training method that extends BERT by (1) masking contiguous random spans rather than individual tokens (sampling spans from a geometric distribution, e.g., masking "an American football game" as a whole rather than individual words), and (2) training span boundary representations to predict the entire content of the masked span through a novel span-boundary objective (SBO) (having the representations of the tokens immediately outside the span reconstruct each token inside it). Evaluated against a well-tuned BERT replication on 17 benchmarks spanning question answering, coreference resolution, relation extraction, and GLUE, SpanBERT achieves substantial gains on span selection tasks in particular—reaching 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively (representing up to 27% error reduction over the tuned BERT baseline), 79.6% F1 on OntoNotes coreference resolution (exceeding the prior state of the art by 6.6 absolute points), and an average of +2.9% F1 across five additional extractive QA benchmarks—while also improving GLUE, establishing that better-designed pre-training objectives targeting span-level semantics provide gains complementary to those from increased data or model size, with single-sequence training (dropping the next-sentence-prediction objective) contributing independently to the improvements.

2. Context and Motivation

The Core Problem: BERT's Token-Level Masking Is Mismatched to Span-Level Reasoning

The fundamental problem this paper addresses is a representational mismatch between how BERT is trained and how it is used for many downstream NLP tasks. BERT pre-trains by masking individual WordPiece tokens uniformly at random and training the model to predict them from surrounding context. But many of the tasks to which BERT is applied—extractive question answering, coreference resolution, semantic role labeling—require reasoning about multi-token spans: identifying that "Denver Broncos" is the answer to "Which NFL team won Super Bowl 50?" or that "the Prime Minister" and "she" refer to the same entity. These tasks fundamentally operate on contiguous chunks of text, not isolated tokens.

This mismatch creates two related problems:

The pre-training signal is too easy when spans are present. When BERT masks tokens independently, it often masks only part of a multi-word entity or phrase. Predicting "Denver" when you can see "Broncos" right next to it (even if masked) is trivially easy because the adjacent word provides overwhelming co-occurrence cues. The model never needs to learn rich, compositional representations of the full span from broader context—it can rely on local lexical statistics. The paper makes this point directly:

"predicting 'Denver Broncos' is much harder than predicting only 'Denver' when you know the next word is 'Broncos'"

Masking at the token level under-challenges the model precisely where span-level semantic understanding would be most useful to learn.
Span-level information is not explicitly stored at accessible locations. Even when BERT does learn implicit span representations, there is no mechanism to ensure this information is concentrated at the span's boundary tokens—the very representations that downstream span selection models use. Extractive QA models and coreference systems typically construct span representations by concatenating the output embeddings of the start and end tokens of a candidate span (Lee et al., 2016, 2017; He et al., 2018). If the pre-training objective never incentivizes the boundary tokens to encode the span's internal content, those representations may be suboptimal for downstream use. The paper states this explicitly as a design goal:

"we would ideally like the representations for the end of the span to summarize as much of the internal span content as possible"

Why This Matters: The Prevalence and Difficulty of Span Reasoning

This problem has both practical and scientific significance.

Practical significance. The tasks that most obviously demand span-level reasoning—question answering and coreference resolution—are among the most commercially and scientifically important in NLP. Extractive QA powers search engines, virtual assistants, and reading comprehension systems. Coreference resolution is a critical preprocessing step for information extraction, dialogue understanding, and document summarization. Any improvement in these tasks directly translates to better end-user experiences and more capable downstream systems.

Moreover, the paper demonstrates that span-enhanced pre-training helps even on tasks that do not explicitly involve span selection, including sentence-level classification in GLUE and relation extraction in TACRED. This suggests that span reasoning is a foundational linguistic competence—the ability to understand that multi-word expressions form coherent units with shared semantic properties is useful broadly, not just when the task's output is itself a span.

Theoretical significance. The paper's approach addresses a deeper question about self-supervised representation learning: what is the optimal granularity of the pre-training task for the downstream tasks? BERT's token-level MLM operates at the finest possible granularity (subword units). SpanBERT moves up one level of linguistic abstraction—to non-overlapping, semantically coherent chunks. The paper can thus be read as an investigation into whether pre-training at the span level induces qualitatively different and more useful representations than pre-training at the token level, even when the architecture and data are identical. The large gains on span selection tasks (2.0–2.8% F1 on SQuAD, 2.7% F1 on coreference in ablation) provide evidence that the answer is yes.

Prior Approaches and Where They Fall Short

BERT's original training design (Devlin et al., 2019). BERT combines a token-level masked language model (MLM) with a next-sentence prediction (NSP) objective. Tokens are masked individually and uniformly at random. The NSP objective trains the model to distinguish whether two text segments are contiguous. While this design produced revolutionary results across NLP, the paper identifies specific shortcomings:

The token-level masking ignores multi-word expressions. A named entity like "New York City" might have "New" masked and "York" visible, making the prediction trivial via local bigram statistics. The model is never forced to predict an entire span from non-local context, which would require deeper understanding.
The two-segment NSP setup may actually harm representation quality. The paper finds (Section 5, confirmed in ablations) that bi-sequence training with NSP underperforms single-sequence training without NSP on most tasks. The authors hypothesize that forcing the model to condition on two potentially unrelated half-length segments "adds noise to the masked language model" and prevents the model from "learning longer-range features" within a single coherent document.

Linguistically-informed masking (ERNIE, Sun et al., 2019). Sun et al. proposed masking named entities and phrases—linguistically coherent units—rather than random tokens, and showed improvements on Chinese NLP tasks. This approach partially addresses the independency problem: by masking entire phrases, the model must predict each token from outside the phrase. However, the paper's ablation study (Table 6) finds that linguistically-informed masking (named entities, noun phrases) is not consistently better than random span masking—and sometimes underperforms it (e.g., noun phrase masking lags on TriviaQA by 1.1% F1). Moreover, linguistic masking depends on external NLP pipelines (named entity recognizers, constituency parsers), which introduce their own errors and biases, and are unavailable or unreliable for many languages and domains.

Concurrent span-masking work (XLNet, MASS). XLNet (Yang et al., 2019) masks spans of 1–5 tokens during pre-training but predicts them autoregressively rather than using a boundary-conditioned objective like SBO. MASS (Song et al., 2019) masks contiguous fragments for sequence-to-sequence generation but uses an encoder-decoder framework focused on generation tasks, and does not develop explicit span boundary representations for use in span selection downstream tasks. Neither work explicitly trains boundary-token representations to encode internal span content—the key innovation of SpanBERT's SBO.

Span selection models with generic pre-trained encoders. Prior to SpanBERT, the standard approach for task like extractive QA and coreference was to take a generically pre-trained encoder (typically BERT), fine-tune it on the downstream task, and construct span representations by concatenating the boundary token outputs from this encoder. But the encoder was never trained to ensure those boundary outputs contained the right kind of information. SpanBERT's SBO directly optimizes for this property during pre-training, so that at fine-tuning time, the boundary tokens already serve as effective span summaries.

The role of data and model scale. The paper explicitly positions itself against the trend of achieving gains through more data and larger models:

"While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pre-training tasks and objectives, which can also have a remarkable impact."

The implication is that objective design is an underexplored axis of improvement—complementary to scaling—and that the field's focus on data and parameters may have obscured gains available through better task design alone.

How SpanBERT Positions Itself

SpanBERT is not a radical departure from BERT but rather a targeted, principled modification that addresses these specific limitations while preserving BERT's architecture and training data. The paper's positioning can be understood along three dimensions:

1. Extending, not replacing, BERT. SpanBERT keeps the core MLM objective, the Transformer architecture, and the training corpus of BERT_base/large_. It changes only: (a) how tokens are selected for masking (contiguous random spans instead of independent tokens), (b) the auxiliary objective (SBO replacing NSP), and (c) the data sampling procedure (single contiguous segments instead of two half-segments). This controlled setup allows clear ablation of each component and ensures that gains can be attributed to the design changes rather than to differences in data, model size, or hardware.

2. Span-based pre-training as a general-purpose improvement. The paper does not design SpanBERT for a single task but evaluates it across 17 benchmarks spanning four task families. The consistent improvements—especially on span selection tasks but also on sentence classification and relation extraction—establish that span-level pre-training is not a task-specific trick but a broadly useful inductive bias. The paper explicitly notes that sentence-level GLUE tasks "might still benefit from implicit span-based reasoning (e.g., the Prime Minister is the head of the government)," suggesting that even tasks without explicit span outputs involve span-level semantic comprehension.

3. A single-sequence, span-focused alternative to BERT's bi-sequence NSP framework. The paper's finding that single-sequence training without NSP outperforms BERT's two-segment setup is presented not as the main contribution but as an important enabling choice. The ablation in Table 7 shows that the gains from span masking and SBO compound with the gains from single-sequence training, confirming that these are independent improvements. This also resolves a tension in the literature: BERT's original ablation found NSP helpful, but the paper attributes this to the confounding of NSP with the bi-sequence data format—the ablation controlled for the objective but not for the segment length reduction.

4. An improved baseline as a foundation. Unlike many papers that compare against the original (often undertuned) public BERT checkpoints, SpanBERT builds on a carefully reimplemented and optimized BERT baseline that already substantially outperforms Google's released model (e.g., 92.6 vs. 91.3 F1 on SQuAD 1.1, 85.9 vs. 83.3 F1 on SQuAD 2.0). This makes the reported gains over the baseline more meaningful—SpanBERT is not just fixing implementation weaknesses in the original BERT but genuinely advancing beyond a strong, well-tuned replication. The paper is transparent about this:

"While building on our baseline, we find that pre-training on single segments, instead of two half-length segments with the next sentence prediction (NSP) objective, considerably improves performance on most downstream tasks."

The baseline improvements stem from updated optimization (AdamW with epsilon 1e-8, 2.4M steps), dynamic on-the-fly masking (different masks each epoch), and removal of BERT's short-sequence pre-training phase—all consistent with the concurrent findings of RoBERTa (Liu et al., 2019b), which the paper acknowledges as contemporaneous work.

Summary of the Gap

The paper fills a specific void in the pre-training landscape circa 2019: while token-level MLM had proven remarkably effective, no method had systematically designed pre-training objectives to target span-level semantics—both in terms of what is masked (forcing compositional prediction from context rather than local co-occurrence) and where the information is stored (concentrating span content at boundary tokens where downstream models access it). SpanBERT's masking scheme and SBO directly address these two aspects, and the paper demonstrates through careful ablation that both contribute independently to gains, with the largest benefits on the tasks most dependent on span reasoning.

3. Technical Approach

3.1 Reader Orientation

SpanBERT is a modification of the BERT pre-training recipe that teaches a Transformer encoder to build better representations of multi-word text spans. The system solves the problem that BERT's token-level masking is mismatched to downstream tasks requiring span reasoning by (a) forcing the model to predict entire contiguous spans from their surrounding context, and (b) explicitly training the boundary tokens—which downstream span selection models use as span representations—to encode the full content of the masked span.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Span Masking Sampler — a data preprocessing step that selects which tokens to mask during pre-training. Unlike BERT, which selects individual tokens uniformly at random, this component samples contiguous spans from a geometric distribution (mean length 3.8 tokens, clipped at 10), then applies the standard BERT replacement rules (80% [MASK], 10% random, 10% unchanged) at the span level rather than token-by-token.
Transformer Encoder (BERT_base/large architecture) — the same deep bidirectional Transformer as BERT. It takes a sequence of up to 512 tokens with some spans masked, processes it through multiple layers of self-attention and feed-forward networks, and produces a contextualized vector $\mathbf{x}_i$ for each input position $i$ . This component is architecturally identical to BERT; all changes are in the objectives that train it.
Span Boundary Objective (SBO) Module — a small 2-layer feed-forward network that sits on top of the Transformer encoder output for specific positions. It takes the output representations of the tokens immediately before and after a masked span ( $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ ), concatenates them with a learned position embedding indicating the target token's offset from the left boundary, and produces a vector $\mathbf{y}_i$ used to predict the $i$ -th token in the masked span. This module is used only during pre-training; it is discarded at fine-tuning time.
Joint Loss Computation — for each masked token within a span, the system computes two losses and sums them: the standard masked language model (MLM) loss (predicting the token from the Transformer's own output at that position, $\mathbf{x}_i$ ) plus the SBO loss (predicting the same token from the boundary-derived representation $\mathbf{y}_i$ ). The input embedding matrix is tied (shared) between MLM predictions and SBO predictions, meaning both losses use the same token vocabulary projection.

Information flow during pre-training:

A single contiguous block of up to 512 tokens is sampled from the corpus (no two-segment pairing, no next-sentence prediction).
The span masking sampler selects random spans totaling ~15% of tokens and applies the [MASK]/random/unchanged replacement at the span level.
The Transformer encoder processes the entire modified sequence and produces output vectors $\mathbf{x}_1, \ldots, \mathbf{x}_n$ .
For each masked token at position $i$ inside a span $(s, \ldots, e)$ :
- The MLM head (a linear projection using the tied input embedding weights) takes $\mathbf{x}_i$ and produces a probability distribution over the vocabulary. Cross-entropy loss $\mathcal{L}_{\text{MLM}}(x_i)$ is computed against the true token.
- The SBO module takes $\mathbf{x}_{s-1}$ , $\mathbf{x}_{e+1}$ , and the relative position embedding $\mathbf{p}_{i-s+1}$ , passes them through a 2-layer GeLU network with layer normalization, and produces $\mathbf{y}_i$ . The same vocabulary projection is applied to $\mathbf{y}_i$ , and cross-entropy loss $\mathcal{L}_{\text{SBO}}(x_i)$ is computed against the same true token.
- The total loss for token $i$ is $\mathcal{L}(x_i) = \mathcal{L}_{\text{MLM}}(x_i) + \mathcal{L}_{\text{SBO}}(x_i)$ .
Gradients from both losses flow back through the Transformer encoder, training the boundary token representations ( $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ ) to encode information sufficient to reconstruct the entire hidden span.

Information flow during fine-tuning (e.g., for extractive QA):

The SBO module is discarded entirely — it was only a pre-training scaffold.
The pre-trained Transformer encoder is used as a feature extractor: it processes the passage+question input and produces contextualized token representations.
For a candidate answer span from position $s$ to $e$ , the span representation is constructed by concatenating $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ (or $\mathbf{x}_s$ and $\mathbf{x}_e$ , depending on the task) — these are the exact boundary representations that the SBO trained to encode internal span content.
Task-specific heads (e.g., linear classifiers for start/end positions) operate on these boundary-derived span representations.

3.3 Roadmap for the Deep Dive

First, the span masking scheme (Section 3.1): how spans are selected (geometric distribution, $p=0.2$ , $\ell_{\text{max}}=10$ ), why whole words are the atomic unit, and why random spans rather than linguistic units were chosen. This defines what the model must predict.
Second, the span boundary objective (Section 3.2): the mathematical definition of the SBO, the architecture of the 2-layer boundary prediction network, how position embeddings encode relative offset, why the same input embedding matrix is reused for both MLM and SBO predictions, and how the loss is summed. This defines how the model learns span-level representations and where that information is stored.
Third, the single-sequence training pipeline (Section 3.3): the decision to drop next-sentence prediction and sample single contiguous segments of up to 512 tokens, why the authors believe this outperforms BERT's bi-sequence approach, and how this interacts with span masking and SBO to produce cumulative gains.
Fourth, the interaction between these three components: explaining why single-sequence training, span masking, and SBO are complementary rather than redundant, and what each contributes empirically (referencing the ablation results from Tables 6-7 while focusing on the mechanism of interaction here).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a pre-training method paper whose core idea is that masking contiguous random spans and adding a boundary-conditioned prediction objective teaches the model to build better span representations—representations that are stored at the very boundary tokens used by downstream span selection models.

Span Masking Scheme

The span masking scheme replaces BERT's independent-token masking with a process that selects contiguous spans of whole words for masking. This is the first of SpanBERT's two key innovations, and it addresses the problem that token-level masking makes prediction of multi-word expressions trivially easy due to local co-occurrence statistics.

The masking procedure operates as follows:

Given an input sequence of tokens $X = (x_1, x_2, \ldots, x_n)$ , the system iteratively samples spans to mask until the total number of masked tokens reaches 15% of $n$ (the same masking budget as BERT). At each iteration:

Sample a span length $\ell$ from a geometric distribution: $\ell \sim \text{Geo}(p)$ , with $p = 0.2$ . A geometric distribution is a discrete probability distribution over the positive integers where the probability of length $k$ is $P(\ell = k) = (1-p)^{k-1} \cdot p$ . This distribution is memoryless (each additional word has probability $(1-p)$ of being added) and is strongly skewed toward shorter spans: the mode is always $\ell = 1$ , and the probability decays exponentially with length.
Clip the sampled length at $\ell_{\text{max}} = 10$ words. Any sampled length exceeding 10 is truncated to 10. This prevents the masking of extremely long spans that would remove too much context and make the prediction task impossible rather than challenging.
Select a random starting position uniformly from all positions in the sequence that correspond to the beginning of a word. The constraint that the starting point must be a word boundary ensures that the span always consists of complete words—the system never masks partial words (e.g., masking "foot" from "football" while leaving "ball" visible). This is critical because subword masking within a word would reintroduce the very local co-occurrence shortcut that span masking aims to eliminate. The implementation samples "a sequence of complete words (instead of subword tokens)," as the paper states.
Mask all tokens within the selected span. Unlike BERT's per-token independent replacement decision, SpanBERT applies the replacement rule at the span level: either all tokens in the span are replaced with [MASK] (80% probability), all are replaced with random tokens (10% probability), or all are kept unchanged (10% probability). The paper states: "we perform this replacement at the span level and not for each token individually; i.e. all the tokens in a span are replaced with [MASK] or sampled tokens."
Continue iterating until the cumulative number of masked tokens reaches 15% of the sequence length.

Why a geometric distribution with $p = 0.2$ ?

The paper reports preliminary experiments with $p \in \{0.1, 0.2, 0.4\}$ and found $p = 0.2$ to perform best. We can interpret the distribution's shape:

With $p = 0.2$ , the probability of a span of length 1 is $0.2$ , length 2 is $0.8 \times 0.2 = 0.16$ , length 3 is $0.8^2 \times 0.2 = 0.128$ , etc. The mean of the unclipped geometric distribution with $p = 0.2$ is $E[\ell] = 1/p = 5$ , but after clipping at $\ell_{\text{max}} = 10$ , the empirical mean is $\text{mean}(\ell) = 3.8$ as reported in the paper.
Figure 2 visualizes this distribution: span lengths of 1-3 words are most common, with probability tapering off smoothly to zero at length 10. The distribution has no hard cutoff for "short" vs. "long" spans—it naturally samples a mix, dominated by short spans but occasionally including longer ones of 5-10 words.
A lower $p$ (e.g., $p = 0.1$ ) would produce a flatter distribution with longer mean span length, making the pre-training task harder (more tokens must be predicted from outside the span) but potentially removing too much information. A higher $p$ (e.g., $p = 0.4$ ) would produce mostly single-word spans, making the scheme closer to BERT's original token-level masking and losing the span-prediction benefit. The choice $p = 0.2$ balances difficulty with feasibility—long enough spans to force compositional prediction from context, but not so long that the task becomes hopeless.

Why mask at the whole-word level?

The paper's ablation study (Table 6) compares five masking granularities:

Subword Tokens: BERT's original approach—mask individual WordPiece tokens uniformly at random. A word like "Broncos" might be a single token, while "playing" might be split into "play" + "##ing", with each subword masked independently.
Whole Words: Sample complete words at random, then mask all subword tokens within each selected word. The total number of masked subword tokens still sums to ~15% of the sequence. This prevents the trivial prediction of a subword given its adjacent subword from the same word.
Named Entities: 50% of the time, mask an entire named entity span (identified by spaCy's NER); 50% of the time, mask a random whole word. This tests whether linguistically meaningful spans are better training targets.
Noun Phrases: 50% noun phrase spans (from spaCy's constituency parser), 50% random whole words. Similar motivation to named entity masking.
Geometric Spans (SpanBERT): Mask random spans from a geometric distribution, always at whole-word boundaries.

The key empirical finding (Table 6): geometric spans perform best or near-best across tasks, while named entity and noun phrase masking are inconsistent—noun phrase masking matches geometric spans on NewsQA (73.0 F1) but underperforms by 1.1 F1 on TriviaQA (77.7 vs. 78.8). The paper's conclusion: linguistically-informed spans are not reliably better than random spans, and the geometric distribution over random spans is a simpler, more general-purpose approach that does not require external NLP tools.

Why does random span masking work better than linguistic masking?

The paper does not definitively answer this, but several hypotheses are consistent with the results:

Linguistic parsers and NER systems make errors, especially on the noisy, domain-diverse text that composes pre-training corpora. Masking incorrect spans would provide misleading training signals (e.g., masking a partial entity and leaving the rest visible).
Linguistic categories (named entities, noun phrases) represent only a subset of multi-word expressions that matter for downstream tasks. Verb phrases ("won the game"), prepositional phrases ("in the fourth quarter"), and other non-NP/NE expressions also need span-level understanding.
The geometric distribution's variety—span lengths from 1 to 10, randomly positioned—exposes the model to a broader range of prediction difficulties than fixed linguistic categories, which tend to have characteristic length distributions (named entities are often short, noun phrases can be long but have predictable internal structure).
Random spans sometimes cross linguistic boundaries (e.g., masking "game to determine" straddling a noun phrase and an infinitive), which may force the model to learn more robust context integration since the span is not a coherent syntactic unit.

Connection to the MLM objective:

Span masking changes which tokens are predicted (now full spans must be reconstructed) but uses the same MLM loss function as BERT. The MLM loss for a masked token $x_i$ is the standard cross-entropy:

$\mathcal{L}_{\text{MLM}}(x_i) = -\log P(x_i \mid \mathbf{x}_i)$

where $\mathbf{x}_i$ is the Transformer encoder's output representation at position $i$ , and $P(x_i \mid \mathbf{x}_i)$ is the probability assigned to the true token by a softmax over the vocabulary, implemented via a linear projection of $\mathbf{x}_i$ using the tied input embedding matrix.

The critical difference from BERT is not the loss function but the context available for prediction. When BERT masks individual tokens independently, a masked token inside "Denver Broncos" has the other word (or subword) as context—the prediction can rely on $P(\text{"Broncos"} \mid \text{"Denver"}, \text{context})$ , which is a high-probability bigram. When SpanBERT masks the entire span, the model must predict "Denver" and "Broncos" from tokens outside the span entirely—the prediction becomes $P(\text{"Broncos"} \mid \text{context\_outside\_span})$ , which requires understanding that the surrounding text (e.g., "won Super Bowl 50") implies an NFL team name.

This is the mechanism by which span masking "forces the model to predict entire spans solely using the context in which they appear," as the paper puts it. The prediction task becomes fundamentally harder and requires genuine semantic understanding of the relationship between the masked span and its surrounding discourse.

Span Boundary Objective (SBO)

The span boundary objective is SpanBERT's second and more novel innovation. While span masking determines what the model must predict, the SBO determines where the model stores the information needed for that prediction. Specifically, it trains the representations of the tokens immediately outside a masked span— $\mathbf{x}_{s-1}$ (the token before the span) and $\mathbf{x}_{e+1}$ (the token after the span)—to encode sufficient information to reconstruct every token inside the span.

Motivation from downstream span selection models:

Downstream tasks like extractive question answering and coreference resolution typically construct a fixed-length representation of a candidate text span by concatenating (or otherwise combining) the output representations of the span's boundary tokens. For example, in a QA model, the representation of a candidate answer from position $s$ to $e$ might be $[\mathbf{x}_s; \mathbf{x}_e]$ or $[\mathbf{x}_{s-1}; \mathbf{x}_{e+1}]$ . The quality of this span representation depends entirely on whether $\mathbf{x}_s$ and $\mathbf{x}_e$ (or their neighbors) contain information about the span's internal content.

BERT's pre-training provides no explicit incentive for boundary tokens to summarize span content. The MLM objective trains each position to predict the token at that position from context, but there is nothing that pushes information about a whole phrase into the tokens at its edges. The SBO is designed to provide exactly this incentive: by forcing the model to predict the contents of a masked span using only the boundary tokens, the boundary representations must learn to encode everything that would be needed to reconstruct the interior.

The paper states this motivation clearly:

"we would ideally like the representations for the end of the span to summarize as much of the internal span content as possible"

Formal definition of the SBO:

Given a masked span $(x_s, \ldots, x_e)$ that belongs to the set of masked tokens $Y$ , let $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ be the Transformer encoder's output representations for the tokens immediately outside the span (the left boundary token and right boundary token, respectively). The system constructs a representation $\mathbf{y}_i$ for each token $x_i$ inside the span ( $s \leq i \leq e$ ) as follows:

$\mathbf{y}_i = f(\mathbf{x}_{s-1}, \mathbf{x}_{e+1}, \mathbf{p}_{i-s+1})$

where:

$\mathbf{x}_{s-1} \in \mathbb{R}^d$ is the $d$ -dimensional output vector from the Transformer encoder for the token immediately preceding the span,
$\mathbf{x}_{e+1} \in \mathbb{R}^d$ is the $d$ -dimensional output vector for the token immediately following the span,
$\mathbf{p}_{i-s+1} \in \mathbb{R}^{200}$ is a learned position embedding that encodes the relative offset of the target token from the left boundary. The index $i-s+1$ starts at 1 for the first token inside the span, 2 for the second, and so on. The dimensionality of 200 for these position embeddings is a design choice (the paper states "we use 200 dimension position embeddings $\mathbf{p}_1, \mathbf{p}_2, \ldots$ to mark positions relative to the left boundary token"),
$f(\cdot)$ is a learned function that combines these three pieces of information into a single vector.

The architecture of $f(\cdot)$ :

The function $f$ is implemented as a 2-layer feed-forward network with GeLU activations and layer normalization, a standard architecture for post-encoder representation refinement. The computation proceeds in three steps:

$\mathbf{h}_0 = [\mathbf{x}_{s-1}; \mathbf{x}_{e+1}; \mathbf{p}_{i-s+1}]$

$\mathbf{h}_1 = \text{LayerNorm}(\text{GeLU}(\mathbf{W}_1 \mathbf{h}_0))$

$\mathbf{y}_i = \text{LayerNorm}(\text{GeLU}(\mathbf{W}_2 \mathbf{h}_1))$

where:

$[\cdot; \cdot; \cdot]$ denotes concatenation, producing a vector of dimension $2d + 200$ (twice the Transformer hidden size plus the 200-dimensional position embedding),
$\mathbf{W}_1$ and $\mathbf{W}_2$ are learned weight matrices of appropriate dimensions (the exact hidden dimension of the intermediate representation $\mathbf{h}_1$ is not explicitly stated in the paper, but it is the standard feed-forward hidden dimension for BERTlarge, implicitly 4× the input dimension or similar),
GeLU (Gaussian Error Linear Unit) is a smooth, non-monotonic activation function that has been shown to outperform ReLU in Transformer architectures, defined as $\text{GeLU}(x) = x \cdot \Phi(x)$ where $\Phi$ is the cumulative distribution function of the standard normal distribution,
LayerNorm applies mean-variance normalization across the feature dimension, stabilizing training by reducing internal covariate shift.

The use of two layers with GeLU and layer normalization mirrors the feed-forward sublayers inside the Transformer encoder itself, providing the SBO module with sufficient representational capacity to learn a mapping from boundary context + relative position → interior token prediction.

Why include the position embedding?

The position embedding is essential because the boundary tokens $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ are the same for every token in the span—without positional information, the network would produce the same prediction for every position inside the span. The position embedding $\mathbf{p}_{i-s+1}$ disambiguates which word in the span is being predicted: $\mathbf{p}_1$ says "I am predicting the first word after the left boundary," $\mathbf{p}_2$ says "second word," etc.

The position embeddings are learned parameters, not fixed sinusoidal encodings. They are indexed by the 1-based offset from the left boundary, so $\mathbf{p}_1$ is the embedding for "first word in the span," $\mathbf{p}_2$ for "second word," and so on up to the maximum span length of 10 (since spans are capped at $\ell_{\text{max}} = 10$ ). Dimensionality of 200 is a hyperparameter choice—large enough to encode fine-grained positional distinctions, small enough to not dominate the concatenated representation.

Using $\mathbf{y}_i$ for prediction:

Once the SBO produces the representation $\mathbf{y}_i$ , the system predicts token $x_i$ from this representation using the same vocabulary projection as MLM:

$\mathcal{L}_{\text{SBO}}(x_i) = -\log P(x_i \mid \mathbf{y}_i)$

Here, $P(x_i \mid \mathbf{y}_i)$ is computed by applying a linear transformation (specifically, multiplication by the tied input embedding matrix—the same weight matrix used for the MLM output projection) to $\mathbf{y}_i$ and then a softmax over the vocabulary. The paper explicitly notes:

"reusing the input embedding (Press and Wolf, 2017) for the target tokens in both MLM and SBO"

Weight tying between input embeddings and output projections is a standard technique in language modeling that reduces parameters and provides a regularizing effect: the model must use the same representation space for encoding tokens as input and decoding them as output.

The combined loss for a masked token:

For each token $x_i$ inside a masked span, SpanBERT computes the sum of the MLM loss and the SBO loss:

$\mathcal{L}(x_i) = \mathcal{L}_{\text{MLM}}(x_i) + \mathcal{L}_{\text{SBO}}(x_i)$

$\mathcal{L}(x_i) = -\log P(x_i \mid \mathbf{x}_i) - \log P(x_i \mid \mathbf{y}_i)$

where:

The first term $-\log P(x_i \mid \mathbf{x}_i)$ is the standard MLM loss: using the Transformer's output at the masked position $i$ itself to predict what token was there. This term operates on $\mathbf{x}_i$ , the representation of the [MASK] token (or random/replacement token) at position $i$ .
The second term $-\log P(x_i \mid \mathbf{y}_i)$ is the SBO loss: using the boundary-conditioned representation $\mathbf{y}_i$ (derived from $\mathbf{x}_{s-1}$ , $\mathbf{x}_{e+1}$ , and $\mathbf{p}_{i-s+1}$ ) to predict the same token.

Both terms are standard cross-entropy losses against the same ground-truth token $x_i$ . The sum means that gradients from both losses flow back through the Transformer encoder, providing complementary learning signals:

The MLM loss gradient trains the encoder to extract information from the surrounding context into $\mathbf{x}_i$ —the representation at the masked position itself.
The SBO loss gradient trains the encoder to store information about the span's content in $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ —the boundary token representations—because these are the only sources of information that $\mathbf{y}_i$ can use (the position embedding carries no lexical content).

What does the SBO actually learn?

The key insight is that the SBO forces information about the entire span to be encoded into the boundary token representations. Consider the example from Figure 1: the span "an American football game" is masked, and the model must predict each token in it. The left boundary is $\mathbf{x}_4$ (the representation of "was"—the word before "an"), and the right boundary is $\mathbf{x}_9$ (the representation of "to"—the word after "game").

To predict "football" (the third token in the span), the SBO module receives $\mathbf{x}_4$ , $\mathbf{x}_9$ , and $\mathbf{p}_3$ . There is no direct path from the tokens inside the span to the prediction—those tokens are masked. The only way for the model to succeed is if $\mathbf{x}_4$ and $\mathbf{x}_9$ have been encoded by the Transformer to contain sufficient information about "football" (and, more precisely, about the fact that "football" is the third word after "was" in this context). This information must come from the surrounding context: the Transformer's self-attention layers must aggregate evidence from other tokens in the sequence (e.g., "Super Bowl", "champion", "game") into the boundary positions.

Over many training examples, this creates a consistent gradient signal: boundary token representations should be good summaries of the spans they delimit. The SBO is thus a form of representation-level supervision—it does not change the architecture at fine-tuning time (the SBO network is discarded) but shapes the representations that the architecture produces so that they are maximally useful for span-oriented downstream tasks.

Why sum the MLM and SBO losses rather than using SBO alone?

The paper keeps both losses, summing them rather than replacing MLM with SBO. This is important for two reasons:

The MLM loss is still valuable for general language understanding. It provides a rich, token-level training signal that forces the encoder to build good contextualized representations at every position, not just span boundaries. Removing MLM entirely would eliminate this broad training signal.
The MLM loss for tokens inside a masked span operates on $\mathbf{x}_i$ —the representation of the masked token itself. During pre-training, this position contains the [MASK] embedding (or a random token), so the gradient flow trains the Transformer to extract information from the unmasked context into this position. During fine-tuning, there are no [MASK] tokens, but the representation at every position still benefits from the general feature extraction capability learned during MLM pre-training.

The two losses are complementary: MLM trains all positions to be good context aggregators; SBO specifically trains boundary positions to be good span summarizers. Keeping both ensures that the boundary representations both (a) benefit from general-purpose contextualization (via MLM gradients) and (b) are specifically optimized for span-content encoding (via SBO gradients).

Why not use the NSP objective instead of SBO?

The paper replaces BERT's next-sentence prediction (NSP) objective with the SBO, based on the finding that NSP actually hurts performance on most downstream tasks when combined with bi-sequence sampling (Section 5, Tables 1-5). Table 7 directly compares the two auxiliary objectives when combined with span masking:

Span masking (1 seq) without any auxiliary objective achieves a baseline on SQuAD 2.0 of 86.7 F1.
Adding SBO raises this to 86.8 F1—a small gain on SQuAD but a substantial +2.7 F1 on coreference resolution (76.3 → 79.0).

The SBO is strictly additive: "Unlike the NSP objective, SBO does not appear to have any adverse effects." This contrasts with NSP, which the paper found to be harmful in the bi-sequence setup because it forces the model to condition on potentially unrelated context from another document, adding noise to the MLM signal and reducing the effective context window per sequence.

The importance of using both boundaries:

The SBO concatenates both the left boundary ( $\mathbf{x}_{s-1}$ ) and the right boundary ( $\mathbf{x}_{e+1}$ ) representations, giving the prediction module access to context from both sides of the span. This is consistent with how bidirectional Transformer encoders work—the representation at any position already contains information from both directions due to the non-masked self-attention mechanism—but explicitly including both boundaries ensures that the SBO module receives symmetric context.

For spans at the beginning of a sequence (no left boundary), or at the end (no right boundary), the paper does not explicitly describe a fallback mechanism. A standard implementation would use a special token (e.g., [CLS] for the left boundary when $s = 1$ , or [SEP] for the right boundary when $e = n$ ) or mask only spans that have both boundaries—this is a minor implementation detail not discussed in the paper.

Computational cost of the SBO:

The SBO module is computationally lightweight compared to the Transformer encoder. It consists of only two feed-forward layers operating on a single vector per masked token, whereas the Transformer encoder has 24 layers (for BERTlarge) operating over the full sequence. During pre-training, the SBO adds a small constant overhead per masked span. During fine-tuning, the SBO is discarded entirely—it incurs zero cost at inference time.

This is an elegant design: the SBO acts as a pre-training scaffold that shapes representations during training but does not constrain the model architecture at deployment. Downstream models receive exactly the same architecture as a standard BERT (the Transformer encoder producing per-position vectors), but the vectors they receive at boundary positions are better tuned for span-based reasoning.

Single-Sequence Training

The third component of SpanBERT's pre-training recipe is a change to the data pipeline: rather than sampling two text segments and using the next-sentence prediction (NSP) objective as BERT does, SpanBERT samples a single contiguous block of up to 512 tokens from the corpus and trains only on the MLM + SBO objectives.

The BERT baseline (for contrast):

To understand why single-sequence training matters, it is essential to understand what it replaces. BERT's pre-training data pipeline (Devlin et al., 2019) constructs each training example as follows:

Sample a "segment A" consisting of a contiguous block of text from the corpus.
With 50% probability, sample "segment B" as the actual continuation of segment A in the corpus. With 50% probability, sample segment B as a random segment from a different document.
Truncate segments A and B such that their combined length (including special tokens [CLS] and [SEP]) is at most 512 tokens.
Apply token-level MLM independently to the combined sequence [CLS] A [SEP] B [SEP].
Train the [CLS] token representation to predict whether B follows A (the NSP objective) in addition to training MLM on all masked positions.

This design means that BERT's pre-training is done on pairs of half-length sequences: each segment is at most ~256 tokens long (since they share the 512-token budget). The model never sees a single contiguous sequence longer than ~256 tokens during pre-training.

SpanBERT's single-sequence pipeline:

SpanBERT abandons this two-segment design entirely:

Divide the pre-training corpus into contiguous blocks of up to 512 tokens, respecting document boundaries (a block never crosses a document boundary; if a document has fewer than 512 remaining tokens, the block is shorter).
At each training step, sample a batch of these blocks uniformly at random.
Apply the span masking scheme (Section 3.1) to each block: mask 15% of tokens using geometrically-distributed random spans.
Compute the combined MLM + SBO loss for all masked tokens.
No [SEP] tokens separating two segments (except possibly at the end of the sequence if the block ends at a document boundary), and no NSP objective.

The resulting sequences are on average 390 tokens long, as noted in the paper: "On the average, this is approximately 390 sequences since some documents have fewer than 512 tokens."

Why single-sequence training outperforms bi-sequence training with NSP:

The paper finds that single-sequence training "considerably improves performance on most downstream tasks" (Section 3.3) and provides two hypotheses for why:

Longer full-length contexts: "the model benefits from longer full-length contexts." In bi-sequence training, the model never processes a single document longer than ~256 tokens, which limits its ability to learn long-range dependencies. Single-sequence training exposes the model to contexts up to 512 tokens from a single document, doubling the effective context window for within-document reasoning.
Reduced noise from unrelated context: "conditioning on, often unrelated, context from another document adds noise to the masked language model." When BERT's segment B is randomly sampled from a different document (50% of training examples), the two halves of the input are semantically unrelated. The model must simultaneously: (a) use context from segment A to predict masked tokens in segment A, (b) use context from segment B to predict masked tokens in segment B (the self-attention mechanism allows cross-segment attention, but the information from the other segment is noise), and (c) determine whether B follows A. The presence of the unrelated segment acts as a distractor for the MLM task, making it harder for the model to learn coherent within-document contextualization.

This finding resolves an apparent contradiction with BERT's original ablation study, which found NSP to be beneficial. The paper notes:

"This is surprising because BERT's ablations showed gains from the NSP objective (Devlin et al., 2019). However, the ablation studies still involved bi-sequence data processing, i.e. the pre-training stage only controlled for the NSP objective while still sampling two half-length sequences."

BERT's ablation compared (a) bi-sequence processing with NSP vs. (b) bi-sequence processing without NSP, finding that NSP helped. But neither condition in that ablation used full-length single sequences. SpanBERT's comparison is different: it compares (a) bi-sequence with NSP vs. (b) single-sequence without NSP. The confound is that both the sequence length and the NSP objective differ between the two conditions. SpanBERT's results suggest that the harm from reduced context length in bi-sequence training outweighs any benefit from NSP, and that simply removing both constraints yields the best performance.

Empirical evidence for single-sequence benefits:

The magnitude of gains from single-sequence training is documented throughout the results:

In Table 1 (SQuAD): Our BERT (bi-sequence, with NSP): 92.6 F1 on SQuAD 1.1; Our BERT-1seq (single-sequence, no NSP): 93.3 F1. Gain: +0.7 F1.
In Table 2 (MRQA average): Our BERT: 78.6 F1; Our BERT-1seq: 79.7 F1. Gain: +1.1 F1.
In Table 5 (GLUE average): Our BERT: 81.1; Our BERT-1seq: 81.7. Gain: +0.6, with particularly large gains on CoLA (+4.9, from 58.6 to 63.5).
In Table 7 (ablation, with span masking): Span Masking (2seq) + NSP reaches 83.4 GLUE average; Span Masking (1seq) reaches 83.8. Gain: +0.4.

These gains are modest per-task but consistent across the board (14 of 17 tasks benefit). Critically, the gains are orthogonal to those from span masking and SBO: in Table 7, single-sequence training improves performance over bi-sequence training with the same span masking scheme, and adding SBO on top of single-sequence training provides further improvement. The three components—single sequences, span masking, and SBO—are additive improvements that compound.

Interaction with span masking:

Single-sequence training and span masking interact synergistically. In bi-sequence training, a randomly sampled span might cross the boundary between segment A and segment B (though the paper does not explicitly state whether boundaries were respected for span sampling). More importantly, the shorter effective context per segment limits how much surrounding context is available for predicting a masked span. A 512-token single sequence provides twice the local context for predicting a masked span compared to a 256-token segment, making the span prediction task both more feasible (more context to draw on) and more informative (the model learns to integrate information over longer distances).

Implementation details of the data pipeline:

The paper's data pipeline includes several modifications beyond single-sequence training that differ from the original BERT:

Dynamic masking (different masks at each epoch): "We use different masks at each epoch while BERT samples 10 different masks for each sequence during data processing." BERT's original implementation pre-generated 10 mask variants per sequence during a preprocessing step and cycled through them during training. SpanBERT generates masks on-the-fly at each training step, meaning the model effectively sees an infinite variety of masks rather than being limited to 10 fixed patterns per sequence. This provides more diversity in the pre-training signal and avoids the model memorizing specific mask patterns.
No short-sequence pre-training phase: "We remove all the short-sequence strategies used before (they sampled shorter sequences with a small probability 0.1; they also first pre-trained with smaller sequence length of 128 for 90% of the steps)." BERT's original recipe included an initial 90% of training steps with sequence length 128 (shorter sequences, faster training) followed by 10% with sequence length 512. SpanBERT always trains with sequences up to 512 tokens from the start. This is consistent with the concurrent RoBERTa findings (Liu et al., 2019b) that the short-sequence phase is unnecessary and may slow convergence.
Documents as sequence boundaries: Sequences are "up to 512 tokens until it reaches a document boundary," meaning that if a document is shorter than 512 tokens, the entire document is used as one sequence (potentially much shorter than 512), and the next sequence starts fresh from the next document. This preserves document coherence—the model never sees sequences that artificially concatenate the end of one document and the beginning of another.
Batch size of 256 sequences: The paper reports a batch size of 256 sequences with a maximum of 512 tokens, yielding approximately 390 sequences on average per batch due to documents shorter than 512 tokens. The total number of tokens per batch is thus approximately $256 \times 390 = 99,840$ tokens.

These implementation choices are not the main contribution but contribute to the "well-tuned replica of BERT" that forms the baseline. The paper is transparent that these choices are shared across all models (Our BERT, Our BERT-1seq, and SpanBERT), so the gains attributed to span masking and SBO are over and above what these improved training practices provide.

Why the NSP objective was not simply replaced with SBO while keeping bi-sequence training:

The paper could have kept the two-segment data pipeline and replaced NSP with SBO, but this would have introduced complications: the SBO requires spans with both left and right boundaries, and segments within a two-segment input have artificial boundaries at the [SEP] token. Single-sequence training simplifies the input format and makes the SBO more natural—every masked span within a single contiguous text has genuine boundary tokens on both sides (except at document edges). The decision to drop NSP and bi-sequence training was thus both empirically motivated (better performance) and architecturally cleaner for the span-based approach.

Interaction Between the Three Components

The three components of SpanBERT—span masking, SBO, and single-sequence training—are not independent tweaks but form a coherent pre-training strategy where each component addresses a different aspect of learning span representations. Their interactions are multiplicative rather than merely additive.

Why span masking alone is insufficient:

Span masking without SBO forces the model to predict entire spans from context (a harder task than token-level MLM), but it does not specify where the model should store the learned span information. The MLM predictions for tokens inside a masked span operate on the [MASK] embeddings at those positions—the model can solve the task by extracting information from the unmasked context into those masked positions during the forward pass, without necessarily encoding span-level information into the boundary tokens.

Consider the example "Denver Broncos won the Super Bowl," where the span "Denver Broncos" is masked. With span masking alone, the model must predict "Denver" at position 1 and "Broncos" at position 2 given the context "won the Super Bowl" at positions 3-5. The self-attention mechanism allows position 1 to attend to positions 3-5, extracting relevant information ("won the Super Bowl" → "who won? → "Denver Broncos") directly into $\mathbf{x}_1$ . The boundary token at position 6 (say, a period) does not receive a specific training signal to encode the span's content. At fine-tuning time, if the downstream model uses $\mathbf{x}_0$ (the [CLS] token) and $\mathbf{x}_6$ to represent the span "Denver Broncos," those representations may not contain the span-level information—it was stored at $\mathbf{x}_1$ and $\mathbf{x}_2$ during pre-training.

Why SBO fixes this:

The SBO explicitly trains $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ to encode the span's content. In the example above, the SBO adds a prediction of "Denver" and "Broncos" from $\mathbf{x}_0$ (before Denver) and $\mathbf{x}_3$ (after Broncos). Now those boundary positions must aggregate information about the masked span from the context, because the SBO module only receives those boundary representations (plus position embeddings). The gradient from the SBO loss flows back through the Transformer encoder to those boundary positions, training them to serve as span summaries.

Table 7 quantifies this interaction on coreference resolution, which is the task most dependent on high-quality span representations:

Span Masking (1seq) without SBO: 76.3 F1
Span Masking (1seq) with SBO: 79.0 F1
Gain from adding SBO: +2.7 F1

This gain is on top of the improvement from span masking over BERT's token masking (which itself raised coreference from 78.3 to ~78.8, as shown in Table 3). The SBO adds a large, targeted benefit specifically for the task that most directly uses boundary-based span representations.

Why single-sequence training amplifies the other components:

Single-sequence training provides longer contexts for the model to draw on when predicting masked spans. In a bi-sequence setup with ~256 tokens per segment, the model often lacks sufficient surrounding context to predict a long span—the informative context might be in the other (potentially unrelated) segment, or might extend beyond the 256-token window. With 512 tokens of contiguous text, the model has roughly twice the amount of local context for any given masked span.

This interacts with span masking: longer contexts make the span prediction task more feasible for longer spans, since there is more surrounding text to provide disambiguating information. It also interacts with SBO: the boundary tokens can integrate information from a wider context window (up to 512 tokens) to build their span summaries, making those summaries more accurate.

Table 7 shows the cumulative effect across multiple tasks (reported as GLUE average, which captures a range of sentence-level and pair-level tasks):

Baseline (Span Masking, 2seq, +NSP): 83.4
- Single-sequence (Span Masking, 1seq): 83.8
- SBO (Span Masking, 1seq, +SBO): 84.0

The gain from each component is modest individually but the stacking produces consistent improvements. On span-focused tasks (SQuAD 2.0, coreference), the gains are larger and more clearly attributable to specific components.

The full pre-training procedure (as described in Appendix A):

The complete SpanBERT pre-training procedure, combining all three components, is:

Divide the entire pre-training corpus (BooksCorpus + English Wikipedia, using cased WordPiece tokens, matching BERT's corpus exactly) into single contiguous blocks of up to 512 tokens each.
For each training step, sample a batch of 256 blocks uniformly at random. The average sequence length in the batch is approximately 390 tokens.
For each block in the batch, apply the span masking scheme: iteratively sample spans from $\text{Geo}(p=0.2)$ , clipped at $\ell_{\text{max}} = 10$ , masking complete words, until 15% of WordPiece tokens in the block are masked. Within each masked span, apply the replacement rule at the span level: 80% probability all tokens are [MASK], 10% all random, 10% all unchanged.
Pass the masked blocks through the Transformer encoder (BERTlarge architecture: 24 layers, 1024 hidden dimension, 16 attention heads, ~340M parameters) to obtain contextualized representations $\mathbf{x}_1, \ldots, \mathbf{x}_n$ .
For each masked token $x_i$ within a masked span $(x_s, \ldots, x_e)$ :
- Compute the MLM prediction from $\mathbf{x}_i$ and the cross-entropy loss $\mathcal{L}_{\text{MLM}}(x_i)$ .
- Construct the SBO representation $\mathbf{y}_i = f(\mathbf{x}_{s-1}, \mathbf{x}_{e+1}, \mathbf{p}_{i-s+1})$ using the 2-layer GeLU network with layer normalization.
- Compute the SBO prediction from $\mathbf{y}_i$ and the cross-entropy loss $\mathcal{L}_{\text{SBO}}(x_i)$ .
- Sum the losses: $\mathcal{L}(x_i) = \mathcal{L}_{\text{MLM}}(x_i) + \mathcal{L}_{\text{SBO}}(x_i)$ .
For masked tokens that are not inside a span (e.g., if a span of length 1 was sampled and falls at a position that happens to be the only masked token in that vicinity), compute only the MLM loss. The paper's masking scheme uses span-level sampling, but single-token spans are still possible (when $\ell = 1$ is sampled) and would have SBO applied since they are still within a span with defined boundaries.
Sum all per-token losses across the batch and backpropagate.

Optimization hyperparameters (pre-training):

The paper provides the following pre-training hyperparameters:

Optimizer: AdamW (Adam with decoupled weight decay, as proposed by Loshchilov and Hutter, 2019)
Learning rate: Peaked at $1 \times 10^{-4}$ , with linear warmup over the first 10,000 steps and linear decay thereafter
Beta parameters: $\beta_1 = 0.9$ , $\beta_2 = 0.999$ (standard Adam defaults)
Weight decay: 0.1 (decoupled from the Adam update, applied directly to weights)
Epsilon: $1 \times 10^{-8}$ for AdamW (a deviation from the more common $1 \times 10^{-6}$ , which the paper notes "converges to a better set of model parameters")
Dropout: 0.1 on all layers and attention weights
Activation: GeLU (Gaussian Error Linear Unit)
Training steps: 2.4M steps (longer than the original BERT's 1M steps)
Batch size: 256 sequences, with a maximum of 512 tokens per sequence (approximately 99,840 tokens per batch on average)
Hardware: 32 Volta V100 GPUs, training for 15 days

Fine-tuning (for downstream tasks):

At fine-tuning time, the SBO network ( $f(\cdot)$ , the 2-layer feed-forward network) is discarded. The pre-trained Transformer encoder weights are loaded, and task-specific heads are added on top. The fine-tuning procedure follows standard BERT practices, with task-specific hyperparameters:

Extractive QA: Max sequence length 512, sliding window of 128 for longer inputs. Learning rates from $\{5 \times 10^{-6}, 1 \times 10^{-5}, 2 \times 10^{-5}, 3 \times 10^{-5}, 5 \times 10^{-5}\}$ , batch sizes from $\{16, 32\}$ , 4 epochs for all datasets. The architecture uses separate linear classifiers for start and end positions, operating on the per-token Transformer outputs.
Coreference Resolution: Documents divided into non-overlapping chunks of length chosen from $\{128, 256, 384, 512\}$ . Each chunk is encoded independently by the pre-trained Transformer. BERT learning rates from $\{1 \times 10^{-5}, 2 \times 10^{-5}\}$ , task-specific learning rates from $\{1 \times 10^{-4}, 2 \times 10^{-4}, 3 \times 10^{-4}\}$ , batch size 1 (one document), 20 epochs. The span pair scoring function uses span representations constructed from the boundary token outputs of the Transformer encoder.
TACRED/GLUE: Max sequence length 128. Learning rates from $\{5 \times 10^{-6}, 1 \times 10^{-5}, 2 \times 10^{-5}, 3 \times 10^{-5}, 5 \times 10^{-5}\}$ , batch sizes from $\{16, 32\}$ , 10 epochs (with the exception of CoLA, which uses 4 epochs to avoid severe overfitting). A linear classifier is added on top of the [CLS] token for classification tasks.

Why this design over alternatives:

The combined design (single-sequence + span masking + SBO) represents a coherent philosophy: pre-training should match the structure of downstream tasks. Span selection tasks use boundary tokens to represent spans → SBO trains boundary tokens to encode span content. Span selection tasks require reasoning about multi-word entities and phrases → span masking forces the model to predict entire spans from context. Real-world text is organized as coherent documents with long-range dependencies → single-sequence training preserves document coherence and provides the full 512-token context window.

Each alternative considered by the paper has specific drawbacks:

Linguistic span masking (named entities, noun phrases): inconsistent performance, requires external NLP tools, tied to specific linguistic categories that may not cover all relevant multi-word expressions.
NSP + bi-sequence training: reduces effective context length, introduces noise from unrelated segment pairs.
Token-level MLM without SBO: makes span prediction too easy when adjacent tokens are visible, provides no incentive for boundary tokens to encode span content.

The paper's approach is thus a carefully reasoned set of design choices, each motivated by a specific limitation of BERT for span-oriented tasks, and supported by ablation experiments that isolate the contribution of each component.

4. Key Insights and Innovations

Innovation 1: Pre-training objectives should match the representational geometry of downstream tasks, not just the token-level statistics

The dominant paradigm in pre-training circa 2019 was to design self-supervised objectives that teach a model to predict held-out tokens from context—the masked language model (MLM) of BERT, the autoregressive prediction of GPT, or permutation-based modeling of XLNet. These objectives all operate at the token level: the model learns to produce good contextualized representations of individual word pieces, and the implicit assumption is that higher-level linguistic structures (phrases, entities, spans) will naturally emerge from token-level training as epiphenomena. The field's attention was focused on how to mask (random tokens vs. linguistically-informed spans in ERNIE; autoregressive orderings in XLNet) and how much data to use, with the presumption that token-level objectives were sufficient as long as the data and model were large enough.

SpanBERT makes explicit a different assumption: token-level objectives are insufficient because they do not train the specific representational geometry that downstream span selection models actually use. Extractive QA models and coreference systems do not access arbitrary per-token embeddings—they construct span representations by concatenating the output vectors at the span's boundary tokens (start and end). This is an architectural convention, a fixed interface between the encoder and the task head. If the pre-training objective never provides a gradient signal that pushes span-level information toward these specific interface points, then the encoder must learn this routing incidentally—or fail to learn it at all, leaving performance on the table regardless of model scale.

This reframing—from "what tokens should we mask?" to "where should the learned information be stored?"—is the paper's deepest conceptual contribution. The span boundary objective (SBO) is not merely another auxiliary loss; it is a mechanism for aligning the pre-training gradient field with the geometry of downstream task architectures. The SBO loss flows gradient to exactly the two boundary token positions ( $\mathbf{x}_{s-1}$ and $\mathbf{x}_{e+1}$ ) that downstream models use to represent spans, explicitly training them to be lossy compressors of the span's internal content. At fine-tuning time, the SBO module is discarded, but the encoder's boundary representations have been shaped to contain span-relevant information—the pre-training scaffold has permanently altered the representational landscape.

The evidence that this matters beyond what token-level MLM can achieve comes from the coreference resolution results in Table 7. Single-sequence training with span masking but without SBO achieves 76.3 F1 on the OntoNotes benchmark. Adding SBO—which adds no new information sources, only a new gradient path to the boundary tokens—raises this to 79.0 F1, a gain of 2.7 points. This is a large improvement for a task where gains are typically hard-won, and it isolates the effect of the SBO's representational geometry: span masking alone forces the model to predict spans from context, but the SBO ensures the resulting knowledge is stored at the endpoints where the coreference model's span pair scorer accesses it.

This insight has implications beyond the specific SBO implementation. It suggests that pre-training objective design should be guided not only by what linguistic phenomena the model should learn, but also by the interface contracts of downstream architectures—the specific representational locations and formats that downstream models expect. For tasks that use [CLS] representations (sentence classification, NLI), pre-training objectives that train [CLS] (like NSP) are natural. For tasks that use boundary tokens, objectives that train boundary tokens are natural. This principle has not been widely articulated in the pre-training literature, making it a genuinely distinctive intellectual contribution.

Innovation 2: Random span masking is competitive with—and often superior to—linguistically-informed masking, despite being simpler

When this paper was written, a natural intuition in the field was that pre-training would benefit from masking linguistically coherent units: named entities, noun phrases, or other syntactic constituents. The ERNIE model (Sun et al., 2019) had shown improvements on Chinese NLP by masking named entities and phrases. The reasoning was straightforward: real-world text is organized into meaningful multi-word chunks, and pre-training should reflect this structure by treating those chunks as atomic masking units. This intuition was appealing enough that it was likely to become a standard practice—why mask arbitrarily when you can mask intelligently?

SpanBERT challenges this intuition with a controlled experiment (Table 6) that compares five masking schemes at the same pre-training budget, model architecture, and training data: subword tokens (BERT's original), whole words, named entities, noun phrases, and geometrically-distributed random spans. The results are not what the linguistic intuition would predict. Random geometric spans perform best overall, while linguistic spans (named entities, noun phrases) are inconsistent—sometimes competitive, sometimes clearly worse (noun phrase masking trails geometric spans by 1.1 F1 on TriviaQA). The finding is that randomness, when appropriately shaped (geometric distribution, whole-word granularity), provides a better training signal than linguistic coherence.

This is a non-obvious result with important implications for how the field thinks about pre-training task design. Why might random spans outperform linguistic spans? The paper does not fully answer this, but the result itself functions as a valuable corrective to the assumption that pre-training should mirror linguistic structure. Several potential explanations are consistent with the data:

Coverage: Linguistic parsers identify only a subset of multi-word expressions that matter for downstream tasks. Named entities and noun phrases exclude verb phrases ("won the game"), prepositional phrases ("in the fourth quarter"), and cross-category spans that straddle syntactic boundaries. Random spans sample from the full distribution of possible spans, providing more diverse training instances.
Parser error propagation: Pre-training corpora are noisy and stylistically diverse; NLP pipelines (NER, constituency parsing) make errors, especially on web text, dialogue, and informal language. Masking incorrect spans would provide misleading training signals, teaching the model to predict tokens from boundaries that do not actually delimit coherent units.
Difficulty calibration: The geometric distribution produces a natural mix of span lengths (mean 3.8, range 1–10) with smoothly varying difficulty. Linguistic spans have characteristic length distributions (named entities tend to be short, noun phrases variable) that may not provide optimal difficulty for learning.
Cross-boundary generalization: Random spans sometimes cross syntactic boundaries, forcing the model to predict tokens that form an incoherent sequence from a linguistic perspective. This may actually strengthen the model's ability to integrate context from both sides of an arbitrary boundary—a useful skill for real downstream tasks where candidate spans are not pre-filtered by syntactic coherence.

The practical upshot is significant: the simplest approach—random spans from a carefully parameterized distribution—is both simpler and more effective than linguistically-informed alternatives, eliminating the need for external NLP preprocessing during pre-training data generation. This finding parallels a broader pattern in deep learning where learned or randomly-structured approaches sometimes outperform hand-designed, linguistically-motivated ones. The paper's contribution here is not the geometric distribution per se (a standard statistical distribution) but the empirical demonstration, through careful ablation, that the field's intuition about linguistic span masking was wrong—or at least incomplete—and that this matters for downstream performance.

Innovation 3: The next-sentence prediction objective is harmful not because it's the wrong task, but because its associated data format corrupts the primary objective

BERT's NSP objective and its two-segment data pipeline were inherited as standard practice throughout the explosion of BERT-based models in 2018–2019. The original BERT paper's ablation study reported that removing NSP hurt performance, establishing it as an accepted component of the recipe. Most subsequent work either kept NSP or replaced it with a different auxiliary objective (e.g., sentence order prediction in ALBERT) while retaining the two-segment data format.

SpanBERT's analysis reveals a confound in BERT's original ablation that the field had not articulated. BERT's ablation compared (a) bi-sequence training with NSP vs. (b) bi-sequence training without NSP, finding NSP beneficial. But neither condition used full-length single sequences—both used the two-segment format where each segment gets at most half the context window (~256 tokens). SpanBERT disentangles the two factors: it compares bi-sequence training with NSP against single-sequence training without NSP, and finds the latter substantially better. The paper's diagnosis is that the NSP objective itself may not be the primary problem—it is that the bi-sequence data format forces the model to operate on shorter, potentially unrelated text segments, which degrades the quality of the MLM training signal. The paper states:

"We hypothesize that bi-sequence training, as it is implemented in BERT, impedes the model from learning longer-range features, and consequently hurts performance on many downstream tasks."

This is a cleaner, more actionable diagnosis than "NSP is harmful." It explains why BERT's ablation saw NSP as beneficial (when both conditions used the same data format, adding NSP added a small helpful signal) while SpanBERT finds the two-segment format itself to be the bottleneck (because it halves the effective context window and introduces unrelated cross-segment context as noise). The key implication is that auxiliary objectives should not be evaluated in isolation from their data pipeline—the format in which data is presented to the model can have effects that dominate the objective itself.

The empirical evidence for this reframing is distributed across the paper's results. Single-sequence training without NSP (Our BERT-1seq) outperforms bi-sequence training with NSP (Our BERT) on 14 of 17 benchmarks (Tables 1–5), with notable gains on tasks requiring long-range reasoning (CoLA: +4.9, from 58.6 to 63.5; SQuAD 2.0: +0.7 F1; MRQA average: +1.1 F1). These gains compound with SpanBERT's other innovations: in the ablation of Table 7, moving from bi-sequence + NSP to single-sequence with span masking adds 0.4 points on the GLUE average, and adding SBO further adds 0.2 points.

This insight is both a methodological contribution (don't confound objective with data format in ablations) and a practical one (the simplest path to improving BERT is to use longer, coherent sequences and drop the two-segment preprocessing entirely). It also harmonizes with concurrent findings from RoBERTa (Liu et al., 2019b), which independently found NSP to be unnecessary—SpanBERT provides a clearer mechanistic explanation for why by attributing the effect to the data format rather than the objective per se.

Innovation 4: Controlled baseline construction reveals that pre-training objective design provides gains orthogonal to—and competitive with—data and model scaling

The dominant narrative in NLP during 2018–2019 was that progress came primarily from scale: more data (GPT-2 with WebText, XLNet with an 8× data increase), larger models (BERTlarge with 340M parameters, GPT-2 with 1.5B), and more compute. The concurrent RoBERTa paper (Liu et al., 2019b) argued that many of BERT's design choices were suboptimal but that better hyperparameters and more data could recover substantial gains without architectural changes. SpanBERT takes a complementary but distinct position: better pre-training objectives can provide gains of similar magnitude to scaling, while using the same data and model size. The paper makes this point explicitly:

"While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pre-training tasks and objectives, which can also have a remarkable impact."

What distinguishes this paper's approach from standard "our method beats the baseline" claims is the rigor of the baseline construction. SpanBERT does not compare against Google's publicly released BERT checkpoints—which, as RoBERTa concurrently showed, were undertrained and suboptimally tuned. Instead, the authors build a carefully reimplemented BERT replica that already substantially outperforms the original (Tables 1–5, Our BERT vs. Google BERT): +1.3 F1 on SQuAD 1.1 (92.6 vs. 91.3), +2.6 F1 on SQuAD 2.0 (85.9 vs. 83.3), +1.3 average F1 on MRQA (78.6 vs. 77.3), +1.2 F1 on coreference (78.3 vs. 77.1). This baseline incorporates the same improved optimization practices (AdamW with epsilon $1\times10^{-8}$ , 2.4M training steps, dynamic masking, no short-sequence phase) that RoBERTa identified as important, plus the finding that single-sequence training outperforms bi-sequence training with NSP.

SpanBERT's gains are measured over this already-improved baseline. On SQuAD 1.1, the improvement chain is: Google BERT (91.3) → Our BERT (92.6, +1.3 from better optimization) → Our BERT-1seq (93.3, +0.7 from single-sequence training) → SpanBERT (94.6, +1.3 from span masking + SBO). This layered improvement structure—where each component adds on top of the previous best—demonstrates that the contributions from pre-training objective design (span masking, SBO) are genuinely additive to what better implementation and more data provide. The 27% error reduction on SQuAD 1.1 cited in the abstract is computed against Our BERT (not Google BERT), making it a conservative estimate of the method's contribution relative to a strong baseline.

This careful baseline methodology is important because it preempts the criticism that SpanBERT's gains are merely from better optimization practices that other work (RoBERTa) had already discovered. By incorporating those practices into the baseline and still showing consistent gains from span-based pre-training, the paper cleanly isolates the effect of its design innovations. The approach also implicitly argues for a methodological standard in pre-training research: new methods should be evaluated against the best available replication of the baseline, not just the original public release, to avoid conflating algorithm improvements with implementation improvements.

The key numbers that support this framing are the ablation in Table 7, where the effect of adding SBO to a single-sequence span-masked model is isolated: +2.7 F1 on coreference, +0.6 F1 on SQuAD 2.0, +0.2 F1 on GLUE average. These are gains from objective design alone—same data, same model size, same implementation quality—and they are non-trivial, particularly for coreference, where a 2.7 F1 gain is substantial. This provides evidence that pre-training objective innovation is a third axis of improvement, alongside data scaling and model scaling, that had been underexplored and that can yield gains competitive with what significantly larger models or datasets provide—at least for tasks where the objective's inductive bias matches the task structure.

5. Experimental Analysis

Evaluation Methodology

Datasets. SpanBERT is evaluated on 17 benchmarks spanning four task families. Extractive Question Answering: SQuAD 1.1 (Rajpurkar et al., 2016), SQuAD 2.0 (Rajpurkar et al., 2018), and five datasets from the MRQA shared task (Fisch et al., 2019)—NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019). Because the MRQA shared task lacks a public test set, the authors split the development set in half to create new development and test sets. Coreference Resolution: CoNLL-2012 shared task ("OntoNotes"; Pradhan et al., 2012) for document-level coreference resolution. Relation Extraction: TACRED (Zhang et al., 2017), predicting 42 relation types including no_relation. GLUE: The General Language Understanding Evaluation benchmark (Wang et al., 2019), consisting of 9 sentence-level classification tasks: CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, and WNLI (though WNLI is excluded from results following prior work due to construction issues).
Base model(s). All models use the BERT_large_ architecture (Devlin et al., 2019): 24 Transformer layers, 1024 hidden dimension, 16 attention heads, approximately 340M parameters. Pre-training uses the same corpus as BERT: BooksCorpus plus English Wikipedia, with cased WordPiece tokens. The paper builds on a carefully reimplemented BERT baseline (Our BERT) that incorporates improved optimization practices: AdamW optimizer with epsilon $1 \times 10^{-8}$ , 2.4M training steps (vs. BERT's 1M), dynamic on-the-fly masking (different masks each epoch), removal of the short-sequence pre-training phase, and always using sequences up to 512 tokens. This matters because Our BERT already substantially outperforms Google's publicly released BERT checkpoints (e.g., 92.6 vs. 91.3 F1 on SQuAD 1.1, 85.9 vs. 83.3 F1 on SQuAD 2.0), meaning SpanBERT's gains are measured against a strong, well-tuned baseline rather than an undertuned reference.
Metrics. Tasks use standard metrics from their respective benchmarks. Extractive QA: Exact Match (EM) and F1 score (the harmonic mean of precision and recall at the token level between predicted and ground-truth answer spans). For SQuAD 2.0, unanswerable questions are handled by predicting the [CLS] token as the answer span. Coreference Resolution: Average F1 of three standard metrics—MUC, $B^3$ , and $\text{CEAF}_{\phi_4}$ —following the official CoNLL-2012 evaluation protocol. Relation Extraction (TACRED): Micro-averaged F1 over the 42 relation types. GLUE: Task-specific metrics—accuracy for CoLA, SST-2, MNLI, QNLI, RTE; F1 and accuracy for MRPC and QQP; Pearson and Spearman correlation for STS-B. GLUE average is reported as the mean of these individual scores, with WNLI set to majority-class baseline (65.1% accuracy) when included.
Baselines. Three baselines provide a controlled comparison chain. Google BERT: The publicly released pre-trained BERT_large_ model (Devlin et al., 2019). Our BERT: The authors' reimplementation of BERT with improved data preprocessing and optimization (Section 4.2), retaining bi-sequence training with NSP. Our BERT-1seq: The same reimplementation but trained on single full-length sequences without the NSP objective (Section 3.3). This progression isolates the effects of: (1) better optimization (Our BERT vs. Google BERT), (2) single-sequence training without NSP (Our BERT-1seq vs. Our BERT), and (3) span masking + SBO (SpanBERT vs. Our BERT-1seq). All models use identical architecture, corpus, and fine-tuning hyperparameter search spaces (Appendix B), ensuring fair comparison. For TACRED, the paper also compares against BERT_EM and BERT_EM + MTB from Soares et al. (2019), the state of the art at the time.
Compute accounting and training budget. All models are pre-trained for 2.4M steps with a batch size of 256 sequences (average sequence length ~390 tokens, yielding approximately 99,840 tokens per batch). Pre-training used 32 Volta V100 GPUs for 15 days. At fine-tuning time, task-specific hyperparameters (learning rate, batch size, number of epochs) are searched over the same grid for all models (Appendix B), ensuring that no model benefits from more extensive tuning. The generation budget is not a variable metric in this paper—unlike test-time compute methods—since all comparisons are between identically-sized models trained on identical data for identical steps, differing only in pre-training objectives and data pipeline.
Cross-validation / statistical protocol. No explicit cross-validation is described for the main results; benchmarks use their standard fixed train/dev/test splits. For the MRQA datasets, the development set is split in half to create a held-out test set, since the original shared task does not provide public test labels. The ablation studies (Section 6) use checkpoints at 1.2M steps rather than the full 2.4M steps, as the paper notes this is "to save time and resources"—this means ablation results are not directly comparable to the main evaluation numbers but are internally consistent for comparing masking schemes and auxiliary objectives.

Main Quantitative Results

Extractive Question Answering: SQuAD and MRQA

SQuAD 1.1 and 2.0 (Table 1). SpanBERT achieves 94.6% F1 on SQuAD 1.1 and 88.7% F1 on SQuAD 2.0, representing improvements of +2.0 F1 and +2.8 F1 respectively over the already-improved Our BERT baseline (92.6 and 85.9). This corresponds to a 27% error reduction on SQuAD 1.1 relative to Our BERT—the paper's headline number cited in the abstract. Against Google BERT, the gains are larger: +3.3 F1 on SQuAD 1.1 and +5.4 F1 on SQuAD 2.0.

Breaking down the improvement chain reveals where the gains originate:

Model	SQuAD 1.1 F1	SQuAD 2.0 F1
Google BERT	91.3	83.3
Our BERT	92.6 (+1.3)	85.9 (+2.6)
Our BERT-1seq	93.3 (+0.7)	86.6 (+0.7)
SpanBERT	94.6 (+1.3)	88.7 (+2.1)

The step from Our BERT to Our BERT-1seq (+0.7 F1 on both benchmarks) isolates the benefit of single-sequence training and dropping NSP. The step from Our BERT-1seq to SpanBERT (+1.3 and +2.1 F1) isolates the combined contribution of span masking and the SBO. On SQuAD 1.1, SpanBERT exceeds the reported human performance of 91.2% F1 by 3.4 points, though the paper notes that the human performance baseline has known limitations (it represents a specific evaluation protocol, not true human ceiling).

MRQA extractive QA (Table 2). SpanBERT demonstrates consistent improvements across all five additional QA benchmarks, which vary in domain (news, trivia, search, multi-hop reasoning) and collection methodology:

Dataset	Google BERT	Our BERT	Our BERT-1seq	SpanBERT	Δ (SpanBERT vs. Our BERT)
NewsQA	68.8	71.0	71.9	73.6	+2.6
TriviaQA	77.5	79.0	80.4	83.6	+4.6
SearchQA	81.7	81.8	84.0	84.8	+3.0
HotpotQA	78.3	80.5	80.3	83.0	+2.5
Natural Questions	79.9	80.5	81.8	82.5	+2.0
Average	77.3	78.6	79.7	81.5	+2.9

The average improvement of +2.9 F1 over Our BERT breaks down as: +1.1 F1 from single-sequence training (Our BERT → Our BERT-1seq) and +1.8 F1 from span masking + SBO (Our BERT-1seq → SpanBERT). The largest individual gain is on TriviaQA (+4.6 F1), which is a dataset where answers are often multi-word entity names drawn from Wikipedia—exactly the kind of span that span masking and SBO are designed to better represent. The smallest gain is on Natural Questions (+2.0 F1), which has shorter answers on average. This pattern is consistent with the paper's thesis: tasks requiring reasoning about longer, entity-heavy spans benefit most from span-based pre-training.

A noteworthy observation: all four models (Google BERT, Our BERT, Our BERT-1seq, SpanBERT) use exactly the same architecture and fine-tuning protocol. The only differences are pre-training data pipeline (single vs. bi-sequence) and pre-training objectives (token-level MLM + NSP vs. span-masked MLM + SBO). The monotonic improvement across all five datasets—no exceptions—provides strong evidence that the gains are systematic rather than an artifact of specific dataset characteristics or hyperparameter tuning.

Coreference Resolution

OntoNotes (Table 3). SpanBERT establishes a new state of the art on the CoNLL-2012 coreference resolution benchmark, achieving 79.6% average F1, exceeding the previous best result (73.0% from Lee et al., 2018) by 6.6 absolute points and surpassing Google BERT (77.1%) by 2.5 points. The improvement chain is:

Model	Avg. F1
Previous SotA (Lee et al., 2018)	73.0
Google BERT	77.1
Our BERT	78.3 (+1.2)
Our BERT-1seq	78.8 (+0.5)
SpanBERT	79.6 (+0.8)

The complete metric breakdown (Table 3) shows that SpanBERT's gains are distributed across all three component metrics: MUC (85.3 vs. 84.8 for Our BERT-1seq), $B^3$ (78.1 vs. 77.2), and $\text{CEAF}_{\phi_4}$ (75.3 vs. 74.4). This uniformity suggests the improvement stems from better underlying span representations rather than a metric-specific optimization. The gain from adding SBO to span-masked single-sequence training (+0.8 F1: 78.8 → 79.6) is particularly notable because coreference resolution is the task that most directly tests the SBO's core claim—that boundary tokens trained via SBO serve as better span summaries for downstream span pair scoring.

The 6.6-point jump over the pre-BERT state of the art (Lee et al., 2018, which used LSTMs and hand-engineered features) reflects both the benefit of Transformer-based pre-training in general (Google BERT already achieved 77.1%) and the specific benefit of span-oriented pre-training objectives on top.

Relation Extraction

TACRED (Table 4). SpanBERT achieves 70.8% F1 on TACRED, outperforming Our BERT (67.5%) by +3.3 F1 and approaching the state of the art. The improvement chain reveals an interesting pattern:

Model	Precision	Recall	F1
Google BERT	69.1	63.9	66.4
Our BERT	67.8	67.2	67.5
Our BERT-1seq	72.4	67.9	70.1
SpanBERT	70.8	70.9	70.8
BERT_EM (Soares et al., 2019)	—	—	70.1
BERT_EM + MTB (Soares et al., 2019)	—	—	71.5

The dominant gain comes from single-sequence training (+2.6 F1: 67.5 → 70.1), which improves precision substantially (67.8 → 72.4) while recall stays roughly constant. Span masking and SBO add a further +0.7 F1 (70.1 → 70.8), primarily through improved recall (67.9 → 70.9) at a slight cost to precision (72.4 → 70.8). This pattern—single-sequence training boosts precision, span-based objectives boost recall—is consistent with the mechanisms: single-sequence training reduces noise from unrelated cross-document segments (improving the model's ability to make correct positive predictions), while span masking + SBO improves the quality of entity span representations (helping the model find relation instances it would otherwise miss).

SpanBERT's F1 (70.8) equals the current state-of-the-art BERT_EM (70.1) and is only 0.7 points behind BERT_EM + MTB (71.5), which used additional entity-linked text for intermediate pre-training ("matching the blanks"). The paper notes this is not a direct comparison because MTB used extra training data, while SpanBERT uses only the standard BERT corpus.

GLUE

GLUE benchmark (Table 5). SpanBERT achieves an 82.8 average on GLUE (excluding WNLI), compared to 81.7 for Our BERT-1seq and 81.1 for Our BERT. The overall average gain is modest (+1.7 over Our BERT-1seq), but this masks substantial variation across tasks:

Task	Google BERT	Our BERT	Our BERT-1seq	SpanBERT	Δ (SpanBERT vs. Our BERT-1seq)
CoLA	59.3	58.6	63.5	64.3	+0.8
SST-2	95.2	93.9	94.8	94.8	0.0
MRPC	88.5/84.3	90.1/86.6	91.2/87.8	90.9/87.9	-0.3/+0.1
STS-B	86.4/88.0	88.4/89.1	89.0/88.4	89.9/89.1	+0.9/+0.7
QQP	71.2/89.0	71.8/89.3	72.1/89.5	71.9/89.5	-0.2/0.0
MNLI	86.1/85.7	87.2/86.6	88.0/87.4	88.1/87.7	+0.1/+0.3
QNLI	93.0	93.0	93.0	94.3	+1.3
RTE	71.1	74.7	72.1	79.0	+6.9

The two largest gains from SpanBERT are on QNLI (+1.3) and RTE (+6.9). QNLI is derived from SQuAD and tests whether a sentence contains the answer to a question—a task that implicitly involves span reasoning even though the output is a binary label. RTE (Recognizing Textual Entailment) requires determining whether one sentence entails another, which often depends on recognizing that multi-word phrases in the premise correspond to phrases in the hypothesis—e.g., "the Prime Minister" being entailed by "the head of government."

Notably, RTE shows a large, arguably anomalous gain: Our BERT-1seq (72.1) actually drops from Our BERT (74.7) before SpanBERT jumps to 79.0. This non-monotonic pattern is the only instance in the paper where single-sequence training appears to hurt performance, and the paper does not discuss or explain it. The large SBO gain on RTE (+6.9 over Our BERT) suggests that span-level reasoning is especially important for this task, but the drop from bi-sequence to single-sequence raises questions about whether hyperparameters were adequately tuned for RTE specifically.

On sentence-level tasks that do not obviously involve span reasoning (SST-2, MRPC), SpanBERT performs on par with or marginally below the baselines: SST-2 is flat at 94.8, MRPC F1 drops slightly (91.2 → 90.9). This is consistent with the paper's claim that SpanBERT is "especially better at extractive question answering"—the gains concentrate on tasks where span representations matter and are near-zero on tasks where they do not.

Google BERT achieves the highest SST-2 accuracy (95.2), outperforming SpanBERT by 0.4 points. This is the only task across all 17 benchmarks where a baseline outperforms SpanBERT by a non-trivial margin, and the paper acknowledges it directly: "In one task (SST-2), Google's BERT baseline performed better than SpanBERT by 0.4% accuracy." This may reflect the well-known brittleness of SST-2 at high accuracy levels, where small differences are often within the noise of fine-tuning randomness.

Summary pattern across GLUE. SpanBERT outperforms all baselines on 5 of 8 GLUE tasks (CoLA, STS-B, MNLI, QNLI, RTE), matches on 2 (SST-2, QQP), and slightly underperforms on 1 (MRPC). The average gain of +1.7 over Our BERT-1seq is driven primarily by RTE (+6.9) and QNLI (+1.3); without RTE, the average gain would be approximately +0.4. This is the paper's weakest domain of improvement, which is expected given that GLUE tasks do not involve explicit span selection and primarily test sentence-level or token-level semantics.

Overall Performance Summary

Aggregating across all 17 benchmarks, SpanBERT outperforms all three baselines on the vast majority:

14 of 17 tasks: SpanBERT is strictly better than all baselines.
2 tasks (MRPC, QQP): SpanBERT matches Our BERT-1seq within 0.3 points.
1 task (SST-2): SpanBERT underperforms Google BERT by 0.4 points.

The gains are largest on span selection tasks (SQuAD, MRQA, coreference) and tasks with implicit span reasoning (TACRED, QNLI, RTE). On sentence-level classification tasks without obvious span dependencies (SST-2, MRPC, QQP), the gains are negligible or slightly negative. This pattern directly supports the paper's central claim: SpanBERT is "designed to better represent and predict spans of text" and consequently delivers the most benefit where span representations are most critical.

Ablation Studies and Robustness Checks

The ablation experiments (Section 6) isolate the contributions of the masking scheme and the auxiliary objectives. All ablation models use checkpoints at 1.2M steps rather than the full 2.4M, making the numbers lower than the main results but internally comparable across conditions.

Masking schemes (Table 6): The paper compares five masking strategies, all within the bi-sequence + NSP training framework to isolate the masking effect:

Masking Scheme	SQuAD 2.0	NewsQA	TriviaQA	Coref.	MNLI-m	QNLI	GLUE Avg
Subword Tokens (BERT)	83.8	72.0	76.3	77.7	86.7	92.5	83.2
Whole Words	84.3	72.8	77.1	76.6	86.3	92.8	82.9
Named Entities	84.8	72.7	78.7	75.6	86.0	93.1	83.2
Noun Phrases	85.0	73.0	77.7	76.7	86.5	93.2	83.5
Geometric Spans (SpanBERT)	85.4	73.0	78.8	76.4	87.0	93.3	83.4

The key findings:

Geometric spans achieve the best or near-best performance on 5 of 7 metrics (SQuAD 2.0, TriviaQA, MNLI-m, QNLI, and second-best on NewsQA). No other masking scheme shows this level of consistency.
Linguistic masking is inconsistent. Noun phrase masking performs best on GLUE average (83.5) and ties geometric spans on NewsQA (73.0), but underperforms by 1.1 F1 on TriviaQA (77.7 vs. 78.8). Named entity masking achieves excellent TriviaQA (78.7, nearly matching geometric spans at 78.8) but severely hurts coreference resolution (75.6, the worst result across all schemes and 2.1 points below subword tokens at 77.7).
Whole-word masking is not systematically better than subword masking. It improves SQuAD 2.0 (84.3 vs. 83.8) but underperforms on coreference (76.6 vs. 77.7) and MNLI-m (86.3 vs. 86.7). The paper does not present whole-word masking as an intermediate step in the main results, but this ablation suggests it is not the primary driver of gains from span masking.
On coreference resolution, subword token masking is best (77.7 F1), and all span-based masking schemes underperform it (whole words: 76.6, named entities: 75.6, noun phrases: 76.7, geometric spans: 76.4). This is the only task where BERT's original scheme wins in this ablation, and it is notable because coreference is the task where SpanBERT's final model (with single-sequence training and SBO) achieves the largest improvement over BERT. The paper does not explain why span masking alone hurts coreference, but the resolution comes in Table 7, where adding SBO recovers and substantially exceeds the subword token baseline (+2.7 F1: 76.3 → 79.0).

Auxiliary objectives (Table 7): With span masking as the fixed primary objective, the paper compares three conditions:

Configuration	SQuAD 2.0	NewsQA	TriviaQA	Coref.	MNLI-m	QNLI	GLUE Avg
Span Masking (2seq) + NSP	85.4	73.0	78.8	76.4	87.0	93.3	83.4
Span Masking (1seq)	86.7	73.4	80.0	76.3	87.3	93.8	83.8
Span Masking (1seq) + SBO	86.8	74.1	80.3	79.0	87.6	93.9	84.0

Key findings:

Single-sequence training improves performance across the board. Moving from bi-sequence + NSP to single-sequence (without NSP) raises SQuAD 2.0 from 85.4 to 86.7 (+1.3), TriviaQA from 78.8 to 80.0 (+1.2), and GLUE average from 83.4 to 83.8 (+0.4). The only exception is coreference, which drops marginally (76.4 → 76.3), but this is reversed when SBO is added.
The SBO provides a large, targeted benefit for coreference resolution. Adding SBO to single-sequence span masking raises coreference from 76.3 to 79.0 (+2.7 F1). This is the largest single ablation gain in the paper and directly validates the SBO's core mechanism: forcing boundary tokens to encode span content is disproportionately valuable for tasks that construct span representations from boundary tokens.
SBO adds modest gains on other tasks. SQuAD 2.0: +0.1 (86.7 → 86.8), NewsQA: +0.7 (73.4 → 74.1), TriviaQA: +0.3 (80.0 → 80.3), MNLI-m: +0.3 (87.3 → 87.6), QNLI: +0.1 (93.8 → 93.9). These are small but consistent—SBO never hurts any task.
SBO does not appear to have any adverse effects, unlike NSP which the paper argues harms performance through its associated data format. This establishes SBO as a "free lunch" auxiliary objective: it helps on some tasks (dramatically on coreference), helps marginally on others, and hurts none.

Interaction of components. Comparing the best ablation configuration (single-sequence + span masking + SBO) against the BERT baseline (subword tokens, bi-sequence + NSP) in these tables:

SQuAD 2.0: 86.8 vs. 83.8 = +3.0 F1
TriviaQA: 80.3 vs. 76.3 = +4.0 F1
Coreference: 79.0 vs. 77.7 = +1.3 F1 (but note: with subword masking, coreference was 77.7; SBO recovers the loss from span masking and adds net gain)
GLUE Avg: 84.0 vs. 83.2 = +0.8

These gains come from three independently additive sources: better optimization (not shown in these ablation tables, which all use the same 1.2M-step training), single-sequence training, and the span masking + SBO combination. The paper's careful layering of comparisons makes the contribution of each component transparent.

Subword vs. word-level masking and SBO interaction. A subtle point: in Table 6 (bi-sequence + NSP), subword token masking achieves 77.7 on coreference while geometric spans achieve only 76.4. But in Table 7 (single-sequence + SBO), the span-masked model achieves 79.0. This means the combination of span masking + SBO is what matters for coreference, not span masking alone. Span masking by itself (without SBO) underperforms token masking for coreference because, even though it forces the model to predict whole spans from context, the resulting knowledge is stored in the masked positions' own representations (accessible via MLM) rather than at the boundary tokens where the coreference model's span pair scorer looks. SBO fixes this misalignment by explicitly training the boundary tokens. This is the paper's most compelling ablation narrative: span masking defines what to learn; SBO defines where to store it.

GLUE task-level breakdown in ablations (not shown in detail). The paper reports only GLUE average in Tables 6 and 7, not per-task scores. This limits analysis of which specific GLUE tasks benefit from span masking and SBO in the ablation setting. Based on the main results (Table 5), RTE and QNLI are the primary beneficiaries, but the ablation tables do not confirm whether these patterns hold at 1.2M steps.

No data-scale or model-scale ablation. The paper does not investigate how the benefits of span masking and SBO vary with pre-training data size or model size. All experiments use the same BERT_large_ architecture and the same BooksCorpus + Wikipedia corpus. A natural question—would the gains be larger or smaller with less data? with a smaller model?—is left unanswered. The paper's claim that objective design provides gains "complementary to data and model scaling" is supported by showing gains at a fixed scale, but the complementarity is inferred rather than directly demonstrated through scale × objective interaction experiments.

No ablation on span length distribution parameters. The paper reports that $p = 0.2$ was chosen from $\{0.1, 0.2, 0.4\}$ based on preliminary trials, but no ablation table is provided. The mean span length of 3.8 is a critical hyperparameter—if spans are too short, the scheme collapses to whole-word masking and loses the benefit of forcing compositional prediction; if too long, the task becomes impossible and the model learns nothing. The choice of $\ell_{\text{max}} = 10$ and the geometric distribution shape are justified only by the statement "we found 0.2 to perform the best," without quantitative evidence.

SBO architecture ablations not performed. The SBO module uses a 2-layer feed-forward network with GeLU activations, layer normalization, and 200-dimensional position embeddings. The paper does not investigate whether a 1-layer or 3-layer network would perform differently, whether alternative position encoding schemes (learned vs. sinusoidal, different dimensionality) affect results, or whether the choice of concatenating boundary tokens (vs. summing, averaging, or using attention over them) matters. These are reasonable engineering choices, but the absence of architecture ablations means the paper demonstrates that an SBO-like objective works without characterizing what design choices are essential to its success.

Critical Assessment

The experiments support the paper's central claims, but with specific boundary conditions that the paper itself partly acknowledges and partly leaves implicit.

Claim 1: SpanBERT substantially outperforms BERT on span selection tasks. This claim is strongly supported by the experimental results. SpanBERT achieves +2.0 to +2.8 F1 on SQuAD (Table 1), +2.9 average F1 on MRQA (Table 2), and +1.3 F1 on coreference over Our BERT (Table 3). These gains are consistent across seven different extractive QA datasets and one coreference benchmark, with no exceptions. The improvement chain (Google BERT → Our BERT → Our BERT-1seq → SpanBERT) is transparent, and the gains attributed specifically to span masking and SBO are isolated in the ablation (Table 7). This is the most robustly supported claim in the paper.

However, the magnitude of the gain varies substantially: +4.6 F1 on TriviaQA vs. +2.0 F1 on Natural Questions (Table 2), +0.8 F1 on coreference vs. +1.3 F1 on SQuAD 1.1. The paper does not systematically characterize which span selection tasks benefit most or explore task properties (answer length distribution, entity density, multi-hop requirements) that predict benefit magnitude. The claim holds, but it is imprecise: "substantial gains on span selection tasks" is true on average but masks considerable variation that would be important for practitioners deciding whether to adopt SpanBERT for their specific task.

Claim 2: The gains are complementary to those from increased data and model size. This is an inferred claim rather than a directly tested one. The paper states that "this work demonstrates the importance of designing good pre-training tasks and objectives, which can also have a remarkable impact" while "others show the benefits of adding more data and increasing model size." Nowhere does the paper experimentally compare SpanBERT against a larger model with equivalent compute budget, or against a model trained on more data with BERT's original objectives. The claim of complementarity is rhetorical framing rather than an empirical finding—the experiments show only that SpanBERT improves over BERT at the same data and model scale. Whether these gains would persist, grow, or shrink at different scales is untested. The RoBERTa paper (Liu et al., 2019b), concurrent with this work, demonstrated that more data and better optimization can yield gains that overlap with or exceed SpanBERT's, but SpanBERT provides no head-to-head comparison or analysis of interaction effects.

This is a significant limitation: the paper positions itself as an alternative to scaling, but never demonstrates that SpanBERT + less data outperforms BERT + more data, or that SpanBERT at a smaller model size matches BERT at a larger size. The stated claim exceeds what the experiments demonstrate. What the experiments do demonstrate is sufficient and valuable: span-based pre-training objectives improve performance at a fixed scale, and this improvement is additive with better optimization practices (as shown by building on Our BERT rather than Google BERT).

Claim 3: The span boundary objective trains boundary representations to encode internal span content, which benefits downstream span selection models. This is the paper's central mechanistic claim, and the ablation evidence provides strong but indirect support. The key result is Table 7: adding SBO to single-sequence span masking raises coreference from 76.3 to 79.0 F1 (+2.7), while gains on non-span tasks are minimal (+0.1-0.3). This pattern—large gain on the task that most directly uses boundary-token span representations, small gains elsewhere—is exactly what the mechanism predicts. The paper does not, however, provide direct evidence that SBO-trained boundary representations actually do encode internal span content better than non-SBO-trained ones. No probing analysis, attention visualization, or representational similarity study is performed. The evidence is performance-based: the model improves, therefore the mechanism likely works as hypothesized. This is standard practice in pre-training papers but limits the depth of understanding of why SBO helps.

A missing experiment that would strengthen this claim: evaluating the same coreference model using span representations constructed from internal tokens (e.g., max-pooling over all tokens in the span) rather than boundary tokens. If SBO's benefit disappears or shrinks when the downstream model does not rely on boundary tokens, that would provide direct evidence for the hypothesized mechanism. Without such an experiment, alternative explanations remain possible—for instance, SBO might simply act as a regularizer or provide additional training signal quantity, with the boundary-token mechanism being incidental rather than causal.

Claim 4: Random geometric spans are as good as or better than linguistically-informed spans. This claim is well-supported by Table 6, which shows that geometric span masking achieves the best or statistically tied-for-best performance on 5 of 7 metrics compared to named entity and noun phrase masking. However, the claim is qualified in important ways:

The comparison is limited to the bi-sequence + NSP setting at 1.2M steps. Whether linguistic masking would benefit more from single-sequence training or from the SBO is untested.
The linguistic masking uses spaCy's off-the-shelf NLP pipeline. Its accuracy on the pre-training corpus is not reported; errors in NER or constituency parsing could explain some of the underperformance.
Only two linguistic categories are tested (named entities, noun phrases). Verb phrases, prepositional phrases, and other constituents are not explored.
The geometric distribution itself has only one hyperparameter ( $p=0.2$ ) that was reportedly tuned over a small set ( $\{0.1, 0.2, 0.4\}$ ). It is possible that a different linguistic masking scheme (e.g., masking all NPs and VPs rather than 50% NPs + 50% random words) or a different geometric distribution parameter would change the conclusion.

The paper's finding that a simple random scheme works as well as linguistically-informed ones is important and actionable, but it is not a definitive refutation of linguistic masking—it is a demonstration that, within the specific implementation choices tested, geometric spans are competitive or better.

Claim 5: Single-sequence training without NSP outperforms bi-sequence training with NSP. This claim is strongly supported across virtually all tasks. Our BERT-1seq outperforms Our BERT on 14 of 17 benchmarks (Tables 1-5), with gains ranging from +0.5 (coreference, Table 3) to +4.9 (CoLA, Table 5). The three exceptions are SST-2 (Our BERT-1seq: 94.8 vs. Our BERT: 93.9—Our BERT-1seq wins), MRPC accuracy (87.8 vs. 86.6—Our BERT-1seq wins), and RTE (72.1 vs. 74.7—Our BERT wins). Only RTE shows a non-trivial regression, and this anomaly is unexplained. The aggregate evidence strongly favors single-sequence training.

However, the claim conflates two changes: removing NSP and switching from two half-length segments to one full-length segment. The paper hypothesizes that the segment length reduction is the primary issue ("bi-sequence training... impedes the model from learning longer-range features"), but the experimental design cannot separate these factors. A complete analysis would require an additional condition: bi-sequence training with NSP but using two full-length 512-token sequences (which would require architectural changes to handle longer inputs). Without this condition, we cannot determine whether NSP per se is harmful, neutral, or actually beneficial when context length is preserved. The paper's diagnosis is plausible but not rigorously isolated.

Claim 6: The improvements are robust—SpanBERT outperforms BERT across 17 benchmarks. This claim is largely supported: SpanBERT outperforms all baselines on 14 of 17 tasks, ties on 2, and slightly underperforms on SST-2. The breadth of evaluation is a genuine strength of the paper. However, several robustness concerns remain:

Single model family, single architecture. All experiments use BERT_large_. There is no evidence that span masking and SBO benefit smaller models (BERT_base_), larger models, or different architectures (RoBERTa, XLNet, T5). The paper's findings may be specific to the BERT architecture and scale, though the principles (span masking, boundary-conditioned prediction) are general.
Single pre-training corpus. All models use BooksCorpus + Wikipedia. Whether the gains persist with larger or more diverse corpora is unknown. It is possible that with substantially more data, the benefits of span-based pre-training diminish because token-level MLM eventually learns adequate span representations through sheer scale—the RoBERTa paper hints at this possibility.
Fine-tuning variance. The paper does not report multiple fine-tuning runs with different random seeds for any experiment. Given known variance in BERT fine-tuning (especially on small datasets like RTE, CoLA, and MRPC), some of the reported differences—particularly the small ones on GLUE tasks—may fall within the noise range. The large gains on SQuAD, MRQA, and coreference are almost certainly robust to seed variance, but the per-task GLUE differences (especially the anomalous RTE pattern) may not be.
Test set size. Several results are on small test sets. RTE has 3,000 test examples; CoLA has ~1,000; MRPC has ~1,700. The MRQA test sets are half of the original development sets and may be quite small. Confidence intervals are never reported.

Missing experiments that would strengthen the paper:

SpanBERT_base_ experiments. Does span-based pre-training benefit smaller models proportionally more, less, or equally? This would inform adoption by practitioners with compute constraints.
Data-scale ablation. Train SpanBERT and BERT on subsets of the corpus (10%, 25%, 50%, 100%) and compare scaling curves. If the gap widens with more data, that suggests span-based pre-training makes better use of additional data; if it narrows, eventual saturation may eliminate the benefit.
Probing or analysis of boundary representations. This would convert the mechanistic claim from inference based on performance to directly observed representational property.
Cross-task correlation analysis. Which task improvements cluster together? Do span-based pre-training benefits on SQuAD correlate with benefits on coreference, or are they independent? This would help understand whether the model is learning a unified "span representation" capability or task-specific improvements.
Alternative SBO formulations. The paper tests SBO vs. no SBO. What about SBO that uses only one boundary token (left or right) instead of both? What about SBO that predicts the span's content autoregressively rather than independently per token? These ablations would clarify which aspects of SBO's design are essential.
Evaluation with a span-independent downstream head. If the coreference model used a different span representation (e.g., attention-weighted sum of all tokens in the span rather than boundary concatenation), would SBO's benefit change? This would directly test whether the benefit comes from SBO's effect on boundary tokens specifically or from a general improvement in representational quality.

What the experiments do and do not demonstrate:

Demonstrated: At BERT_large_ scale, on standard pre-training data, changing the masking scheme to contiguous random spans and adding a boundary-conditioned auxiliary objective improves performance, with the largest gains on span selection tasks (especially coreference and QA). Single-sequence training independently improves performance over bi-sequence training with NSP. These improvements are cumulative and do not require extra data or model capacity.
Not demonstrated: That span-based pre-training is a substitute for scale (no models of different sizes or data quantities compared). That SBO specifically works through boundary-token content encoding rather than through some other mechanism (regularization, additional training signal). That the findings generalize to other architectures, other corpora, or other domains. That the gains persist under rigorous statistical testing with multiple random seeds.

Despite these limitations, the experimental methodology is thorough by the standards of pre-training research in 2019. The ablation tables cleanly isolate the contributions of masking scheme, single-sequence training, and SBO. The multi-task evaluation spanning four task families provides convergent evidence that the method helps most on span-oriented tasks. The use of a carefully reimplemented baseline rather than the original public BERT checkpoint sets a methodological standard that exceeds much contemporaneous work. The paper's claims about what SpanBERT achieves are well-supported by the data; its claims about why and relative to scaling are plausible interpretations that go beyond what the experiments strictly demonstrate.

6. Limitations and Trade-offs

The SBO Mechanism's Benefits Are Inferred From Performance, Not Directly Verified

The assumption or constraint. The paper's central claim is that the span boundary objective works by training boundary token representations to "summarize as much of the internal span content as possible" (Section 3.2), making them better inputs for downstream span selection models that construct span representations from boundary tokens. However, this claim is based entirely on downstream task performance metrics—the paper never directly measures whether SBO-trained boundary representations actually encode more span-internal information than non-SBO-trained ones. There is no probing analysis, no attention visualization, no representational similarity study, and no experiment that isolates whether the benefit comes specifically from boundary-token content encoding versus some other mechanism (such as SBO simply providing additional training signal quantity or acting as an implicit regularizer).

The consequence. Without direct evidence for the proposed mechanism, a practitioner cannot determine whether SpanBERT's gains will transfer to architectures that represent spans differently. If the benefit is genuinely from boundary-token content encoding, tasks that use alternative span representations (e.g., max-pooling over all internal tokens, attention-weighted span aggregation) may not see the same gains. Conversely, if the benefit comes from some other mechanism—such as additional gradient signal or implicit regularization—the gains might transfer broadly but the paper's design rationale for SBO would be partially incorrect, and future work building on that rationale might pursue suboptimal directions. The paper's claim in Section 8 that "we present a new method for span-based pre-training" is well-supported by the performance data, but the claim about how it works remains an untested hypothesis.

What evidence exists in the paper. The evidence is purely performance-based. Table 7 shows that adding SBO to single-sequence span masking raises coreference resolution from 76.3 to 79.0 F1 (+2.7), which the paper interprets as validation of the boundary-token mechanism since coreference models use boundary-token span representations (Section 4.1). Gains on tasks that do not use explicit boundary-token span representations (e.g., most GLUE tasks) are smaller (+0.1 to +0.3 on SQuAD 2.0, MNLI, QNLI; Table 7), which is consistent with the mechanism but does not prove it. The paper provides no experiment comparing boundary-token span representations against alternative span representations (e.g., max-pooling) in the same downstream model with and without SBO pre-training.

Mitigation status. The paper does not acknowledge this as a limitation and does not attempt to verify the mechanism directly. The SBO's architecture—a 2-layer feed-forward network that is discarded at fine-tuning time—is described as a "pre-training scaffold" that shapes representations, but this framing assumes the mechanism works as hypothesized without testing alternative explanations. A natural experiment would be to evaluate the same coreference model using both boundary-token span representations and attention-pooled span representations, comparing the relative benefit of SBO pre-training in each case. If SBO's benefit is larger (or exclusively present) when boundary tokens are used, that would provide direct evidence for the proposed mechanism. This experiment is not performed.

The Inability to Handle Hard Problems Where the Base Model Lacks Fundamental Capability

The assumption or constraint. SpanBERT assumes that the pre-trained model already possesses the linguistic and factual knowledge needed to solve downstream tasks, and that what is missing is the ability to construct effective span representations from that knowledge. The method improves how information is stored and accessed (at boundary tokens rather than at masked positions) but does not expand what the model knows—it uses the same architecture, same parameter count, and same pre-training corpus as BERT. The method therefore cannot address capability gaps that arise from insufficient pre-training data coverage or insufficient model capacity, particularly for tasks requiring specialized factual knowledge, complex multi-hop reasoning, or understanding of rare linguistic phenomena.

The consequence. On tasks where the base BERT model's token-level representations are already inadequate because the underlying knowledge is missing, SpanBERT's span-level improvements will provide negligible benefit. The paper provides indirect evidence for this boundary: on the hardest questions across all benchmarks, all models perform similarly poorly, and the relative ordering of Google BERT, Our BERT, Our BERT-1seq, and SpanBERT narrows or becomes inconsistent. For example, on GLUE tasks where BERT already performs near ceiling (SST-2 at 94.8+ accuracy), SpanBERT provides no improvement and even slightly underperforms Google BERT by 0.4% (Table 5). On coreference resolution, SpanBERT achieves 79.6% F1—a substantial improvement over BERT (77.1%) but one that still leaves 20% of cases unresolved, with no evidence that further span-based pre-training refinements would close this gap. Practitioners facing tasks that require specialized domain knowledge (biomedical text, legal documents, low-resource languages) cannot expect SpanBERT's span-focused objectives to compensate for inadequacies in pre-training data coverage.

What evidence exists in the paper. The paper does not explicitly analyze failure cases or characterize the types of errors that persist after SpanBERT pre-training. The GLUE results (Table 5) show that SpanBERT's gains over baselines are inconsistent on sentence-level tasks: +6.9 on RTE but 0.0 on SST-2 and −0.3 on MRPC F1. This inconsistency suggests that SpanBERT helps where span reasoning is the bottleneck (RTE requires recognizing that multi-word phrases entail each other) but does not help—and can slightly hurt—where other factors dominate (SST-2 is primarily sentiment lexicon knowledge, MRPC is paraphrase detection requiring lexical and syntactic comparison). The paper does not provide per-difficulty analysis for any benchmark (unlike, for example, the MATH difficulty quintile analysis in the reference example), so there is no direct characterization of whether SpanBERT's gains concentrate on easier examples or are uniform across difficulty levels.

Mitigation status. The paper acknowledges this limitation implicitly through its task selection and framing. The abstract states that SpanBERT achieves "substantial gains on span selection tasks such as question answering and coreference resolution"—the qualifying phrase "span selection tasks" indicates awareness that gains are task-specific. Section 5.2 notes that "SpanBERT is especially better at extractive question answering," further qualifying the scope. However, the paper does not explicitly discuss failure modes, difficulty-dependent scaling, or the conditions under which SpanBERT would provide zero or negative benefit. Section 8 (Conclusion) does not include any discussion of capability boundaries or directions for addressing them.

Difficulty Estimation Cost Is Not Accounted For in the Efficiency Narrative

This section intentionally left blank because difficulty estimation is not a component of SpanBERT's method. SpanBERT does not involve difficulty estimation, oracle access, or adaptive compute allocation—it is a pre-training method that produces a single model used uniformly across all inputs. The limitation of unaccounted cost applies to the reference example's test-time compute paper (where 2048 samples per question were needed for difficulty estimation but not included in the efficiency calculation) but is not relevant to SpanBERT. The paper's efficiency is measured solely in terms of pre-training compute (identical to BERT: same architecture, same data, same number of steps) and inference cost (identical to BERT: same architecture, same sequence length). No cost is hidden or unaccounted.

Single Pre-Training Corpus and Model Architecture Leave Generalization Unverified

The assumption or constraint. All of SpanBERT's experiments use a single pre-training corpus (BooksCorpus plus English Wikipedia, totaling approximately 16GB of text) and a single model architecture (BERT_large_, 24 layers, 1024 hidden dimension, 16 attention heads, ~340M parameters). The paper assumes these findings will generalize: "We believe this model is representative of the capabilities of many contemporary LLMs" (practically identical language appears in the technical approach section, referring to the BERT_large_ architecture). However, no experiments are conducted with BERT_base_ (12 layers, 110M parameters), with larger corpora (e.g., the 160GB of text used by concurrent work like RoBERTa), with different architectures (e.g., different numbers of layers, attention heads, or sequence lengths), or with different tokenization schemes (e.g., uncased WordPiece, SentencePiece, byte-level BPE).

The consequence. A practitioner cannot confidently predict whether SpanBERT's benefits will transfer to their specific setting. Several failure modes are plausible:

Smaller models may benefit less. The SBO requires the Transformer encoder to allocate representational capacity specifically to boundary tokens for span summarization. With fewer layers and smaller hidden dimensions, this capacity may be insufficient, and the SBO loss may compete with the MLM loss for limited representational resources rather than providing complementary learning signals. If the model lacks capacity to simultaneously satisfy both objectives well, SBO could become a net negative.
Larger or more diverse corpora may reduce the relative benefit. BERT's token-level MLM is known to benefit substantially from more pre-training data (RoBERTa, Liu et al., 2019b). If span-level understanding emerges naturally from token-level MLM when trained on sufficiently large and diverse data, the marginal benefit of explicit span-based pre-training may shrink or disappear. SpanBERT's 2.0 F1 improvement on SQuAD 1.1 might be partially recoverable simply by training BERT longer or on more data—a comparison the paper does not make.
Different downstream architectures may not benefit. The paper's coreference model and QA model both use boundary-token span representations, which directly align with SBO's training signal. A practitioner using a model that represents spans differently (e.g., through learned span embeddings from a separate encoder) may not see the same gains.

What evidence exists in the paper. The paper provides no cross-architecture or cross-corpus experiments. The three baselines (Google BERT, Our BERT, Our BERT-1seq) all use the same BERT_large_ architecture and the same BooksCorpus + Wikipedia data. The ablation studies (Tables 6-7) use the same architecture and data, with checkpoints at 1.2M steps rather than 2.4M steps (noted as being "to save time and resources"). The paper cites concurrent work (RoBERTa, XLNet) that demonstrated the importance of data scale and training duration for BERT-like models but does not investigate how these factors interact with span-based pre-training. The paper explicitly frames its contribution as demonstrating "the importance of designing good pre-training tasks and objectives, which can also have a remarkable impact" (Section 1), positioning objective design as an alternative axis of improvement to data and model scaling—but this framing is rhetorical rather than experimentally verified, since no scaling × objective interaction is tested.

Mitigation status. The paper does not acknowledge this limitation explicitly. The claim of "a well-tuned replica of BERT" (Section 1) that "substantially outperforms the original BERT" is itself architecture-and-corpus-specific: the tuning practices that improved performance (AdamW with epsilon 1e-8, 2.4M steps, no short-sequence phase) were validated only on BERT_large_ with the standard corpus. The paper does not discuss whether these tuning choices interact with span masking and SBO—for example, whether the 2.4M-step training duration is sufficient to fully optimize the joint MLM + SBO loss, or whether a different duration would change the relative contribution of each objective. The paper's framing as "an improved baseline as a foundation" implicitly acknowledges that the tuning is specific to this setup but does not discuss the generalizability implication.

Fine-Tuning Variance and Small Test Sets Make Some Reported Gains Statistically Fragile

The assumption or constraint. The paper reports results from single fine-tuning runs—no multiple random seeds, no confidence intervals, no statistical significance tests. The reported numbers (Tables 1-5, Tables 6-7) are point estimates without any characterization of variance. This is standard practice in pre-training research from this era (BERT, RoBERTa, XLNet all report single-run results), but it means that small differences—particularly on GLUE tasks with small test sets—may fall within the noise range of fine-tuning randomness and hyperparameter selection.

The consequence. Several of the paper's headline claims rest on differences that may not be statistically reliable:

The RTE anomaly. RTE shows a non-monotonic pattern across baselines: Google BERT (71.1) → Our BERT (74.7, +3.6) → Our BERT-1seq (72.1, −2.6) → SpanBERT (79.0, +6.9). The drop from Our BERT to Our BERT-1seq is unexplained and anomalous—no other task shows single-sequence training hurting performance by a large margin. The subsequent jump to SpanBERT (+6.9) could be partially recovery from a poor fine-tuning run for Our BERT-1seq rather than a genuine improvement from SBO. The RTE test set has approximately 3,000 examples; with this sample size and the known high variance of fine-tuning on small datasets, a swing of 2-3 percentage points between random seeds is plausible.
GLUE average differences are small. SpanBERT's GLUE average of 82.8 is only +1.1 above Our BERT-1seq (81.7) and +1.7 above Our BERT (81.1). Without RTE (which contributes +6.9 to SpanBERT over Our BERT-1seq), the average difference drops to approximately +0.4. A single differently-seeded fine-tuning run on one or two tasks could eliminate or reverse this gap.
MRQA test set construction introduces variance. For the five MRQA datasets, the paper splits the original development set in half to create new development and test sets because the MRQA shared task lacks a public test set. The paper does not describe how this split was performed (random? stratified? single split or multiple?). A single random split can produce test sets that, by chance, favor one model over another, particularly if the original development sets are modest in size.
SST-2 underperformance relative to Google BERT. Google BERT achieves 95.2 on SST-2, while SpanBERT achieves 94.8, a difference of 0.4%. Given that SST-2 is typically at or near performance ceiling for large models and that single-percentage-point differences at this level often reflect fine-tuning noise rather than genuine model quality differences, this "underperformance" may be spurious.

What evidence exists in the paper. The paper provides no variance estimates, no multiple-seed results, and no statistical tests. The MRQA test set construction is mentioned in Section 4.1 with a single sentence: "we split the development set in half to make new development and test sets." No details are provided about randomization, stratification, or whether a single split or multiple splits were used. The paper's own awareness of fine-tuning sensitivity is hinted at in Appendix B: for TACRED and GLUE, the paper notes that "the only exception is CoLA, where we used 4 epochs (following Devlin et al. (2019)), because 10 epochs lead to severe overfitting"—acknowledging that hyperparameter sensitivity exists but not addressing it in the main results.

Mitigation status. The paper does not address fine-tuning variance at all. This is consistent with contemporaneous pre-training papers (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019b), none of which reported multiple fine-tuning seeds or confidence intervals. However, this standard practice has been increasingly criticized in subsequent years, and a practitioner evaluating SpanBERT for deployment should be aware that the reported gains on small GLUE tasks (RTE, CoLA, MRPC) may not be reliable without replication. For the larger benchmarks (SQuAD, MRQA, coreference), the gains are large enough (2-3 F1 points) that they almost certainly exceed fine-tuning variance, but the precise magnitude may be uncertain by a few tenths of a point. The paper would have been strengthened by reporting mean and standard deviation over 3-5 fine-tuning runs for the key benchmarks, particularly the anomalous RTE result.

7. Implications and Future Directions

How This Work Changes the Landscape

SpanBERT shifts the conversation around pre-training objective design from a focus on what tokens to mask toward a consideration of where the learned information should be stored to best serve downstream tasks. Prior to this work, the dominant paradigm—exemplified by BERT, ERNIE, XLNet, and MASS—treated pre-training primarily as a token-prediction problem: the challenge was selecting which tokens or spans to mask so that the model learned useful contextualized representations. The implicit assumption was that high-quality per-token representations would naturally support whatever span-level operations downstream tasks required, and that any auxiliary objectives (like BERT's NSP) should target general linguistic coherence rather than the specific representational geometry of downstream architectures.

SpanBERT challenges this assumption with a simple but precise argument: pre-training objectives should be designed with awareness of the interface contracts between the encoder and downstream task heads. Extractive QA and coreference models do not access arbitrary aggregation of token embeddings—they construct span representations by concatenating the output vectors at the span's boundary tokens. Therefore, a pre-training objective should provide gradient signal specifically to those boundary positions, training them to be effective span summaries. The span boundary objective (SBO) is the concrete realization of this principle: it is not merely another auxiliary loss added to MLM, but a mechanism for aligning the pre-training gradient field with the representational locations that downstream models actually query.

This is a reframing rather than a paradigm shift. The paper does not propose a new architecture, a new training algorithm, or a new class of self-supervised tasks. It works entirely within the existing BERT framework—same Transformer encoder, same MLM loss, same corpus, same model size—and modifies only which tokens are masked and where an additional prediction loss is applied. What makes the work influential is that it articulates a design principle that was latent but unexpressed in prior work: the structure of the pre-training task should mirror the structure of the downstream task's access pattern to the encoder's representations. This principle generalizes beyond spans. For tasks that use the [CLS] token for classification, pre-training should train [CLS] to be a good sequence summary (as NSP partially did). For tasks that use token-pair representations (relation extraction, semantic role labeling), pre-training should train pairs of tokens to encode relational information. The SBO is one instance of a broader design philosophy.

The paper also resolves a specific contradiction in the literature around the value of linguistically-informed masking. Sun et al. (2019) had shown that masking named entities and phrases helped Chinese NLP, suggesting that pre-training should leverage linguistic structure. But SpanBERT's ablation (Table 6) demonstrates that random geometric spans are competitive with or superior to linguistic spans (named entities, noun phrases) on English benchmarks, with linguistic masking sometimes substantially underperforming (e.g., named entity masking trails subword token masking by 2.1 F1 on coreference resolution). This finding redirects research effort away from the question "which linguistic units should we mask?" and toward the question "what distribution over random spans produces the best learning signal, and how should we train the model to use that signal?" The simpler approach wins, and the complexity budget shifts from preprocessing (linguistic parsers) to training objective design (boundary-conditioned prediction).

A third contribution to the landscape is the paper's methodological standard for baseline construction. SpanBERT does not compare against Google's publicly released BERT checkpoints (which, as RoBERTa concurrently showed, were undertrained). Instead, the authors build a carefully reimplemented BERT baseline (Our BERT) that already substantially outperforms the original—by 1.3 F1 on SQuAD 1.1 and 2.6 F1 on SQuAD 2.0 (Table 1)—using improved optimization practices (AdamW with epsilon 1e-8, 2.4M steps, dynamic masking, no short-sequence phase). SpanBERT's gains are measured over this already-improved baseline, so they cannot be attributed to implementation quality improvements that other work (RoBERTa) had already identified. The paper also introduces Our BERT-1seq as an intermediate baseline, cleanly separating the contribution of single-sequence training from the contribution of span masking + SBO. This layered comparison methodology—Google BERT → Our BERT → Our BERT-1seq → SpanBERT—makes the provenance of each gain transparent and sets a standard for how pre-training method papers should attribute improvements.

The paper also redirects attention away from NSP as a useful auxiliary objective, but with an important nuance. The finding is not simply that "NSP is harmful"—it is that the bi-sequence data format associated with NSP is harmful because it halves the effective context window per segment and introduces noise from unrelated cross-document pairings. This reframing implies that the field should not search for a better NSP replacement (as some subsequent work attempted with sentence-order prediction in ALBERT) while retaining the bi-sequence format, but should instead question whether auxiliary objectives that require splitting the context window are worth the cost. The SBO succeeds in part because it operates within a single contiguous sequence, preserving the full 512-token context for each training example while still providing an auxiliary training signal.

Finally, SpanBERT makes span-based pre-training a more attractive research direction while making certain alternatives less attractive. After this work, masking linguistically-informed spans without a boundary-conditioned objective appears less promising—the gains are inconsistent and the approach requires external NLP pipelines that introduce errors and limit domain generality. Similarly, auxiliary objectives that force context-window splitting (like NSP) are harder to justify when single-sequence training with a within-sequence auxiliary objective (SBO) provides consistent gains without the context-length penalty. The paper implicitly argues that future pre-training research should evaluate auxiliary objectives not only by their direct contribution to a loss function but also by their side effects on the data pipeline—does the objective require shorter sequences? does it introduce noise from unrelated text? does it constrain the masking scheme?

Follow-Up Research This Work Enables

Direct probing of boundary-token representations with and without SBO. The paper's central mechanistic claim—that SBO trains boundary tokens to encode internal span content—rests entirely on downstream performance evidence (Table 7: +2.7 F1 on coreference when SBO is added). No experiment directly measures whether x_{s-1} and x_{e+1} in an SBO-trained model actually contain more information about missing span tokens than in a model trained without SBO. A natural follow-up would train linear probes to predict masked span tokens from boundary representations alone, comparing SBO-trained vs. non-SBO-trained encoders at various pre-training checkpoints. If the SBO-trained encoder shows substantially higher probing accuracy—and if this accuracy correlates with downstream coreference performance—that would provide direct evidence for the proposed mechanism. A stronger version would use causal intervention: fine-tune the same coreference model but replace the span representation with one that uses only the left boundary, only the right boundary, or attention-pooled internal tokens, and measure whether SBO's benefit is concentrated in the boundary-token representations (as the mechanism predicts) or distributed across all representations (suggesting a general quality improvement rather than a boundary-specific one).

SpanBERT at smaller model scales and with reduced pre-training data. The paper evaluates only BERT_large_ (~340M parameters) on the full BooksCorpus + Wikipedia corpus. Two natural stress tests would characterize the generality of the method. First, train SpanBERT_base_ (12 layers, ~110M parameters) with the same objectives and compare against a BERT_base_ baseline on the same 17-benchmark suite. If the relative gains are proportionally similar (+2-3 F1 on SQuAD, +2-3 F1 on coreference), that would suggest the method is scale-invariant. If gains shrink substantially, that would suggest SBO requires sufficient representational capacity (deeper networks, larger hidden dimensions) to simultaneously satisfy MLM and SBO objectives without destructive interference, and would qualify the method as primarily beneficial at larger scales. Second, train both SpanBERT_large_ and BERT_large_ on subsets of the pre-training corpus (10%, 25%, 50%, 100%) and plot scaling curves. If SpanBERT's advantage widens with less data, that would be strong evidence for data efficiency—span-based objectives extract more useful signal per pre-training example. If the advantage narrows or disappears with less data, that would suggest span-based pre-training requires a critical mass of data to overcome the increased difficulty of the span-prediction task. The paper's claim that objective design provides gains "complementary to data and model scaling" (Section 1) is currently an inference; these experiments would test it directly.

Interaction of span masking distribution parameters with task characteristics. The paper uses a single geometric distribution with p = 0.2, clipped at ℓ_max = 10, yielding a mean span length of 3.8 (Figure 2). It reports testing p ∈ {0.1, 0.2, 0.4} and choosing 0.2 based on preliminary trials, but provides no quantitative ablation. The optimal span length distribution plausibly depends on the downstream task: short answers in SQuAD (mean ~3 words) might benefit from masking shorter spans, while coreference resolution (where entity mentions can be long) might benefit from longer spans. A systematic follow-up would train separate SpanBERT models with varying p (controlling mean span length) and ℓ_max, then evaluate each on SQuAD, coreference, and GLUE separately. If different tasks have different optimal span length distributions, that would motivate task-adaptive span masking during pre-training (sampling span lengths from a mixture calibrated to the expected downstream task distribution) or even multi-task pre-training objectives that apply different masking distributions to different examples. The paper's conclusion that "random geometric spans are as good as or better than linguistic spans" is currently limited to p = 0.2; understanding how sensitive this finding is to the distribution parameters would determine whether practitioners need to tune this hyperparameter for their target tasks or can use the paper's default.

Scaling the SBO architecture and exploring alternative formulations. The SBO uses a 2-layer feed-forward network with GeLU activations, layer normalization, and 200-dimensional learned relative position embeddings. The paper provides no ablation of these architectural choices. A follow-up could systematically vary: (a) network depth (1-layer vs. 2-layer vs. 3-layer), measuring whether deeper SBO networks produce better boundary representations at the cost of pre-training computation; (b) position embedding dimensionality (50, 100, 200, 400) and type (learned vs. sinusoidal), measuring impact on the model's ability to distinguish positions within long spans; (c) boundary representation combination method (concatenation vs. addition vs. attention-weighted combination of left and right boundaries), measuring whether both boundaries are necessary or one suffices. A more fundamental ablative question: what if the SBO predicted the span's tokens autoregressively (left-to-right or right-to-left) rather than independently in parallel? Independent prediction makes each token's prediction a bag-of-words problem given the boundaries; autoregressive prediction would force the model to also learn ordering within the span from boundary context, which might produce richer span representations. This connects SpanBERT to XLNet's autoregressive span prediction and MASS's seq2seq reconstruction, providing a controlled comparison of parallel vs. sequential span decoding for representation learning (as opposed to generation).

SpanBERT with larger pre-training corpora and longer training. The paper uses the standard BERT corpus (BooksCorpus + Wikipedia, ~16GB). Concurrent work (RoBERTa, Liu et al., 2019b) demonstrated that BERT's performance improves substantially with more data (160GB) and longer training. A critical open question is whether SpanBERT's advantages persist, grow, or shrink when both models are trained on much larger corpora and for more steps. If span-level understanding eventually emerges from token-level MLM given sufficient data—such that the marginal benefit of explicit span-based pre-training approaches zero—then SpanBERT's contribution is primarily about data efficiency (getting the same performance with less data) rather than about asymptotic capability (reaching a higher ceiling). If SpanBERT's advantage persists even at scale, that would suggest span-based pre-training teaches something that token-level MLM fundamentally cannot, regardless of data quantity. The experiment would train SpanBERT and BERT baseline on the RoBERTa corpus (160GB of text) for comparable total steps, controlling for optimization improvements that RoBERTa introduced, and measure the gap across the 17-benchmark suite. A null result (gap closes to zero) would be informative: it would imply that span-based pre-training is a training efficiency technique, not a capability-unlocking one.

SpanBERT for cross-lingual and multilingual pre-training. The paper evaluates only on English benchmarks, using English pre-training data. The core idea—that boundary tokens should summarize span content—is language-agnostic: every written language has multi-word expressions (named entities, compound words, idiomatic phrases) that benefit from span-level representation. However, the optimal span masking distribution may be language-dependent: languages with rich morphology and frequent multi-word compounds (e.g., German, Finnish, Turkish) might benefit from different mean span lengths or different masking granularity (subword vs. word vs. morpheme). A cross-lingual follow-up would replicate SpanBERT on a multilingual corpus (e.g., Wikipedia in 10+ languages) and evaluate on typologically diverse QA and NER benchmarks (XQuAD, WikiAnn). If SpanBERT's relative gains are consistent across languages, that establishes span-based pre-training as a universally beneficial inductive bias. If gains vary with morphological typology (larger for analytic languages like English, smaller for synthetic languages where subword tokenization already captures multi-morpheme units), that would refine our understanding of when and why span masking helps.

Practical Applications and Downstream Use Cases

Improved extractive question answering systems in production search and virtual assistants. SpanBERT's most robust and practically significant result is the +2.0 F1 improvement on SQuAD 1.1 (94.6% vs. 92.6% for the already-improved baseline) and the +2.9 average F1 improvement across five additional QA datasets (Table 2). In a production QA system—such as those powering web search answer boxes, voice assistant responses, or enterprise document search—a 2-3 F1 point improvement translates directly to more queries receiving correct answers on the first attempt. For a system handling millions of queries daily, a 2% reduction in answer errors is a meaningful user experience improvement. The deployment path is straightforward: replace the existing BERT encoder in the QA pipeline with a SpanBERT encoder, with no changes to the downstream architecture (the SBO module is discarded) and no increase in inference latency or model size. The gain is realized immediately at fine-tuning time and requires no additional runtime cost. The consistency of improvement across five diverse QA datasets (news, trivia, search, multi-hop, naturally-occurring questions) makes this a low-risk deployment decision: SpanBERT is unlikely to regress on any particular QA domain. The paper's result that SpanBERT reduces error by 27% over the tuned BERT baseline on SQuAD 1.1—a benchmark where models are near human performance—indicates that the method helps even on "easy" questions where BERT already performs well, not just on challenging edge cases.

Document-level coreference resolution as a preprocessing step for information extraction pipelines. SpanBERT achieves 79.6% F1 on OntoNotes coreference resolution, exceeding the prior state of the art by 6.6 absolute points (Table 3). Coreference resolution is a foundational preprocessing step for many downstream information extraction tasks: entity linking, relation extraction, event extraction, and document summarization all depend on knowing which mentions refer to the same entity. A 6.6-point improvement in coreference quality propagates through these pipelines—for example, a relation extraction system that uses coreference-resolved entity clusters as input will have cleaner, more complete entity information to work with. The specific mechanism that drives this gain (SBO training boundary tokens to be better span summaries, as evidenced by the +2.7 F1 gain from adding SBO in Table 7) is well-matched to the architecture of modern coreference systems, which use boundary-token representations to score mention pairs. A practitioner building an IE pipeline can adopt SpanBERT for the coreference step with high confidence: the gain is large, the mechanism is understood, and the implementation is a drop-in replacement for a BERT encoder.

Pre-training for domain-specific span selection tasks with limited labeled data. The paper demonstrates that span masking and SBO improve performance on span selection tasks without requiring any task-specific labeled data during pre-training—all gains come from the self-supervised objectives on unlabeled text. This makes SpanBERT particularly valuable for domains where labeled span-annotation data is scarce or expensive. A practitioner building a biomedical named entity recognition system, a legal document clause extractor, or a customer support answer extractor can pre-train SpanBERT on large amounts of unlabeled in-domain text (clinical notes, legal contracts, support tickets) using the same self-supervised objectives, then fine-tune on whatever small amount of labeled span data is available. The span-based pre-training inductive bias means the model arrives at fine-tuning time already equipped with representations that are structured for span extraction, reducing the amount of labeled data needed to achieve a given performance level. While the paper does not directly measure data efficiency of fine-tuning after SpanBERT pre-training, the mechanism (boundary tokens pre-trained to summarize spans) directly aligns with the fine-tuning task (predicting span boundaries from those same token representations), making improved sample efficiency a well-motivated expectation. The 2.7 F1 gain on coreference from SBO alone (Table 7)—a task where labeled data is notoriously expensive to produce—is a concrete datapoint supporting this use case.

Relation extraction and entity-centric NLP where entity span quality is the bottleneck. SpanBERT's 70.8% F1 on TACRED (Table 4) represents a +3.3 F1 improvement over the Our BERT baseline, with the gain primarily coming from improved recall (+3.7 points: 67.2 → 70.9). This pattern—better recall at similar precision—is consistent with the mechanism that SBO improves the quality of entity span representations, enabling the model to correctly identify relation instances that it would otherwise miss. In production relation extraction systems (e.g., constructing knowledge graphs from news articles, extracting drug-gene interactions from biomedical literature, or identifying supply-chain relationships from business reports), recall is often the more economically valuable metric: missing a critical relation can have higher downstream cost than surfacing a candidate that requires human verification. A practitioner deploying SpanBERT for relation extraction can expect the model to find more true relations without a proportional increase in false positives, directly increasing the yield of the extraction pipeline. The paper notes that SpanBERT approaches the state of the art (71.5 F1 from BERT_EM + MTB, which used additional entity-linked pre-training data) without requiring extra data, suggesting that span-based pre-training provides some of the same benefits as entity-specific pre-training but in a more general and data-efficient way.

When to Prefer This Method

The paper positions SpanBERT against BERT explicitly, and within that framing, the tradeoffs are clear:

Prefer SpanBERT over BERT when your downstream task involves explicit span selection (extractive QA, coreference resolution, slot filling, semantic role labeling). The empirical evidence is strongest here: +2.0-4.6 F1 on multiple QA benchmarks (Tables 1-2), +1.3 F1 on coreference over the tuned BERT baseline (Table 3). These gains come at zero inference-time cost, since the SBO module is discarded after pre-training and the model architecture is identical to BERT.
Prefer SpanBERT when your downstream task involves implicit span reasoning, even if the task output is not a span. The +6.9 accuracy improvement on RTE (Table 5) and the +1.3 on QNLI (SQuAD-derived NLI) suggest that tasks requiring the model to recognize relationships between multi-word expressions benefit from span-based pre-training, even when the output is a classification label. Relation extraction (TACRED, Table 4) also falls in this category, with +3.3 F1 improvement.
Prefer standard BERT when your downstream task is purely sentence-level classification without span structure (sentiment analysis, acceptability judgments). The gains on these tasks are negligible or slightly negative: 0.0 on SST-2, −0.3 F1 on MRPC, −0.2 F1 on QQP (Table 5). In these settings, the span-focused pre-training objectives add complexity to the pre-training pipeline without providing downstream benefit, and standard BERT (or RoBERTa, which further improves sentence-level performance) is the simpler choice.
When doing your own pre-training from scratch (rather than using a public checkpoint), prefer SpanBERT's single-sequence pipeline over BERT's bi-sequence + NSP pipeline regardless of the target task. The paper's findings on single-sequence training are unilateral: Our BERT-1seq outperforms Our BERT on 14 of 17 benchmarks, with gains up to +4.9 on CoLA (Table 5), and the authors provide a clear mechanistic hypothesis (longer contexts, less noise) that is consistent with the data. The only task where single-sequence training might not help is RTE (where the anomalous drop from 74.7 to 72.1 appears), but this single datapoint is insufficient to justify retaining the bi-sequence format given the weight of evidence on all other tasks.
Prefer random geometric span masking over linguistically-informed masking (named entities, noun phrases) when implementing a span-based pre-training scheme. Table 6 demonstrates that geometric spans are competitive with or superior to linguistic spans across all tested benchmarks, while being simpler to implement (no dependency on external NLP pipelines) and domain-agnostic (no need for language-specific parsers or NER systems). The single hyperparameter (p = 0.2, controlling the geometric distribution) can be tuned if needed, but the paper suggests the default works well across a range of tasks.

Caveat: These preference rules assume the same model architecture, corpus size, and training budget used in the paper (BERT_large_, ~340M parameters, BooksCorpus + Wikipedia, 2.4M steps). The paper does not test whether SpanBERT's advantages hold at different scales (larger corpora, longer training, different model sizes, different architectures), so these recommendations are specific to the regime in which the method was evaluated. A practitioner working with substantially more data (e.g., 160GB as in RoBERTa) or a different architecture (e.g., T5 encoder-decoder) should treat SpanBERT as a promising but unverified direction rather than a proven improvement.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: BERT's Token-Level Masking Is Mismatched to Span-Level Reasoning

Why This Matters: The Prevalence and Difficulty of Span Reasoning

Prior Approaches and Where They Fall Short

How SpanBERT Positions Itself

Summary of the Gap

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Span Masking Scheme

Span Boundary Objective (SBO)

Single-Sequence Training

Interaction Between the Three Components

4. Key Insights and Innovations

Innovation 1: Pre-training objectives should match the representational geometry of downstream tasks, not just the token-level statistics

Innovation 2: Random span masking is competitive with—and often superior to—linguistically-informed masking, despite being simpler

Innovation 3: The next-sentence prediction objective is harmful not because it's the wrong task, but because its associated data format corrupts the primary objective

Innovation 4: Controlled baseline construction reveals that pre-training objective design provides gains orthogonal to—and competitive with—data and model scaling

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Extractive Question Answering: SQuAD and MRQA

Coreference Resolution

Relation Extraction

GLUE

Overall Performance Summary

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

The SBO Mechanism's Benefits Are Inferred From Performance, Not Directly Verified

The Inability to Handle Hard Problems Where the Base Model Lacks Fundamental Capability

Difficulty Estimation Cost Is Not Accounted For in the Efficiency Narrative

Single Pre-Training Corpus and Model Architecture Leave Generalization Unverified

Fine-Tuning Variance and Small Test Sets Make Some Reported Gains Statistically Fragile

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

RoBERTa: A Robustly Optimized BERT Pretraining Approach