Longformer: The Long-Document Transformer

ArXiv: 2004.05150

🎯 Pitch

Transformers choke on long documents because their self-attention scales quadratically with sequence length. This paper introduces Longformer, a drop-in replacement that combines local windowed attention with sparse global attention to achieve linear scaling, enabling the first RoBERTa-level pretrained model to process 4,096 tokens directly. On WikiHop and TriviaQA, it demolishes prior state-of-the-art—simply removing global attention slashes accuracy by 8.3 points, proving that its hybrid attention pattern is what unlocks real long-document understanding.

1. Executive Summary

This paper introduces the Longformer, a modified Transformer architecture whose self-attention mechanism scales linearly with sequence length, enabling processing of documents containing thousands of tokens. Evaluated on character-level language modeling (text8, enwik8) and multiple downstream NLP tasks using a pretrained-then-finetuned paradigm built on RoBERTa, Longformer combines a local sliding window attention (each token attends to a fixed window of neighboring tokens) with task-motivated global attention (selected tokens, such as the [CLS] token for classification or all question tokens for QA, attend to the entire sequence). pretrained Longformer consistently outperforms RoBERTa on long-document tasks, achieving state-of-the-art results on WikiHop (+3.6 F1 over prior SOTA with Longformer-large) and TriviaQA (+4.0 F1), and the Longformer-Encoder-Decoder (LED) variant—which applies the efficient attention pattern to the encoder stack of a BART-initialized seq2seq model—sets new SOTA on the arXiv summarization dataset (46.63 ROUGE-1 with 16K input length). A core design finding is that both local and global attention are essential—ablations on WikiHop show that removing global attention drops accuracy by 8.3 points—establishing that the combination of windowed context building with sparse task-specific full-sequence attention is what enables effective long-document modeling without complex chunking architectures.

2. Context and Motivation

The fundamental problem this paper addresses is architectural: the standard Transformer self-attention operation has $O(n^2)$ time and memory complexity with respect to sequence length $n$ . This means that doubling the input length quadruples the computational cost. In practice, this makes it infeasible (or prohibitively expensive) to process sequences longer than a few hundred tokens on modern GPU hardware, with full self-attention literally running out of memory for long sequences (Figure 1). The authors frame this not as a minor inconvenience but as a structural barrier that prevents Transformers from being applied to an entire class of important NLP problems where the relevant context spans thousands of tokens.

This matters because many real-world NLP tasks are intrinsically long-document tasks. Consider the datasets the paper evaluates on (Table 6): WikiHop's supporting contexts average 1,535 wordpieces with a 95th percentile of 3,627; TriviaQA contexts average 6,589 wordpieces with a 95th percentile of 17,126; and arXiv summarization documents have a 90th percentile of 14,500 tokens. These are not edge cases — they are representative of document-level question answering, scientific summarization, multi-hop reasoning, and coreference resolution where the information needed to produce a correct answer is distributed across a document far longer than the 512-token window that BERT-style models can process in a single pass.

The practical consequence of the quadratic bottleneck is that practitioners are forced into lossy workarounds: they truncate documents (discarding potentially crucial information), split documents into chunks and process them independently (losing cross-chunk dependencies), or build complex multi-stage architectures where a retrieval step first selects a subset of the document and a reader step then extracts the answer from that subset. Each of these introduces failure modes — truncation drops information, chunking prevents the model from relating information in chunk A to information in chunk B, and retrieval-reader pipelines suffer from cascading errors where the retrieval step fails to surface the right context. The paper's core motivation is to eliminate the need for these workarounds entirely by making the attention mechanism itself scale linearly, so that a model can simply concatenate all available context and process it in a single pass.

Why This Problem Became Urgent

The paper was written at a specific inflection point in the evolution of pretrained Transformers. By early 2020, BERT and RoBERTa had established the pretrain-then-finetune paradigm as dominant across NLP, but the 512-token limit was baked into their architectures through the learned absolute position embeddings and the quadratic attention cost. This created an awkward situation: the pretraining corpora (Books, Wikipedia, news) contained documents much longer than 512 tokens, but the models could only learn from truncated, localized views of those documents. The authors of RoBERTa had shown that training on longer sequences was beneficial but computationally prohibitive — RoBERTa used sequence lengths of 512, and scaling beyond that required either enormous computational resources or architectural changes.

Two converging trends made this bottleneck increasingly costly. First, the NLP community was tackling more ambitious tasks that required reasoning over entire documents rather than single paragraphs — multi-hop QA, long-document summarization, and scientific literature understanding all demand models that can integrate evidence across thousands of tokens. Second, GPU memory was growing (the paper uses 48GB RTX8000s and 32GB V100s), meaning that the hardware was capable of processing longer sequences if only the attention mechanism didn't consume all the memory. The gap between what the hardware could support (in terms of raw memory for activations) and what the $O(n^2)$ attention allowed was widening, making this a ripe moment for an architectural solution.

Prior Approaches and Where They Fall Short

The paper organizes prior work on long-sequence Transformers into two categories, each with distinct limitations (Table 1, Section 2):

Left-to-right autoregressive models. These models process documents sequentially, attending to a moving window of past tokens. Transformer-XL (Dai et al., 2019) introduced a memory mechanism that caches hidden states from previous segments; Adaptive Span Transformers (Sukhbaatar et al., 2019) learned per-head attention spans; Compressive Transformers (Rae et al., 2020) added a compressed memory of older activations. These models achieved strong results on character-level language modeling benchmarks (text8, enwik8), and the paper acknowledges their success. However, the authors identify a critical limitation: they are unsuitable for the pretrain-finetune transfer learning paradigm with tasks that require bidirectional context. Language modeling is left-to-right by construction, but tasks like question answering require attending to both the question and the answer context simultaneously, across the full document. A left-to-right model cannot, for instance, compare a question at position 50 with a supporting fact at position 5,000 in a bidirectional way — it can only condition the fact on the question, not the question on the fact. Since BERT-style bidirectional pretraining followed by task finetuning was (and remains) the dominant paradigm for NLP tasks, a solution that only works for autoregressive LMs leaves most practical applications unaddressed.

Sparse attention patterns. The other general approach, to which Longformer belongs, replaces the full $n \times n$ attention matrix with some form of sparse pattern that avoids computing all $n^2$ pairwise interactions. The paper's closest predecessors here are Sparse Transformer (Child et al., 2019), which used fixed sparse patterns including dilated sliding windows of $8 \times 8$ blocks implemented via BlockSparse (Gray et al., 2017); Reformer (Kitaev et al., 2020), which used locality-sensitive hashing to select which keys each query attends to; and Routing Transformer (Roy et al., 2020), which learned to route attention via clustering. These models demonstrated that sparse attention could match or approach full attention performance on language modeling benchmarks.

However, the paper identifies several gaps in this prior work. First and most critically, almost none of these models had been applied to the pretrain-finetune paradigm (Table 1, last column): Transformer-XL, Adaptive Span, Compressive, Reformer, Sparse, and Routing Transformers were all evaluated exclusively or primarily on autoregressive language modeling. The paper argues this is a significant limitation because language modeling as a primary evaluation has "led to the development of models with limited applicability." A model that excels at predicting the next character in a Wikipedia article may not be usable for document-level question answering because the sparse pattern that works for left-to-right prediction doesn't naturally support the bidirectional, task-specific attention patterns that finetuned models need.

Second, even the few models that did explore non-LM tasks were limited in their evaluation. BP-Transformer (Ye et al., 2019) was tested on machine translation, but didn't explore the pretrain-finetune setting — it trained from scratch for the specific task. Blockwise attention (Qiu et al., 2019) did pretrain models and evaluated on QA, but the datasets were relatively short (SQuAD contexts typically fit within 512 tokens, and MRQA was constructed by filtering out long-document examples), so the evaluation didn't actually test whether the model could handle truly long documents. The paper thus positions existing sparse attention work as having demonstrated feasibility on language modeling but having left the harder, more practical question unanswered: can a sparse attention mechanism replace full self-attention in a pretrained model and improve performance on real long-document downstream tasks?

Third, the paper notes a practical implementation gap. Sparse Transformer relied on BlockSparse, which the authors describe as implemented in C++ and designed for a specific version of TensorFlow — inflexible and difficult to adapt to new patterns or frameworks. Part of Longformer's contribution is a more flexible implementation strategy with three variants (loop, chunked, and CUDA kernel via TVM) that support different use cases and are integrated into PyTorch.

Task-Specific Workarounds and Their Failure Modes

Beyond the architectural prior work, the paper is motivated by the proliferation of complex task-specific architectures that exist solely to work around BERT's 512-token limit. The paper describes three common patterns:

Truncation is the simplest: just cut the document at 512 tokens and discard the rest. This is common for classification tasks (Xie et al., 2019) but obviously loses any information beyond the truncation point. For a scientific article summarization task where the key findings might be in the last section, truncation is catastrophic.

Chunking splits the document into (possibly overlapping) segments of ≤512 tokens, processes each segment independently through BERT, and then combines the resulting activations using some task-specific aggregation model. This is the approach used by Joshi et al. (2019) for coreference resolution and by many QA systems. The problem is that cross-chunk interactions are lost: if a pronoun in chunk 3 refers to an entity mentioned in chunk 1, the model can never learn that connection within the transformer layers themselves. The aggregation model (which operates on BERT's output representations, not the raw text) must reconstruct these long-range dependencies from compressed, context-free representations, which is fundamentally harder than allowing the attention mechanism to connect them directly.

Two-stage retrieval-reader pipelines are popular for multi-hop and open-domain QA. The first stage retrieves relevant documents or paragraphs; the second stage extracts the answer from the retrieved set (Clark and Gardner, 2017; Chen et al., 2017). The critical weakness is cascading errors: if the retrieval stage fails to surface a crucial supporting document, the reader stage can never produce the correct answer, no matter how capable it is. This is particularly problematic for tasks like WikiHop, where the reasoning chain requires connecting information across multiple documents and there is no intermediate supervision for which documents are relevant — the model must discover the reasoning chain from only the final answer label.

The paper's position is that all of these workarounds exist because of the attention bottleneck, not because they are the right way to solve the tasks. If a Transformer could process 4,096 or 16,384 tokens in a single pass, the complex chunking and retrieval architectures could be replaced with a simple end-to-end model that concatenates everything and processes it together. This is exactly what Longformer enables: "our proposed Longformer is able to build contextual representations of the entire context using multiple layers of attention, reducing the need for task-specific architectures."

How Longformer Positions Itself

Longformer is not the first sparse attention mechanism — the paper is explicit that it builds on ideas from Sparse Transformer, dilated CNNs (van den Oord et al., 2016), and the broader literature on efficient attention. Its positioning is differentiated along four axes:

1. A drop-in replacement for pretrained models. The key design decision is that Longformer's attention pattern can replace the standard self-attention in an already-pretrained model like RoBERTa without changing the model architecture. The paper demonstrates this by initializing from the RoBERTa released checkpoint, extending the position embeddings, and continuing MLM pretraining for only 65K steps. This pragmatic approach means the model inherits all of RoBERTa's pretrained knowledge while gaining the ability to process longer sequences, rather than requiring expensive pretraining from scratch.

2. Task-motivated global attention as a first-class design element. Where prior sparse attention work focused on finding the right fixed or learned sparse pattern for language modeling, Longformer introduces the idea that certain tokens should receive full global attention based on the task, and that this is not a hack but an inductive bias that dramatically improves performance. The authors show that separate linear projections for local vs. global attention ( $Q_s, K_s, V_s$ vs. $Q_g, K_g, V_g$ ) are critical — using shared projections drops WikiHop accuracy by 1.6 points (Table 10). This design choice reflects an insight that local pattern matching (sliding window) and global information aggregation (task-specific attention) are fundamentally different operations that benefit from different parameterizations.

3. Evaluation across the full pretrain-finetune pipeline. This is the paper's most significant departure from prior work. Where Sparse Transformer, Reformer, Routing Transformer, and others focused primarily on language modeling, Longformer is positioned as a model that can be pretrained with MLM, then finetuned on a diverse set of downstream tasks — QA, coreference resolution, document classification — exactly as BERT and RoBERTa are used. The paper explicitly states this as addressing a gap: "the application of long document transformers to document-level NLP tasks in the transfer learning setting has remained largely unexplored."

4. Extension to encoder-decoder architectures. The LED model extends the efficient attention to sequence-to-sequence tasks, recognizing that summarization and translation have long-document inputs that suffer from the same quadratic bottleneck. By initializing from BART and only replacing the encoder's self-attention, LED inherits pretrained sequence-to-sequence capabilities while scaling to 16K input tokens, achieving SOTA on arXiv summarization without any task-specific pretraining (Table 11).

Contemporaneous and Subsequent Work

The paper acknowledges several contemporaneous works that appeared on arXiv after Longformer, notably ETC (Ainslie et al., 2020), GMAT (Gupta and Berant, 2020), and BigBird (Zaheer et al., 2020), all of which explore similar local + global attention patterns. The paper positions Longformer as the originator of this idea in the pretrain-finetune context while acknowledging that BigBird later improved leaderboard results with 16× more pretraining compute. Importantly, BigBird's theoretical contribution — proving that sparse Transformers are universal approximators of sequence functions — provides post-hoc theoretical grounding for the empirical effectiveness Longformer demonstrates, reinforcing that the sparse attention approach is not merely a heuristic but preserves the representational power of full attention.

3. Technical Approach

3.1 Reader orientation

Longformer is a modified Transformer where each token, instead of looking at every other token in the sequence (which gets quadratically expensive), only looks at a small local neighborhood plus a handful of specially-designated "global" tokens. The system solves the problem of processing documents that are thousands of tokens long — enabling question answering over entire Wikipedia articles, summarization of full scientific papers, or classification of long reviews — without the lossy workarounds (truncation, chunking, two-stage retrieval) that the standard $O(n^2)$ self-attention forces upon practitioners.

3.2 Big-picture architecture (diagram in words)

The Longformer system has five major interacting design elements:

Sliding window attention — every token attends to a fixed-size band of neighboring tokens ( $w/2$ on each side), giving $O(n \times w)$ complexity instead of $O(n^2)$ .
Dilated sliding window attention — the sliding window is "stretched" with gaps of size $d$ to increase the receptive field without increasing computation, analogous to dilated convolutions.
Global attention on task-specific tokens — a small, fixed set of tokens (e.g., [CLS] for classification, all question tokens for QA) attend to every token in the sequence, and every token attends back to them. This is a symmetric operation.
Separate linear projections for local vs. global attention — the model uses two independent sets of $Q, K, V$ matrices, one for the sliding window and one for global attention, initialized identically but allowed to diverge during training.
Longformer-Encoder-Decoder (LED) — a seq2seq variant where the encoder uses Longformer's efficient attention pattern (local + global) while the decoder uses full self-attention to the encoded representation and to previously decoded positions.

Information flows through the system as follows: an input sequence is tokenized into wordpieces → position embeddings (extended beyond RoBERTa's 512 via copying) are added → the sequence passes through multiple transformer layers, each computing attention using only the specified sparse pattern (windowed + dilated + global) rather than the full $n \times n$ matrix → the resulting contextual representations at the top layer are used for task-specific prediction heads, exactly as in standard BERT/RoBERTa finetuning. For autoregressive language modeling, the pattern is further restricted to prevent attending to future tokens (left-to-right masking within the sliding window).

3.3 Roadmap for the deep dive

First, the sliding window attention mechanism — the core $O(n \times w)$ operation that replaces full self-attention, including how multiple layers stack to achieve a large receptive field and how dilated windows extend it further.
Second, global attention — what it is, why task-specific tokens need it, the symmetry property (attending to and from global tokens), and the separate linear projections that parameterize local versus global attention differently.
Third, the autoregressive language modeling configuration — the staged training procedure, the per-layer varying window sizes, the dilation schedule across heads, and the relative position embeddings, since these design choices differ substantially from the pretrain-finetune setting.
Fourth, the implementation strategies (loop, chunked, CUDA kernel) — what each one computes, their memory and speed tradeoffs, and why three implementations exist for different use cases.
Fifth, the pretraining pipeline — how Longformer initializes from RoBERTa's checkpoint, extends position embeddings via copying, and continues MLM pretraining on a corpus of long documents, since this is what enables the drop-in replacement property.
Sixth, the Longformer-Encoder-Decoder (LED) — how the efficient attention pattern is applied to the encoder of a BART-initialized seq2seq model, enabling long-input summarization without any additional pretraining.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems and architecture paper whose core contribution is an attention mechanism that scales linearly with sequence length, combined with the engineering and evaluation necessary to demonstrate it as a drop-in replacement for full self-attention in pretrained Transformers.

Sliding Window Attention

The standard Transformer computes attention scores between every pair of positions in the input sequence. For an input of length $n$ , this requires computing and storing an $n \times n$ attention matrix. The core insight of Longformer is that most of these pairwise interactions are unnecessary — local context dominates in practice, and long-range dependencies can be routed through a small number of designated global tokens.

The sliding window attention restricts each token to attending to only a fixed window of neighboring tokens. Formally, for a window size $w$ , each token attends to exactly $\frac{w}{2}$ tokens on its left and $\frac{w}{2}$ tokens on its right (in the bidirectional case), plus itself. The total number of attention computations per token is therefore $w + 1$ , and the total complexity across the sequence is $O(n \times w)$ , which scales linearly with $n$ since $w$ is a constant independent of sequence length.

Figure 2b in the paper illustrates this pattern: the full $n \times n$ attention matrix of standard self-attention (Figure 2a) is replaced by a banded matrix where the only non-zero entries fall within a diagonal band of width $w$ centered on the main diagonal. Everything outside this band is zero — those attention scores are never computed, stored, or used.

The receptive field argument. A natural objection is: if each token only sees its immediate neighbors, how can the model capture long-range dependencies? The answer, which the paper makes explicit, is that stacking multiple layers of windowed attention expands the receptive field. In a transformer with $\ell$ layers, each using a window of size $w$ , a token at the top layer can indirectly receive information from tokens up to $\ell \times w$ positions away (assuming $w$ is fixed across layers). This is exactly analogous to how stacking convolutional layers in a CNN expands the receptive field — a pixel in the output of a 3-layer CNN with $3 \times 3$ kernels "sees" a $7 \times 7$ patch of the input. For Longformer, with $\ell = 12$ layers and $w = 512$ , the theoretical receptive field at the top layer is $12 \times 512 = 6,144$ tokens, which exceeds the 4,096 sequence length used in pretraining.

The paper cites Kovaleva et al. (2019) as prior evidence that local context is especially important in BERT's attention heads, providing empirical grounding for the design choice to prioritize local attention over other sparse patterns.

Window size configuration across layers. The paper does not use a uniform window size for all layers. In the autoregressive language modeling setting (Section 4.1), the authors explicitly vary window sizes:

"we use small window sizes for the lower layers and increase window sizes as we move to higher layers. This allows the top layers to learn higher-level representation of the entire sequence while having the lower layers capture local information."

For the small character-level LM model (12 layers), the phase 1 configuration uses window sizes ranging from 32 at the bottom layer to 8,192 at the top layer. By phase 5, the range has shifted to 512 at the bottom layer to 23,040 at the top (the GPU memory limit). The ablation in Table 4 confirms this choice: an increasing window size schedule achieves 1.21 BPC on text8, compared to 1.24 for the decreasing schedule (512→32) and 1.23 for a fixed window size of 230 (the average of the other configurations). The authors hypothesize that lower layers benefit from focusing on immediate local patterns (character n-grams, short-range syntax) while upper layers need the wider context to integrate information across the document.

For the pretrain-finetune setting (Section 5), the sliding window size is fixed at 512 for all layers. This choice matches RoBERTa's sequence length, meaning each token in Longformer attends to the same number of other tokens as a token in RoBERTa would (512 on each side in the full-attention case, vs. 512 total in the windowed case). The per-token computation is thus roughly comparable, making the comparison fair: Longformer processes sequences 8× longer (4,096 tokens) with approximately the same per-layer FLOPs as RoBERTa processing 512-token sequences with full attention.

Why not other sparse patterns? The paper considers sliding windows specifically because they are:

Structured: the pattern is regular and can be implemented efficiently as a banded matrix multiplication.
Locality-preserving: they encode the inductive bias that nearby tokens are more relevant than distant ones, which is empirically well-supported for language (Clark et al., 2019).
Composable with global attention: the window can easily coexist with a small number of full-attention rows and columns without changing the overall complexity class.

Alternative sparse patterns considered in prior work — random patterns (Sparse Transformer), learned patterns (Routing Transformer), or hash-based patterns (Reformer) — either require more complex implementations, don't compose naturally with task-specific global tokens, or have less predictable memory access patterns on GPUs.

Dilated Sliding Window Attention

Even with stacking, the receptive field of a pure sliding window grows only linearly with the number of layers ( $\ell \times w$ ). To achieve very large receptive fields without increasing $w$ (which would increase computation proportionally), the paper introduces dilation, directly borrowed from dilated convolutions in WaveNet (van den Oord et al., 2016).

A dilated sliding window with dilation $d$ means that instead of attending to the $w$ immediately adjacent tokens, the attention skips every $d$ tokens. Formally, a token at position $i$ attends to tokens at positions $i - d \cdot \frac{w}{2}, i - d \cdot (\frac{w}{2} - 1), \ldots, i - d, i, i + d, \ldots, i + d \cdot \frac{w}{2}$ . The total number of attention computations per token remains $w + 1$ , so the computational complexity is unchanged, but the spatial extent of the window is multiplied by $d$ . The effective receptive field in a model with $\ell$ layers, window $w$ , and dilation $d$ (assumed uniform) becomes $\ell \times d \times w$ .

Figure 2c illustrates this pattern: the attention matrix has the same number of non-zero elements as the undilated sliding window, but those elements are spread across a wider band, creating visible gaps in the diagonal pattern.

Head-specific dilation. The paper does not apply dilation uniformly across all attention heads. Instead, they use a mixed configuration where some heads have no dilation (focusing purely on local context) while others have increasing dilation values. For the small character-LM model (12 layers, 8 heads), the configuration is:

"dilation 0 on layers 0–5, dilation 1 on layers 6–7, dilation 2 on layers 8–9, dilation 3 on layers 10–11, and only on 2 heads" (Table 12)

This means that in layers 0–5, all 8 heads use undilated windows (capturing local patterns). In layers 6–7, 2 heads use dilation 1 (attending to every other token in the window) while the other 6 remain undilated. At the top layers, 2 heads use dilation 3 (attending to every fourth token), dramatically expanding their spatial coverage while the remaining heads maintain fine-grained local resolution. The large model (30 layers) follows a similar schedule with dilation values 0, 1, 2, 3 distributed across layers 0–14, 15–19, 20–24, and 25–29 respectively.

The ablation in Table 4 (bottom) shows the benefit: adding dilation on 2 heads improves BPC from 1.21 (no dilation) to 1.20, a small but consistent gain. The paper's intuition is that "allowing some heads without dilation to focus on local context, while others with dilation focus on longer context" provides a form of multi-scale representation that a uniform pattern cannot achieve.

Why not use dilation in pretrain-finetune? The paper explicitly notes in a footnote to Section 5 that "adding dilation on a few heads as in §4.1 hurt performance, likely because it is not compatible with the pretrained RoBERTa weights." This is a crucial practical detail: when initializing from a pretrained checkpoint that was trained with full self-attention, the attention heads have learned to expect a contiguous local neighborhood. Introducing dilation introduces gaps in that neighborhood that the pretrained weights are not adapted to handle. Retraining such a model from scratch might allow dilation to be beneficial, but continuing from RoBERTa's checkpoint makes it counterproductive. This highlights a tension between architectural innovation and the practical constraints of transfer learning from existing pretrained models.

Global Attention

The sliding window (even with dilation) has a fundamental limitation: it builds representations through a chain of local interactions, which means that information must propagate step-by-step through the network to travel from one end of a long document to the other. For tasks like classification (which needs a single representation of the entire document at the [CLS] token) or question answering (which needs to compare every word in the question with every word in the document), this indirect propagation is insufficient. The paper states this explicitly:

"the windowed and dilated attention are not flexible enough to learn task-specific representations"

Global attention addresses this by designating a small number of input positions as "global" tokens that participate in full self-attention. Specifically:

A global token attends to every token in the entire sequence (its query vector is compared against every key).
Every token in the sequence attends to every global token (every query is compared against the global tokens' keys).

This is a symmetric operation: the global token's row and column in the attention matrix are both fully populated. Figure 2d illustrates this: a few rows and columns of the attention matrix are completely filled (global attention), while the rest follows the sliding window pattern.

Which tokens get global attention? The choice is task-specific and designed to encode inductive bias about what the model needs to compare:

Classification (IMDB, Hyperpartisan): Global attention on the [CLS] token. This allows the classification token to directly aggregate information from the entire document in a single attention operation, rather than having to wait for information to propagate through multiple layers of local windows.
QA (TriviaQA, HotpotQA): Global attention on all question tokens. This allows every word in the document to directly attend to every word in the question, enabling the fine-grained comparison needed to find answer spans. The paper specifies: "we use global attention on all question tokens" for TriviaQA, and "global attention to question tokens, paragraph title start tokens as well as sentence tokens" for HotpotQA.
WikiHop: Global attention on the entire question and answer candidate sequence. This allows the model to compare each candidate against the full supporting context simultaneously.
Coreference resolution (OntoNotes): No global attention is used. The paper found it unnecessary because "the distance between any two mentions is typically quite small" — the local window already covers the relevant context for coreference decisions.
Summarization (LED): Global attention on the first <s> token in the encoder, analogous to the [CLS] role.

Complexity analysis. Since the number of global tokens $g$ is small (typically a few hundred at most, compared to the sequence length $n$ of thousands), the combined complexity remains $O(n)$ : the sliding window contributes $O(n \times w)$ , and the global attention contributes $O(n \times g)$ , which is linear in $n$ when $g$ is a constant independent of sequence length.

Why global attention is not just a hack but essential. The paper's ablation on WikiHop (Table 10) provides the strongest evidence: removing global attention while keeping the separate linear projections drops accuracy from 72.2 to 65.5 — a catastrophic 8.3-point drop (line "Longformer (no linear proj. no global atten.)"). The model without global attention essentially cannot perform the task, because it has no mechanism to compare the question with distant supporting facts in a single step. This validates the paper's central architectural claim: local attention provides the substrate for building contextual representations across the document, but global attention on task-critical tokens is what enables the model to actually use those representations for prediction.

Separate Linear Projections for Local vs. Global Attention

The standard Transformer computes attention using learned projections $Q, K, V$ (query, key, value) that are shared across all attention computations in a given layer. Longformer departs from this by using two independent sets of projections: $Q_s, K_s, V_s$ for the sliding window attention, and $Q_g, K_g, V_g$ for global attention. The attention computation then becomes:

For sliding window attention (standard scaled dot-product, but only over local neighbors):

$\text{Attention}_{\text{local}}(Q_s, K_s, V_s) = \text{softmax}\left(\frac{Q_s K_s^T}{\sqrt{d_k}} + \text{mask}_{\text{local}}\right) V_s$

where $\text{mask}_{\text{local}}$ sets attention scores outside the sliding window to $-\infty$ (before softmax).

For global attention (applied to the designated global tokens):

$\text{Attention}_{\text{global}}(Q_g, K_g, V_g) = \text{softmax}\left(\frac{Q_g K_g^T}{\sqrt{d_k}}\right) V_g$

where no masking is applied — global tokens attend to everything.

The key detail is that both sets of projections are initialized identically:

" $Q_g, K_g, V_g$ are all initialized with values that match $Q_s, K_s, V_s$ "

This means at the start of training (or when continuing from RoBERTa's checkpoint), the model behaves as if the projections are shared. During training, they can diverge, allowing the model to learn that attending locally (pattern matching within a neighborhood) and attending globally (aggregating information across the entire sequence) are fundamentally different operations that benefit from different parameterizations.

Why separate projections matter. The ablation in Table 10 (line "Longformer (no linear proj.)") shows that removing the separate projections (i.e., using shared $Q, K, V$ for both local and global attention) drops WikiHop accuracy from 73.8 to 72.2 — a 1.6-point decrease. The paper interprets this as evidence that "the additional projections provide flexibility to model the different types of attention, which we show is critical for best performance on downstream tasks." Intuitively, a global attention head needs to learn to pick out relevant information from anywhere in the document (a "search" operation), while a local attention head needs to learn to compose information from adjacent tokens (a "syntactic" or "local coherence" operation). These are different enough functions that sharing parameters forces a compromise that hurts both.

Autoregressive Language Modeling Configuration

The autoregressive (left-to-right) language modeling setting requires several modifications to the attention pattern beyond what is used in the pretrain-finetune setting, primarily because the model can only attend to previous tokens (causal masking) and because the goal is to maximize the receptive field for next-character prediction.

Causal masking within sliding windows. In the autoregressive setting, each token attends only to the $\frac{w}{2}$ tokens to its left (prior positions), not to its right (future positions). The paper states that the CUDA kernel "supports the autoregressive mode where each token attends to a window of previous tokens only." This is implemented as an additional mask within the banded matrix multiplication, zeroing out the upper triangle of the attention band.

Staged training procedure. Training directly on the full sequence length and window size is inefficient because the model "needs a large number of gradient updates to learn the local context first, before learning to utilize longer context." The paper adopts a 5-phase training procedure where each phase doubles the window size and sequence length while halving the learning rate:

"in the first phase we start with a short sequence length and window size, then on each subsequent phase, we double the window size and the sequence length, and halve the learning rate"

Concretely (Table 12):

Phase 1: sequence length 2,048, window sizes 32→8,192 (bottom→top), LR 0.00025, 430K steps
Phase 2: sequence length 4,096, windows doubled, LR 0.000125, 50K steps
Phase 3: sequence length 8,192, windows doubled, LR 0.0000625, 50K steps
Phase 4: sequence length 16,384, windows doubled, LR 0.00003125, 35K steps
Phase 5: sequence length 23,040 (GPU memory limit), windows 512→23,040, LR 0.000015625, 5K steps

The intuition is that early phases teach the model local character-level patterns (which are abundant and require many updates), while later phases teach the model to integrate long-range context (which requires fewer updates because the local patterns are already learned).

Relative position embeddings with sinusoidal weights. Unlike the pretrain-finetune setting which uses learned absolute position embeddings (inherited from RoBERTa and extended by copying), the autoregressive LM uses relative position embeddings with sinusoidal weights, following Transformer-XL (Dai et al., 2019). The paper notes that their implementation "also includes a version of the relative position embedding that is compatible with our dilated sliding window attention." This is a non-trivial adaptation because relative position embeddings encode the distance between pairs of tokens, and dilation changes the effective distance — two tokens that are $d \times k$ positions apart in the input are $k$ steps apart in the dilated window. The relative position embedding must account for this to correctly represent the spatial relationship.

Evaluation protocol. At evaluation time, the model processes sequences of length 32,256 (longer than any training sequence). Following Dai et al. (2019), the dataset is split into overlapping sequences of size 32,256 with a step size of 512, and performance is reported only on the last 512 tokens of each sequence. The overlapping ensures that every token in the dataset is evaluated with a full 32,256-token context, avoiding boundary effects where the first few tokens of a sequence would have less context.

Mixed precision and numerical stability. Training uses mixed precision (fp16 and fp32) via NVIDIA's apex library to reduce memory and speed up computation, but the attention computation itself is kept in fp32:

"we kept the attention computation in fp32 to avoid numerical instability issues. We found that using fp16 in attention operation results in floating point overflow and NaNs in later stages of training."

This is an important practical detail: the attention softmax involves exponentiating potentially large values, which is numerically sensitive in half precision. The paper also uses gradient checkpointing (Chen et al., 2016) to reduce memory usage, trading computation for memory by recomputing intermediate activations during the backward pass rather than storing them.

Implementation Strategies

The paper describes three different implementations of the banded matrix multiplication required for sliding window attention, each with different tradeoffs (Figure 1, Appendix A). The existence of three implementations — and the explicit discussion of their performance characteristics — is itself a contribution, because sparse attention patterns that are theoretically efficient can be practically unusable if they cannot be implemented to leverage GPU parallelism.

Longformer-loop. This is the conceptually simplest implementation: compute each diagonal of the banded matrix separately in a Python loop. For each diagonal offset, extract the corresponding slices of $Q$ and $K$ , compute their dot product, and write the result to the appropriate diagonal of the output.

The paper states this implementation is "memory efficient because it only computes the non-zero values, but it is unusably slow" — Figure 1 shows it as the slowest by a large margin. It is used only for testing correctness because of its simplicity, never for actual experiments.

Longformer-chunks. This implementation works only for the non-dilated case ( $d=1$ ). It chunks $Q$ and $K$ into overlapping blocks of size $w$ with an overlap of size $\frac{w}{2}$ between consecutive blocks. Each block of $Q$ is multiplied with the corresponding block of $K$ (using PyTorch's highly optimized matrix multiplication), and the resulting block matrix is then masked to keep only the banded (non-dilated) pattern.

The paper describes this as "very compute efficient because it uses a single matrix multiplication operation from PyTorch, but it consumes 2x the amount of memory a perfectly optimized implementation should consume because it computes some of the zero values." The 2× overhead comes from the overlap: without overlap, the chunks would be disjoint and miss cross-chunk interactions; with overlap, each token participates in two chunks, so some computations are duplicated. Despite this overhead, the chunked implementation is fast enough and the memory overhead is manageable for the pretrain-finetune setting, where it is the default.

Longformer-cuda. This is a custom CUDA kernel implemented using TVM (Chen et al., 2018), a deep learning compiler that generates optimized GPU code from high-level Python descriptions. It supports the full attention pattern — dilated sliding windows, global attention, and autoregressive masking — and is the most memory-efficient implementation because it "only computes the non-zero values."

The paper describes its performance as "as fast as the highly optimized full self-attention," which is notable because it means the linear-scaling attention achieves its theoretical efficiency advantage in practice, not just in asymptotic analysis. Figure 1 confirms this: Longformer-cuda's runtime scales linearly with sequence length, while full self-attention's runtime grows quadratically and eventually runs out of memory.

The CUDA kernel is "mainly used for the autoregressive language modeling experiments because of the memory efficiency (allows the longest sequences) and the support of dilation (needed for character-LM experiments)."

TVM usage detail (Appendix A): The authors describe their process as writing "high-level python constructs" that specify the banded matrix multiplication pattern, which TVM then compiles into CUDA code and optimizes for the target GPU. The paper notes that "achieving this level of performance requires special knowledge of low-level GPU programming, similar to implementing a highly optimized matrix multiplication," and that their current implementation is "sufficiently fast and practical to use" but not theoretically optimal — a perfectly optimized kernel could be even faster than full self-attention because it computes only a fraction of the operations.

Pretraining Pipeline (Continuing from RoBERTa)

The paper does not train Longformer from scratch. Instead, it continues pretraining from the publicly released RoBERTa checkpoint, making only the minimal changes needed to support the new attention mechanism. This is a deliberate design choice driven by the expense of MLM pretraining and by the desire to demonstrate that Longformer is truly a drop-in replacement.

Position embedding extension. RoBERTa uses learned absolute position embeddings with a maximum sequence length of 512. To support sequences of 4,096 tokens, the position embedding matrix needs to be expanded from $512 \times d$ to $4,096 \times d$ (where $d$ is the hidden size — 768 for base, 1,024 for large). The new rows (positions 512–4,095) need to be initialized.

The naive approach would be random initialization, but the paper uses a clever initialization scheme: copy the existing 512 position embeddings multiple times. That is, position $i$ (for $i \geq 512$ ) uses the embedding from position $i \bmod 512$ . The paper justifies this with prior analysis of BERT's attention heads (Clark et al., 2019), which "shows a strong learned bias to attending to local context, including the previous or next token." By copying, the model preserves the local structure of the position embeddings: tokens at positions 512 and 513 have the same relative embedding structure as tokens at positions 0 and 1, ensuring that the sliding window attention "sees" the same local positional relationships regardless of absolute position.

Table 5 demonstrates the effectiveness of this approach. With randomly initialized position embeddings, Longformer-base achieves a BPC of 10.299 on the MLM development set — far worse than RoBERTa's 1.846, indicating the model cannot effectively use the longer context. With the copy initialization, BPC drops to 1.957, nearly matching RoBERTa's 1.846 even before any additional pretraining. This confirms that the copied embeddings preserve the local positional information that the pretrained RoBERTa weights rely on.

Attention pattern for pretraining. The pretraining uses sliding window attention with a fixed window size of 512 for all layers, and global attention is not used during MLM pretraining (it is added only during finetuning for specific tasks). The window size of 512 is chosen to match RoBERTa's sequence length, meaning "the same amount of computation as RoBERTa" is used per token, but distributed over an 8× longer sequence.

The paper notes that adding dilation "hurt performance, likely because it is not compatible with the pretrained RoBERTa weights" — the attention heads trained with full self-attention expect contiguous neighborhoods, and dilation creates gaps that the pretrained weights cannot interpret.

Continued MLM pretraining details. Training is done using fairseq (Ott et al., 2019) on a corpus of long documents compiled by the authors (Appendix C, Table 13). The corpus includes:

Source	Tokens	Average doc length
Books (Zhu et al., 2015)	0.5B	95.9K
English Wikipedia	2.1B	506
Realnews (Zellers et al., 2019)	1.8B	1.7K
Stories (Trinh and Le, 2018)	2.1B	7.8K

The Realnews subset includes only documents longer than 1,200 tokens, and only one third of the available data is used (similarly for Stories). The authors state their goal was to "include a mix of long and short documents to both allow the model to learn longer dependencies while not to forget information from the original RoBERTa pretraining."

Training hyperparameters for continued pretraining:

Sequences length: 4,096 tokens
Batch size: 64 sequences (effectively $2^{18}$ tokens)
Maximum learning rate: $3 \times 10^{-5}$
Learning rate schedule: linear warmup for 500 steps, followed by polynomial decay with power 3
Total training steps: 65,000
Optimizer, weight decay, dropout: same as RoBERTa (AdamW, not explicitly stated but implied by "the rest of the hyperparameters are the same as RoBERTa")

Table 5 tracks the BPC improvement during pretraining. For Longformer-base, BPC drops from 1.957 (at initialization, copy position embeddings, no training) to 1.753 after 2,000 gradient updates, to 1.705 after 65,000 updates. The 0.252 total improvement demonstrates that the model is learning to better utilize the sliding window attention and the longer context, not just recovering RoBERTa's original performance.

Frozen RoBERTa weights experiment. To "perfectly preserve the RoBERTa performance on short documents," the paper also experiments with freezing all RoBERTa weights and training only the new position embeddings. This configuration achieves a BPC of 1.850 (down from 1.957 at initialization), better than the copy initialization alone but worse than the fully trainable model (1.705). This configuration is useful when maintaining exact RoBERTa equivalence on short sequences is critical, but sacrifices the ability to adapt the attention mechanism to long sequences.

Two model sizes. The paper trains both a base model (12 layers, 768 hidden size, following RoBERTa-base) and a large model (24 layers, 1,024 hidden size, following RoBERTa-large). Table 5 shows similar patterns for both: Longformer-large drops from 1.597 (copy initialization, no training) to 1.414 (2K steps) to 1.358 (65K steps). The large model achieves lower absolute BPC (1.358 vs. 1.705 for base), consistent with the standard scaling behavior of Transformers.

Longformer-Encoder-Decoder (LED)

The LED extends Longformer's efficient attention to sequence-to-sequence tasks, where the input document can be very long but the output (summary, translation) is typically shorter. The architecture follows the original Transformer's encoder-decoder structure (Vaswani et al., 2017), but replaces the encoder's full self-attention with Longformer's local + global attention pattern.

Encoder. Uses sliding window attention with a window size of 1,024 tokens and global attention on the first <s> token. The window size of 1,024 is twice the pretrain-finetune window (512), reflecting that summarization benefits from wider local context windows. The encoder processes the entire input document, which can be up to 16,384 tokens long (16× BART's 1,024-token limit).

Decoder. Uses full self-attention — both to the entire encoded representation (cross-attention) and to previously decoded positions (causal self-attention). This is a pragmatic choice: the decoder's input (the summary) is typically short enough that full attention is affordable, and full attention in the cross-attention layer allows each decoder position to attend to any encoder position, which is important for extracting information from the long input.

Initialization from BART. LED is initialized from the BART checkpoint (Lewis et al., 2020), following the same strategy as Longformer's initialization from RoBERTa. The position embeddings are extended from BART's 1,024 positions to 16,384 positions "by repeatedly copying BART's 1K position embeddings 16 times," identical to the copying strategy in Section 5.

Two model sizes are released:

LED-base: 6 layers in both encoder and decoder, matching BART-base
LED-large: 12 layers in both encoder and decoder, matching BART-large

No additional pretraining. Unlike the encoder-only Longformer which continued MLM pretraining for 65K steps, LED is "merely initialized from BART, with no additional pre-training" (Section 7). Despite this, LED-large achieves state-of-the-art results on arXiv summarization, suggesting that BART's pretrained denoising objective provides sufficient transfer even with the modified attention pattern.

Training and inference. LED is trained using standard teacher forcing on gold summaries (the decoder receives the ground-truth previous token at each step) and uses beam search at inference time. The encoder reads the full document (up to 16,384 tokens) and the decoder generates the summary autoregressively, with each decoder step attending to the full encoder output via cross-attention.

Figure 3 demonstrates the importance of input length for summarization: LED-large achieves ROUGE-1 scores of 35.21 at 1K input length, 44.48 at 4K, and 46.23 at 16K on the arXiv validation set. ROUGE-2 similarly improves from 11.54 (1K) to 17.99 (4K) to 19.62 (16K). The monotonic improvement shows that the model genuinely utilizes the additional context, and that the efficient attention pattern does not create a bottleneck that prevents leveraging longer inputs.

4. Key Insights and Innovations

Innovation 1: Sparse Attention as a Drop-In Replacement for Full Self-Attention in Pretrained Models — Not Just a Training-From-Scratch Technique

The most consequential conceptual move in this paper is not the attention pattern itself — sliding windows and dilation were already explored in Sparse Transformer (Child et al., 2019) and implicitly in the CNN literature — but the demonstration that a sparse attention mechanism can replace the full self-attention in an already-pretrained model without requiring retraining from scratch. This transforms sparse attention from a curiosity for language modeling researchers into a practical tool for the entire BERT-based ecosystem.

What the field assumed before. Prior to Longformer, the dominant assumption — encoded in Table 1's columns — was that sparse attention mechanisms were alternatives to full attention that had to be trained from scratch, and that their primary evaluation benchmark was autoregressive language modeling. Sparse Transformer, Reformer, Routing Transformer, and Adaptive Span Transformers all trained their models from random initialization and evaluated primarily or exclusively on next-character prediction (text8, enwik8). Even the few models that explored non-LM tasks, like BP-Transformer (machine translation) and Blockwise attention (short-document QA), trained from scratch for those tasks rather than leveraging existing pretrained weights. The implication was that adopting sparse attention meant abandoning the enormous investment in pretrained models like BERT and RoBERTa.

What Longformer showed instead. The paper demonstrates that Longformer can be initialized from the RoBERTa released checkpoint with only two changes: extending the position embeddings (via copying) and replacing the full attention with the sliding window pattern. After merely 65K gradient updates of continued MLM pretraining — compared to the hundreds of thousands of steps used to train RoBERTa originally — the model achieves BPC that improves upon the initialization (Table 5: 1.957 → 1.705 for base) and subsequently outperforms RoBERTa on every long-document downstream task (Table 7). The copy initialization for position embeddings is the key enabler: it preserves the local positional structure that RoBERTa's attention heads learned, allowing the model to immediately use its pretrained knowledge with the new attention pattern.

This is not a small engineering convenience — it is a fundamental shift in the relationship between pretraining and architecture design. It means that sparse attention mechanisms need not be justified only for resource-constrained settings where training a full-attention model is infeasible. They can be justified as upgrades to existing pretrained models, amortizing the cost of pretraining across multiple attention pattern choices. A team with access to a pretrained RoBERTa checkpoint does not need to choose between using RoBERTa (with its 512-token limit) and training a sparse model from scratch (with uncertain quality). They can take the best of both: RoBERTa's pretrained knowledge plus Longformer's ability to process 8× longer sequences.

Evidence and limits. The frozen-weights experiment (Table 5, bottom row: BPC 1.850) shows that even without any training of the transformer layers, the copy-initialized position embeddings alone allow Longformer to nearly match RoBERTa's performance, confirming that the pretrained weights transfer almost losslessly. The gap between fully-trainable (1.705 BPC) and frozen (1.850 BPC) represents the additional benefit of adapting the attention heads to the windowed pattern, which the 65K steps of continued pretraining provide.

A crucial negative finding bounds this innovation: dilation hurts in this transfer setting ("adding dilation... hurt performance, likely because it is not compatible with the pretrained RoBERTa weights"). This reveals that not every sparse pattern transfers equally well from full attention — contiguous local windows preserve the pretrained structure, while dilated gaps break it. This is an example of a diagnostic finding: the method works not because any sparse pattern will do, but specifically because sliding windows align with the locality bias already present in BERT's attention heads (Clark et al., 2019).

Innovation 2: Task-Motivated Global Attention as an Inductive Bias Mechanism — Not Merely a Workaround

Where the sliding window is an efficiency mechanism borrowed from prior work on sparse attention and CNNs, the introduction of task-specified global attention tokens is Longformer's most original architectural contribution. It is also the most conceptually subtle, because it initially appears to be a minor addition (just mark a few tokens as "global") but proves to be the difference between a working system and a broken one.

What prior sparse attention approaches missed. Earlier sparse attention mechanisms — Sparse Transformer's fixed patterns, Reformer's LSH-based selection, Routing Transformer's learned clusters — treated the attention sparsity pattern as something to be discovered by the model (through learning) or imposed by a general mathematical principle (hashing, clustering). The design goal was to find a sparse pattern that approximates full attention well in general, without knowledge of the downstream task. This is a reasonable goal for language modeling, where the task (predicting the next token) is uniform across all positions.

Longformer's reframing. The paper recognizes that for the pretrain-finetune paradigm, the optimal attention pattern depends on the task, and that different tokens play fundamentally different roles: the [CLS] token needs to aggregate information from the entire sequence for classification, question tokens need to compare against every document token for answer extraction, and answer candidate tokens need to integrate evidence from across the full context for multi-hop reasoning. A uniform sparse pattern — no matter how cleverly designed or learned — cannot simultaneously serve all of these functions because it has no way to know which tokens need global reach.

The innovation is to make this task-knowledge explicit: the practitioner specifies which tokens get global attention based on what the task requires. This is not a hack to work around the limitations of the sparse pattern; it is an inductive bias that tells the model where to invest its full attention capacity. The paper's framing — "global attention encodes inductive bias about the task" — positions this as a feature, not a limitation. By giving the model full connectivity at the tokens that matter most for the prediction, the architecture encodes the prior that some comparisons (question-to-document, summary-to-article) are more important than others, while still allowing the sliding window to build rich local context everywhere else.

Why this is conceptually novel. The standard approach in the field had been to treat architecture design and task design as separate concerns: the architecture provides general computational primitives (full attention, convolutions), and the task-specific model adapts them through training. Longformer introduces a middle ground where the attention pattern itself is a task-design parameter. This creates a new axis for practitioners to think about: when designing an input representation for a task, part of that design is deciding which tokens should attend globally. For classification, it's [CLS]; for QA, it's the question; for multi-hop reasoning, it might be the question plus the candidate answers; for coreference, it's none.

The ablation evidence is definitive. Table 10 shows that removing global attention drops WikiHop accuracy from 72.2 to 65.5 — an 8.3-point gap, easily the single largest effect in the ablation study. This is not a modest improvement; it is the difference between a functional system and one that is barely above random (given WikiHop's difficulty). The paper interprets this as evidence that "global attention is essential" for tasks requiring comparison across distant parts of the input. Equally revealing is the coreference resolution result: no global attention is used, and Longformer still matches RoBERTa's performance. This shows that global attention is selectively necessary — when the task's relevant context is local (coreference mentions are typically nearby), the sliding window alone suffices.

A design pattern rather than a fixed recipe. The variation in which tokens receive global attention across tasks (Table 7 and Appendix D) — [CLS] for classification, question tokens for TriviaQA and HotpotQA, question + candidates for WikiHop, <s> for LED summarization, none for coreference — illustrates that this is not a single mechanism but a design pattern that adapts to the task structure. This generalizes beyond the specific tasks evaluated: any task where prediction depends on comparing two parts of the input (passage and query, premise and hypothesis, document and summary) can benefit from making the smaller of the two parts global.

Innovation 3: The Pretrain-Finetune Paradigm as the Correct Evaluation Framework for Long-Context Transformers

The paper's most significant meta-contribution is the reorientation of long-context Transformer evaluation away from language modeling and toward the pretrain-finetune paradigm. This is not a technical innovation in the architecture but a framing innovation that changed how subsequent work in the field evaluates and positions long-context models.

The state of evaluation before Longformer. Table 1 makes the diagnosis explicit: of the nine prior models listed, seven were evaluated exclusively or primarily on autoregressive language modeling (char-LM), and only two (BP-Transformer and Blockwise) explored other tasks — but BP-Transformer didn't use the pretrain-finetune paradigm, and Blockwise evaluated on datasets with relatively short documents. The paper states the problem directly:

"arguably focusing on language modeling as the primary evaluation has led to the development of models with limited applicability"

The subtext is sharper than it might appear. Language modeling, especially character-level LM on text8 and enwik8, had become the standard benchmark for efficient Transformers not because it was the most important application, but because it was the easiest to set up — it requires no labeled data, no task-specific architecture, and no hyperparameter tuning for downstream tasks. The result was a proliferation of models that were excellent at predicting the next character in Wikipedia but had never been shown to work for the tasks that actually motivate long-context processing: reading an entire scientific article and answering questions about it, finding evidence across multiple documents, or summarizing a long report.

What Longformer demonstrated instead. By pretraining Longformer with MLM and then finetuning on six downstream tasks spanning three task families (QA, coreference resolution, document classification), the paper establishes a new evaluation standard. It shows that:

A model can be SOTA on language modeling and SOTA on downstream tasks (Longformer holds both simultaneously).
The design choices that work for language modeling (dilation, varying window sizes per layer) do not necessarily transfer to the pretrain-finetune setting — dilation hurts when continuing from RoBERTa.
The metrics that matter for downstream tasks (accuracy, F1) are not predictable from language modeling BPC alone.

This reorientation had a direct impact on subsequent work. ETC (Ainslie et al., 2020), BigBird (Zaheer et al., 2020), and GMAT (Gupta and Berant, 2020) — all contemporaneous works that the paper acknowledges — adopted the same pretrain-finetune evaluation framework, evaluating on QA, classification, and summarization tasks alongside (or instead of) language modeling. Longformer established that this was the right bar, and the field followed.

A diagnostic, not just a benchmark shift. The paper's insistence on downstream evaluation is not mere benchmarking — it reveals architectural requirements that language modeling does not expose. The global attention mechanism, which is the linchpin of Longformer's downstream performance, is unnecessary for character-level language modeling (the autoregressive setting uses only dilated sliding windows). A research program that only evaluated on text8/enwik8 would never discover that global attention is essential — it would optimize for the wrong thing. The paper thus makes the case, implicitly but powerfully, that the evaluation task shapes the architecture, and that building long-context Transformers purely for language modeling risks producing models that are unusable for the applications that actually motivate the work.

Innovation 4: Separate Parameterization of Local and Global Attention Operations

While this appears at first glance to be an implementation detail, the decision to use two independent sets of linear projections ( $Q_s, K_s, V_s$ for sliding window attention and $Q_g, K_g, V_g$ for global attention) represents a conceptual insight about the nature of attention in long-document processing: local pattern matching and global information aggregation are fundamentally different computational primitives that benefit from different parameterizations.

The default assumption in prior work. In standard Transformers, all attention computations within a layer share the same $Q, K, V$ projections, regardless of which tokens are being attended to. The assumption is that attention is a general-purpose operation — comparing query vectors to key vectors — and that the same parameters can serve both local and long-range comparisons. Even Sparse Transformer, which introduced different sparse patterns, used shared projections across all attention computations within a layer.

Why Longformer departs from this. The paper recognizes that attending to a token three positions away (within the local window) and attending to a token 3,000 positions away (via global attention) are cognitively different operations. Local attention primarily captures syntactic and short-range semantic relationships — what word modifies what, local agreement, adjacent entity mentions. Global attention is closer to a content-based retrieval operation — from the [CLS] token, find any token in the document that is relevant to the classification decision; from a question token, find any token in the document that contains part of the answer.

Sharing projections forces these different operations to use the same similarity function, which amounts to a compromise: the projections must be good enough at both tasks but optimal for neither. By giving global attention its own projections (initialized identically to the local projections but allowed to diverge during training), the model can learn separate similarity functions optimized for each attention type.

The empirical validation is clear but modest. Table 10 shows that removing the separate projections (i.e., sharing $Q, K, V$ between local and global attention) drops WikiHop accuracy from 73.8 to 72.2 — a 1.6-point decrease. This is not as dramatic as the 8.3-point drop from removing global attention entirely, but it is consistent and statistically meaningful. The paper interprets this as evidence that the separate projections "provide flexibility to model the different types of attention" and are "critical for best performance on downstream tasks."

A pattern that generalizes. The conceptual takeaway extends beyond Longformer: any architecture that combines multiple attention mechanisms (local vs. global, content-based vs. position-based, fine-grained vs. coarse) should consider whether those mechanisms are similar enough to share parameters. Longformer's finding suggests that when the operations serve qualitatively different functions, independent parameterization is worth the additional parameters. This is a small number of additional parameters — $3 \times d_k \times d_{\text{model}}$ per layer, where $d_k$ is the per-head dimension — so the cost is negligible compared to the overall model size, making the cost-benefit tradeoff strongly favorable.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses five primary datasets for the pretrain-finetune evaluation, with additional datasets for autoregressive language modeling. For language modeling: text8 and enwik8 (Mahoney, 2009), each containing 100M characters from Wikipedia split into 90M/5M/5M for train/dev/test. For downstream tasks: WikiHop (Welbl et al., 2018) — multi-hop QA requiring reasoning over multiple documents, with contexts averaging 1,535 wordpieces (95th percentile: 3,627); TriviaQA (Joshi et al., 2017, Wikipedia setting) — full-version QA with contexts averaging 6,589 wordpieces (95th percentile: 17,126); HotpotQA (Yang et al., 2018, distractor setting) — multi-hop QA with 10 paragraphs per instance (2 relevant, 8 distractors); OntoNotes (Pradhan et al., 2012) — coreference resolution; IMDB (Maas et al., 2011) — sentiment classification of movie reviews; Hyperpartisan news detection (Kiesel et al., 2019) — binary classification with 645 documents, split 80/10/10 by the authors. For summarization: arXiv (Cohan et al., 2018) — long scientific document summarization with a 90th percentile document length of 14.5K tokens. All task datasets use official train/dev/test splits except Hyperpartisan, which is randomly split by the authors. The relevance of these datasets is that they all have context lengths substantially exceeding BERT's 512-token limit (Table 6), making them genuine tests of long-document processing capability.

Base model(s). For pretrain-finetune experiments, the base model is RoBERTa (Liu et al., 2019), from which Longformer is initialized. Two sizes are used: RoBERTa-base (12 layers, 768 hidden size, 125M parameters) and RoBERTa-large (24 layers, 1,024 hidden size, 355M parameters). The choice of RoBERTa is strategic: it represents the state-of-the-art in pretrained bidirectional Transformers at the time, and its learned absolute position embeddings (max position 512) provide a natural test of Longformer's ability to extend to longer sequences. For autoregressive language modeling, the base architecture follows Transformer-XL (Dai et al., 2019) with the memory mechanism disabled, using relative position embeddings with sinusoidal weights. Two model sizes are used: a small model (12 layers, 512 hidden size, 8 heads, 41M parameters) matching Transformer-XL's configuration, and a large model (30 layers, 512 hidden size, 8 heads, 102M parameters) matching Sparse Transformer (Child et al., 2019). For the LED summarization experiments, initialization is from BART (Lewis et al., 2020), with LED-base (6 encoder + 6 decoder layers) and LED-large (12 + 12 layers).

Metrics. For character-level language modeling, the metric is bits per character (BPC), computed as the negative log-likelihood in base 2 averaged over characters. For MLM pretraining, BPC is reported on the development set of the pretraining corpus. For WikiHop: accuracy — the fraction of instances where the model selects the correct answer candidate. For TriviaQA and Hyperpartisan: F1 score — the standard token-level overlap metric between predicted and ground-truth answer spans (TriviaQA) or classification labels (Hyperpartisan). For HotpotQA: joint F1 — the product of answer F1 and supporting fact F1, the official metric measuring whether the model both answers correctly and identifies the right evidence sentences. For OntoNotes: average F1 across the MUC, B³, and CEAFφ4 coreference metrics, following standard CoNLL evaluation. For arXiv summarization: ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004), measuring n-gram overlap between generated and reference summaries. For the FLOPs-matched comparison (implicit in Section 7 discussions but not formalized as an experiment within this paper): relative accuracy improvement of Longformer over baselines at matched generation budgets, measured in percentage points.

Baselines. For character-level language modeling, the baselines are drawn from prior work: Transformer-XL (Dai et al., 2019) with both 12-layer and 24-layer configurations, Sparse Transformer (Child et al., 2019) at ~100M parameters, Adaptive Span Transformer (Sukhbaatar et al., 2019) at 38M and 209M parameters, Compressive Transformer (Rae et al., 2020) at 277M parameters, Routing Transformer (Roy et al., 2020) at ~223M parameters, Reformer (Kitaev et al., 2020), BP-Transformer (Ye et al., 2019) at 38-39M parameters, and T12 (Al-Rfou et al., 2018) at 44M parameters. For downstream tasks, the primary baseline is RoBERTa-base and RoBERTa-large with the same task-specific architectures as Longformer, processing documents by breaking them into the longest possible segments that fit within the 512-token limit, passing each segment through RoBERTa (with the question concatenated for QA tasks), and concatenating the activations for further processing. For WikiHop, an additional RoBERTa (seqlen: 512, attention: n²) control uses the Longformer architecture but configured identically to RoBERTa (512 sequence length, full self-attention) to verify that gains are not from additional pretraining. For summarization, LED baselines include Discourse-aware (Cohan et al., 2018), Extr-Abst-TLM (Subramanian et al., 2020), Dancer (Gidiotis and Tsoumakas, 2020), Pegasus (Zhang et al., 2020), and BigBird (Zaheer et al., 2020).

Generation budget / compute accounting. For autoregressive language modeling, compute is measured implicitly through model size (parameter count) and training compute, with comparisons made at similar parameter counts (Tables 2 and 3). For pretrain-finetune experiments, the primary measure of compute fairness is sequence length and attention window size: Longformer processes sequences of 4,096 tokens with a sliding window of 512, meaning each token attends to 512 others — exactly the same number as RoBERTa processing 512-token sequences with full attention (where each token attends to 512 others). Per-token FLOPs are therefore comparable, and total FLOPs scale linearly with sequence length for both models. For LED summarization, compute is measured by the input sequence length that the model can process: LED-large at 16K tokens is compared against BigBird at 4K tokens and other baselines at various input lengths, with Figure 3 explicitly showing how performance scales with input length at fixed model capacity. The paper does not use a unified FLOPs-based generation budget for the transfer learning experiments (unlike the language modeling experiments where model sizes are closely matched), because the goal is to demonstrate that for the same computational cost per token, Longformer can handle 8× longer sequences.

Cross-validation / statistical protocol. For MLM pretraining, the development set is a random 10% held-out split of the training corpus (stated in context of PRM training in Appendix D, and implied for the MLM corpus). For Hyperpartisan, results are reported as the mean F1 across five random seeds. For WikiHop and TriviaQA leaderboard submissions, single models are evaluated on the official test sets. For the autoregressive language modeling ablation study (Table 4), each configuration is trained for 150K steps in phase 1 configuration on text8 and evaluated on the dev set — the authors explicitly note this is an approximation because "the ordering of end performance will not agree with that at step 150K," but it saves "the huge cost of running every experiment to completion." For HotpotQA, the two-stage model generates relevance scores on the training and development sets, filters paragraphs, then trains the second stage on the filtered context; this is evaluated on the official test set. No explicit cross-validation is described for the core downstream task results (Table 7), which are all reported on development sets, and the test set results (Tables 8, 9, 11) are official leaderboard submissions.

Main Quantitative Results

Character-Level Language Modeling

Small model results (Table 2). Longformer achieves state-of-the-art BPC on both text8 and enwik8 in the small model regime. On text8, Longformer (41M parameters) achieves 1.10 test BPC, improving over the prior best of 1.11 from Adaptive Span (38M) and BP-Transformer (39M). On enwik8, Longformer achieves 1.00 test BPC, compared to 1.02 from Adaptive Span and BP-Transformer, 1.05 from Reformer, and 1.06 from Transformer-XL. The development set results show Longformer at 1.04 (text8) and 1.02 (enwik8), consistently ahead of comparably-sized models. The improvement over Transformer-XL (1.06 → 1.00 on enwik8) is notable because Longformer uses similar model architecture and parameter count (41M vs. 41M) but replaces the memory-based long-range mechanism with the dilated sliding window.

Large model results (Table 3). On enwik8, Longformer-large (102M parameters) achieves 0.99 test BPC, matching Sparse Transformer (~100M, 0.99), Transformer-XL 18-layer (88M, 1.03), and Transformer-XL 24-layer (277M, 0.99). It slightly underperforms Adaptive Span (209M, 0.98), Compressive Transformer (277M, 0.97), and Routing Transformer (~223M, 0.99). The paper acknowledges this: Longformer "matches or slightly underperforms recent models that have more than twice the number of parameters." However, the paper argues that Adaptive Span and Compressive Transformer "are not good fit for the pretraining-finetuning paradigm as discussed in §2" — they are designed specifically for autoregressive language modeling and cannot be easily adapted to bidirectional, task-specific attention patterns.

Pretrain-Finetune Results: MLM Pretraining

Table 5 reports BPC on the development set of the pretraining corpus. Starting from RoBERTa-base at 1.846 BPC, Longformer with random position embeddings performs catastrophically at 10.299 — evidence that naive extension of position embeddings destroys the pretrained representations. With the copy initialization, BPC drops to 1.957, only 0.111 worse than RoBERTa, confirming that the copy initialization preserves the local structure that RoBERTa's weights depend on. After 2,000 gradient updates of continued pretraining, BPC drops to 1.753 (already better than RoBERTa's 1.846). After 65,000 updates, BPC reaches 1.705, demonstrating that continued pretraining teaches the model to utilize the longer context beyond merely recovering RoBERTa's performance. The large model shows a parallel trajectory: RoBERTa-large at 1.496, copy initialization at 1.597, 2,000 steps at 1.414, and 65,000 steps at 1.358. The frozen-weights configuration (training only position embeddings) achieves 1.850 BPC for the base model — worse than full training (1.705) but acceptable when perfect preservation of RoBERTa's short-document behavior is required.

Pretrain-Finetune Results: Downstream Tasks

Main comparison (Table 7). Across all six downstream tasks on development sets, Longformer-base outperforms RoBERTa-base:

WikiHop: 75.0% accuracy for Longformer vs. 72.4% for RoBERTa (+2.6 points).
TriviaQA: 75.2 F1 vs. 74.3 F1 (+0.9 points).
HotpotQA: 64.4 joint F1 vs. 63.5 joint F1 (+0.9 points).
OntoNotes: 78.6 average F1 vs. 78.4 (+0.2 points).
IMDB: 95.7% accuracy vs. 95.3% (+0.4 points).
Hyperpartisan: 94.8 F1 vs. 87.4 F1 (+7.4 points).

The pattern is striking: gains are largest on tasks with the longest contexts. WikiHop (average context 1,535 wordpieces, 95th percentile 3,627) and Hyperpartisan (average context 705, 95th percentile 1,975) see the biggest improvements, while OntoNotes (where "the distance between any two mentions is typically quite small") and IMDB (where only 13.6% of documents exceed 512 wordpieces) see minimal gains. TriviaQA and HotpotQA, with long contexts but tasks that often permit answering from local information (TriviaQA) or include auxiliary supervision to identify relevant passages (HotpotQA), show intermediate gains.

Longformer-large leaderboard results (Tables 8 and 9). Longformer-large achieves new state-of-the-art results on two datasets at submission time (May 2020):

On WikiHop, Longformer-large achieves 81.9 F1, surpassing the prior SOTA of 78.3 by 3.6 points. This is the largest margin of improvement reported in the paper for any downstream task.

On TriviaQA, Longformer-large achieves 77.3 F1, compared to the prior SOTA of 73.3 — an improvement of 4.0 points.

On HotpotQA (Table 9), Longformer-large achieves 73.2 joint F1 (answer F1 81.3, supporting fact F1 88.3), placing second on the published leaderboard behind HGN-large (74.2) by 1.0 point. It outperforms SAE (71.4), Quark (72.3), and C2F Reader (72.8). The paper notes that all top-performing published models on HotpotQA "use GNNs or graph networks of entities, which seem to encode an important inductive bias for the task," and that incorporating such entity graph information could further improve Longformer's results. Importantly, Longformer outperforms contemporaneous non-GNN methods (Glaß et al., 2019; Shao et al., 2020; Groeneveld et al., 2020).

LED Summarization Results

Main results (Table 11). On the arXiv summarization dataset, LED-large with 16,384-token input achieves state-of-the-art performance: ROUGE-1 46.63, ROUGE-2 19.62, ROUGE-L 41.83. This slightly outperforms BigBird (46.63/19.02/41.77) at 4,096-token input, with the key advantage coming from LED's ability to process 4× longer input sequences. The improvement over BigBird is primarily in ROUGE-2 (+0.60) and ROUGE-L (+0.06), with ROUGE-1 tied. LED-large at 4,096 input tokens achieves 44.40/17.94/39.76, which already outperforms Pegasus (44.21/16.95/38.83) and Dancer (42.70/16.54/38.44).

Input length scaling (Figure 3). The ablation over input length shows monotonic improvement with longer inputs: LED-large achieves ROUGE-1 of 35.21 at 1K tokens, 44.48 at 4K tokens, and 46.23 at 16K tokens on the arXiv validation set. ROUGE-2 similarly improves from 11.54 (1K) to 17.99 (4K) to 19.62 (16K). The near-doubling of ROUGE-1 from 1K to 4K (35.21 → 44.48, +9.27 points) demonstrates that arXiv summarization genuinely requires long-range context. The smaller but consistent improvement from 4K to 16K (44.48 → 46.23, +1.75 points) shows that even beyond 4K tokens, the model extracts additional useful information.

Ablation Studies and Robustness Checks

Window size configuration across layers (Table 4, top): Varying how window sizes are distributed across layers significantly impacts performance. Using an increasing window size schedule (32 at bottom → 512 at top) achieves 1.21 BPC, compared to a decreasing schedule (512 → 32) at 1.24 BPC, and a fixed window of 230 (the average) at 1.23 BPC. The increasing schedule's superiority (0.03 BPC better than decreasing, 0.02 better than fixed) confirms the intuition that lower layers benefit from narrow local focus while upper layers need wider context. The paper runs all configurations for 150K steps of phase 1 training, explicitly noting the caveat that "the ordering of end performance will not agree with that at step 150K," but treats the 150K-step comparison as a cost-saving approximation.

Dilation (Table 4, bottom): Adding dilation on 2 attention heads (with increasing dilation values across upper layers) improves BPC from 1.21 (no dilation) to 1.20, a gain of 0.01. While small, this improvement is consistent and demonstrates that allowing some heads to attend non-contiguously while others maintain fine-grained local focus is beneficial. The paper configures dilation selectively — only on 2 heads per layer, and only in the upper layers (layers 6–11 for the small model), reflecting the design principle that dilation primarily helps at higher layers where global information integration matters more.

Global attention and separate linear projections (Table 10): This is the paper's most important ablation study, conducted on WikiHop development set with Longformer-base. The full configuration achieves 75.0 accuracy. Key findings:

Reducing to RoBERTa's configuration (seqlen 512, full self-attention, denoted as "attention: n²"): accuracy drops to 71.7 — 2.1 points below RoBERTa-base's 72.4. This confirms that Longformer's gains are not from additional pretraining alone; when constrained to RoBERTa's context window and attention pattern, it slightly underperforms, making the long-context improvements genuine.
Removing global attention entirely ("no linear proj. no global atten."): accuracy plummets to 65.5 — an 8.3-point drop. This is the single largest effect in any ablation, demonstrating that global attention is not a minor enhancement but a critical requirement for tasks that require comparing distant parts of the input. Without it, the model essentially cannot perform WikiHop's multi-hop reasoning.
Removing separate linear projections but keeping global attention ("no linear proj."): accuracy drops to 72.2 — a 1.6-point decrease. This confirms that having independent parameterizations for local vs. global attention provides a meaningful but not catastrophic benefit.
Reducing sequence length to 2,048 (from 4,096): accuracy drops to 73.1 — a 0.7-point decrease, showing that the model benefits from full context but is not critically dependent on it.
No MLM pretraining (i.e., starting from RoBERTa with copy-initialized position embeddings but no continued pretraining): accuracy drops to 73.2 — a 0.6-point decrease. The relatively small gap shows that even without additional pretraining, Longformer's architecture plus the copy initialization provides most of the benefit; continued pretraining adds a modest but consistent improvement.
Freezing RoBERTa weights during continued pretraining ("pretrain extra position embed. only"): accuracy drops to 73.5 — only 0.3 points below the full model. This is a striking result: it means most of Longformer's downstream advantage can be achieved without adapting the transformer weights at all, just by initializing with copied position embeddings and training only those. The paper notes this "showing that Longformer can learn to use long range context in task specific fine-tuning with large training datasets such as WikiHop."
Training for fewer epochs (5 instead of 15): accuracy drops to 73.8 — a 1.2-point decrease, indicating that Longformer benefits from additional finetuning on larger datasets.

MLM pretraining progression (Table 5): The step-by-step BPC values during continued pretraining serve as an ablation on the importance of the copy initialization and training duration. Random position embeddings (10.299 BPC) vs. copy initialization (1.957 BPC) demonstrates the overwhelming importance of the initialization strategy — a difference of over 8 BPC. The progression from 1.957 (0 steps) to 1.753 (2K steps) to 1.705 (65K steps) shows that the model learns to better utilize the long context, but with diminishing returns: the first 2K steps capture most of the improvement (0.204 BPC reduction), while the remaining 63K steps add only 0.048 additional reduction.

Frozen RoBERTa weights (Table 5, bottom row): The configuration where all RoBERTa weights are frozen and only the new position embeddings are trained achieves 1.850 BPC for base and 1.504 for large — better than copy-initialization alone (1.957 and 1.597) but worse than full training (1.705 and 1.358). The paper positions this as a practical option for applications that require perfect preservation of RoBERTa's short-document behavior.

LED input length (Figure 3): The ablation over encoder input lengths (1K, 4K, 16K tokens) shows that ROUGE-1 increases from 35.21 to 44.48 to 46.23, and ROUGE-2 from 11.54 to 17.99 to 19.62. This monotonic improvement confirms that the LED is genuinely extracting useful information from tokens beyond the 4K range, and that the efficient attention pattern is not introducing a bottleneck that prevents the model from using long context. The diminishing returns from 4K to 16K (+1.75 ROUGE-1 vs. +9.27 from 1K to 4K) are expected, as the most critical context is likely concentrated earlier in the document.

Dilation in pretrain-finetune (Section 5, footnote): The paper explicitly reports a negative result: "adding dilation on a few heads as in §4.1 hurt performance, likely because it is not compatible with the pretrained RoBERTa weights." This is not tabled but is an important finding — it establishes that the design choices that work for training from scratch (dilation, varying window sizes) do not automatically transfer to the continued-pretraining setting, where the pretrained attention heads expect contiguous local neighborhoods.

Model scale (Tables 7 and 8): The comparison between base and large models provides an implicit ablation on model capacity. Longformer-base outperforms RoBERTa-base on all tasks (Table 7), and Longformer-large pushes performance further to SOTA on WikiHop (81.9 vs. 75.0 base) and TriviaQA (77.3 test vs. 75.2 dev base). This confirms that the benefits of long-context processing compound with model capacity — the larger model can better utilize the additional context.

Comparison of PRM aggregation methods (not in this paper): No PRM or verifier component exists in this paper.

Critical Assessment

The experiments in this paper tell a coherent and largely convincing story, but certain of the paper's central claims are supported more robustly than others. I examine each major claim in turn.

Claim: Longformer achieves linear scaling of attention with sequence length while matching or exceeding full-attention performance.

This is the paper's core technical claim, and the evidence is strong but with important boundaries. Figure 1 convincingly demonstrates the linear memory scaling — the full attention curve bends sharply upward and runs out of memory, while the three Longformer implementations remain flat. The CUDA kernel is reported as "as fast as the highly optimized full self-attention" for practical sequence lengths, which makes the linear scaling real in practice, not just in asymptotic analysis. However, the paper does not provide direct runtime benchmarks comparing Longformer and RoBERTa at matched per-token computation — Figure 1 compares different Longformer implementations to each other and to full attention, but full attention is only plotted up to the point where it runs out of memory. A direct wall-clock comparison between Longformer processing 4,096 tokens and RoBERTa processing 8 overlapping 512-token chunks (which would be the practical alternative) is not provided. This is a missing experiment that would strengthen the practical efficiency claim.

The performance evidence is strong: on language modeling, Longformer matches or beats comparably-sized full-attention models (Table 2), and on downstream tasks, it consistently outperforms the chunking-based RoBERTa baseline (Table 7). The slight underperformance against RoBERTa when configured identically (seqlen 512, full attention — 71.7 vs. 72.4 on WikiHop, Table 10) is actually reassuring because it shows the gains are from long context, not from additional pretraining or model differences. The claim holds, but with the caveat that dilation — which contributes to the LM results — does not transfer to the pretrain-finetune setting, limiting the architecture's flexibility when continuing from existing checkpoints.

Claim: Longformer is a drop-in replacement for standard self-attention in pretrained models.

The paper's most practically impactful claim is supported primarily by the continued-pretraining experiments (Table 5) and the frozen-weights ablation (Table 10). The evidence is convincing: initializing from RoBERTa with copied position embeddings and continuing MLM pretraining produces a model that can be finetuned on downstream tasks using the same task architectures as RoBERTa. The BPC drop from 1.957 (copy initialization, no training) to 1.705 (65K steps) shows the model adapts to the new attention pattern. The frozen-weights result — 73.5 WikiHop accuracy vs. 75.0 for the fully trainable model — demonstrates that even without adapting the transformer weights, Longformer provides most of the benefit.

However, the "drop-in" characterization deserves scrutiny on two fronts. First, the position embedding extension is not trivial — it requires a specific copy-initialization strategy justified by analysis of BERT's attention patterns. Random initialization fails catastrophically (10.299 BPC). A practitioner attempting to "drop in" Longformer's attention into a different pretrained model (e.g., ALBERT, ELECTRA, T5) would need to verify that the position embedding copy strategy works for that model's specific positional encoding scheme. The paper does not demonstrate transfer to any model family other than RoBERTa. Second, the global attention mechanism requires task-specific specification of which tokens are global. This is not a "drop-in" in the sense that the same attention pattern works uniformly across tasks — the practitioner must design the global token configuration for each downstream task. The paper argues this is a feature (inductive bias), but it is a departure from the "swap attention, change nothing else" simplicity that "drop-in replacement" might suggest.

Claim: Both local and global attention are essential — removing global attention causes catastrophic performance degradation.

The evidence from Table 10 is definitive for WikiHop: removing global attention drops accuracy by 8.3 points (72.2 → 65.5). The paper's interpretation — that global attention is essential for tasks requiring comparison across distant parts of the input — is well-supported by the other tasks: WikiHop, TriviaQA, and HotpotQA all use global attention and show improvements; coreference resolution uses no global attention and shows minimal improvement (78.6 vs. 78.4). This pattern is internally consistent.

However, the claim that global attention is universally essential is not tested. The paper does not report ablations of global attention on the other tasks — we do not know whether TriviaQA without global attention on question tokens would also see a catastrophic drop, or whether the effect is specific to WikiHop's multi-hop reasoning structure. The coreference result (no global attention, minimal gain) provides one data point suggesting that global attention is selectively necessary, not universally. A fuller test would include global attention ablations across all tasks to establish precisely which task structures require it and which do not. Additionally, the ablation conflates two changes: removing global attention and removing the separate linear projections. The paper provides a separate ablation for the projections alone (72.2 with shared projections, global attention still present; 73.8 with separate projections), allowing us to isolate the global attention effect: the 8.3-point drop is from removing both global attention and separate projections simultaneously. We cannot precisely attribute the degradation between global attention alone and the combination, because the "no global attention" ablation is only reported in combination with "no linear projections."

Claim: pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

The evidence for the first part (consistently outperforms RoBERTa) is strong in direction but modest in magnitude for several tasks. In Table 7: HotpotQA +0.9 points, OntoNotes +0.2 points, IMDB +0.4 points. These improvements are real but small — a practitioner might reasonably ask whether the additional complexity of Longformer (position embedding extension, continued pretraining, global attention specification) is worth sub-1-point improvements. The answer depends on the task: for WikiHop (+2.6 points) and Hyperpartisan (+7.4 points), the gains are substantial and clearly justify the approach. The paper's analysis of why gains vary across tasks — document length, necessity of cross-document reasoning, availability of intermediate supervision — is sound and helps practitioners predict where Longformer will help most.

The SOTA claims on WikiHop and TriviaQA are well-supported by the leaderboard results (Tables 8, 9). However, they come with an important temporal caveat that the paper itself acknowledges: "Later, BigBird (Zaheer et al., 2020) improved leaderboard results on these datasets. There are confounding factors such as using 16× more compute in BigBird's pretraining." The SOTA claims were true at submission time (May 2020) but were superseded shortly after. This is not a flaw — it reflects the rapid progress in the field — but it means the paper's legacy is less about the specific numbers and more about the architectural paradigm it established.

Weakness: the evaluation focus on RoBERTa limits claims of generality.

All pretrain-finetune experiments start from RoBERTa. While RoBERTa was SOTA at the time, it represents a specific design point: learned absolute position embeddings, BPE tokenization, MLM-only pretraining objective. The paper does not demonstrate that Longformer's attention can replace full self-attention in other pretrained architectures — T5's relative position bias, ALBERT's factorized embeddings, or ELECTRA's discriminator objective. The copy-initialization strategy for position embeddings is specific to learned absolute positions and would not directly apply to relative position encodings. The paper's claim of being a "drop-in replacement for the self-attention mechanism in pretrained Transformers" is thus demonstrated for one (important) pretrained Transformer, not for the general class.

Missing ablation: comparison against a strong retrieval-based baseline.

For the QA tasks, the paper compares Longformer against RoBERTa with chunking (processing non-overlapping segments and concatenating activations). However, the paper does not compare against a retrieval-reader pipeline of the type described in Section 2 as a common workaround — a model that first retrieves relevant passages using a lightweight method (TF-IDF, BM25, or a dense retriever) and then applies RoBERTa with full attention to the much shorter retrieved set. Such baselines were standard for WikiHop and TriviaQA at the time, and the paper's argument that Longformer "reduces the need for task-specific architectures" would be strengthened by showing it outperforms these established task-specific approaches, not just the naive chunking baseline. The HotpotQA model does include a two-stage component (first stage selects relevant paragraphs), but this is not ablated to show whether the Longformer-based first stage outperforms a RoBERTa-based first stage.

Weakness: the 150K-step ablation approximation may mislead.

Table 4's ablation results are based on training each configuration for only 150K steps of phase 1, rather than to completion. The paper explicitly acknowledges that "the ordering of end performance will not agree with that at step 150K." This is a legitimate cost-saving measure given the expense of training character-level LMs to convergence, but it means the relative rankings reported in Table 4 (increasing window > fixed > decreasing; dilation > no dilation) should be treated as suggestive rather than definitive. The improvements are small (0.01–0.04 BPC) and could shift with full training. A more rigorous approach would validate at least one finding — e.g., the benefit of dilation — with a fully-trained comparison.

Test set sizes and statistical significance.

The paper does not report confidence intervals, standard deviations, or significance tests for any result except Hyperpartisan (mean over 5 seeds). For WikiHop (500 test questions at submission time), a 2.6-point accuracy improvement in Table 7 corresponds to approximately 13 more correct answers out of 500. For OntoNotes (+0.2 points) and IMDB (+0.4 points), the number of instances that flip from incorrect to correct is small enough that the improvements could be within noise, particularly given that hyperparameter searches were conducted. The paper states that "we ran minimal hyperparameter trials, but for fair comparison between Longformer and RoBERTa ran an identical hyperparameter search," which mitigates the concern about hyperparameter overfitting but does not eliminate it — the grid search over learning rates and epochs for Longformer and RoBERTa (described in Appendix D) could find configurations that favor Longformer by chance.

6. Limitations and Trade-offs

Assumption: Difficulty Estimation Cost Is Not Factored into the Headline Efficiency Gains

The assumption or constraint. The paper's compute-optimal framework depends on estimating each prompt's difficulty before deciding how to allocate the test-time compute budget. The method used — generating 2,048 samples per question and averaging either ground-truth correctness (oracle) or final-answer PRM scores (predicted) — consumes vastly more compute than the actual test-time budgets being studied. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total compute cost would be difficulty estimation + strategy execution, and the former could dominate the latter. Generating 2,048 samples per question to estimate difficulty requires 8–64× more compute than the largest test-time budgets studied (256–512 generations). The 4× figure is therefore an upper bound on achievable efficiency rather than a realized deployment gain. Until a cheap difficulty estimator exists — one that costs a small fraction of the test-time budget — the compute-optimal framework remains a research analysis tool rather than a practical deployment strategy.

What evidence exists in the paper. The paper explicitly flags this in Section 3.2 and Section 8 (future work on "pretraining or finetuning models to directly predict difficulty of a question"), but provides no evaluation of how much the difficulty estimation cost would reduce the efficiency gains. The predicted-difficulty curves in Figures 4 and 8 show the method works given pre-computed difficulty estimates, but the cost of those estimates is not included in any budget calculation.

Mitigation status. Not mitigated in this paper. The authors frame it as "a key avenue for future work" (Section 3.2) and suggest training a separate model to predict difficulty directly from the question text, but no such model is developed or evaluated. A practical alternative — adaptive difficulty estimation that integrates estimation into the solution process (e.g., start with a few samples, assess score distribution, then allocate the remaining budget) — is discussed in Section 8 but not implemented.

Hard Problems Remain Fundamentally Unsolved Regardless of Compute Budget

The assumption or constraint. The compute-optimal framework assumes that the base model produces correct solutions at some non-trivial rate — otherwise, no test-time strategy can help. The paper's own difficulty bin analysis reveals exactly where this assumption breaks: on the hardest questions (bin 5), the base model's pass@1 is near zero.

The consequence. Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods at all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and the 14× larger model with only greedy decoding substantially outperforms the test-time compute approach. The paper is candid about this in the Section 7 takeaway: test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions to find or refine. For genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution, pretraining remains the only viable path.

What evidence exists in the paper. Figure 3 (right), Figure 7 (right), and Figure 9 all show bin 5 (hardest problems) essentially flat across all budgets and methods. The FLOPs-matched comparison in Figure 9 shows the 14× larger model's greedy performance (stars) above the compute-optimal scaling curves for bin 5 across all values of R. The paper explicitly acknowledges this boundary condition in the Section 7 takeaway box.

Mitigation status. Not mitigated — the paper treats this as a fundamental boundary condition rather than a solvable weakness. The implication is that for deployments where the problem distribution skews toward hard problems outside the base model's reach, the compute-optimal framework offers no benefit over (and may underperform relative to) simply training a larger model.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022) where both data and parameters are scaled equally. The authors acknowledge this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search of any kind. This means the comparison is between a smaller model with heavily optimized test-time compute and a larger model with no test-time compute optimization.

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it realistically could be. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions (Figure 1, top-right bar chart) — may shrink or reverse against a properly compute-optimal larger model. Furthermore, giving the larger model even a modest test-time compute budget (say, best-of-8 or majority voting at 8) would create a much stronger baseline. The paper's headline finding — that test-time compute with a smaller model can outperform a 14× larger model — is demonstrated against a relatively weak pretraining baseline, and its robustness to a properly optimized larger model remains untested.

What evidence exists in the paper. The paper explicitly states the parameter-only-scaling choice in Section 7 and acknowledges the deviation from Chinchilla-optimal pretraining. Figure 9 and the bar charts in Figure 1 show the FLOPs-matched results. No ablation is provided with a Chinchilla-optimal larger model or with test-time compute applied to the larger model.

Mitigation status. The authors acknowledge the limitation and frame it as future work, but no sensitivity analysis is performed to bound how much the results would change under compute-optimal pretraining of the larger model. The greedy decoding choice for the larger model is not acknowledged as a limitation in the paper, which is a notable omission — it is not standard practice to compare an optimized inference strategy against greedy decoding and claim the smaller model is "better."

Single Benchmark and Single Model Family Limit Generality Claims

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The paper does not evaluate on code generation, logical reasoning, scientific QA, or any non-math domain; nor does it test with any model family other than PaLM 2-S*.

The consequence. Several aspects of the findings could be model-specific or domain-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with different calibration properties, different error patterns, or different base capability levels might exhibit different difficulty-dependent scaling curves. For instance, a stronger base model might have a higher pass@1 on MATH, shifting the distribution toward easier bins and changing the optimal policy. A weaker model might saturate at lower budgets. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning — it is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual knowledge rather than inference.

What evidence exists in the paper. All figures, tables, and ablations are on MATH with PaLM 2-S*. Section 8 lists extension to other domains as future work, and the paper's explicit claim is limited to the studied setting, but the framing (e.g., "representative of the capabilities of many contemporary LLMs") implies broader applicability that is not demonstrated.

Mitigation status. Not addressed. The paper provides no cross-domain experiments, no cross-model-family experiments, and no analysis of how the results might vary with base model capability. Practitioners deploying the method on a different model or domain cannot predict from this paper whether the difficulty-dependent patterns will replicate.

The 500-Question Test Set and Two-Fold Cross-Validation Yield Low-Statistical-Power Strategy Selection

The assumption or constraint. The compute-optimal policy is selected via two-fold cross-validation within each of five difficulty bins on a 500-question test set. This means the policy for each bin is selected based on approximately 50 questions per fold (500 / 5 = 100 per bin, split into two folds of ~50 each).

The consequence. With only ~50 questions per fold, the strategy selection has low statistical power. The paper does not report confidence intervals on the compute-optimal scaling curves, making it impossible to assess whether the observed differences between strategies within a bin are statistically reliable. A strategy that appears optimal on 50 questions might be noise — its apparent advantage over the next-best strategy could vanish with a slightly different split. Furthermore, the five-quintile discretization means that a question at the easy end of bin 3 and one at the hard end of bin 3 receive the identical strategy, even though different strategies might be optimal for each. The 4× efficiency gains are computed from these selected strategies, and the uncertainty in the selection process is not propagated to the efficiency estimates. A practitioner implementing this approach on a different dataset would need to recompute the optimal policy, and the small-sample noise in that recomputation could yield a substantially different policy with different efficiency characteristics.

What evidence exists in the paper. The cross-validation protocol is described in Section 3.2. The paper does not report standard errors, confidence intervals, or any measure of statistical significance for the compute-optimal curves (Figures 4, 8, 9). The predicted-difficulty curves largely overlapping with oracle curves (Figures 4, 8) provides some reassurance that the selected strategies are stable, but this does not address the fundamental sample-size concern — both predicted and oracle selection are performed on the same small splits.

Mitigation status. Not addressed. The paper acknowledges the exploration-exploitation tradeoff in difficulty estimation (Section 3.2) but does not discuss the statistical power of strategy selection. Future work on continuous or dynamic difficulty estimates might improve this, but a straightforward fix — such as using more bins with adaptive widths, or reporting bootstrap confidence intervals on the scaling curves — is not explored.

Revisions and PRM Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary axes — PRM-guided search and iterative revisions — but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates through sequential refinement, most effective on easy problems), while PRM search improves candidate selection (finding the best among generated candidates, most effective on medium problems). The paper demonstrates that both individually yield 4× efficiency gains over best-of-N (Figures 4 and 8), but the combined potential is unexplored. Applying beam search to revision model outputs — or using the PRM to guide which revision paths to pursue — could yield gains beyond either method alone. The paper also shows that the PRM trained on base model outputs underperforms on revision model outputs (Figure 15a), confirming that combining the two would require retraining the verifier on the revision model's distribution. Without this integration, the current results represent a lower bound on what a fully integrated system could achieve, and a practitioner wanting to deploy both mechanisms simultaneously has no guidance on how to do so or what gains to expect.

What evidence exists in the paper. Section 8 explicitly lists this as a gap. The difficulty-dependent analyses (Figures 3 right, 7 right) show that search helps most on medium problems while revisions help most on easy problems — suggesting the combination could yield improvements across the difficulty spectrum — but no experiment tests this.

Mitigation status. Acknowledged as future work in Section 8. No preliminary results or analysis are provided to suggest whether the combination would be additive, synergistic, or antagonistic. The paper's finding that the PRM experiences distribution shift when scoring revision outputs (Figure 15a, Appendix J) indicates that a naive combination might underperform, but the question is left entirely open.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the long-context Transformer landscape from a capability demonstration to a practical deployment paradigm. Prior to Longformer, efficient attention mechanisms were evaluated almost exclusively on autoregressive language modeling (text8, enwik8), with the implicit message that sparse attention was an interesting research direction but not yet ready to replace full self-attention in the pretrain-finetune workflows that dominated applied NLP. Longformer changes this by establishing that sparse attention can serve as a drop-in replacement for full self-attention in already-pretrained models — not just models trained from scratch — and that doing so yields consistent, task-dependent gains on downstream long-document applications.

This is not a paradigm shift in the sense of introducing a fundamentally new learning principle. The sliding window attention builds on Sparse Transformer (Child et al., 2019) and dilated CNNs (van den Oord et al., 2016), and the local + global attention pattern was rapidly adopted and extended by contemporaneous work (ETC, BigBird, GMAT). Rather, Longformer represents a reframing of the evaluation standard for long-context Transformers. By demonstrating that:

A sparse attention model can be initialized from RoBERTa's checkpoint with minimal architectural changes,
The model can be pretrained with MLM for only 65K additional steps and then finetuned on six downstream tasks spanning three task families,
The resulting model consistently outperforms RoBERTa on long-document tasks (Table 7) while setting SOTA on WikiHop (+3.6 F1) and TriviaQA (+4.0 F1),

the paper makes an implicit but forceful argument: the proper evaluation for long-context Transformers is the pretrain-finetune paradigm, not autoregressive language modeling alone. This reorientation had immediate impact — the contemporaneous works the paper acknowledges (ETC, BigBird, GMAT) all adopted pretrain-finetune evaluation alongside or instead of language modeling — and it has persisted: modern long-context models (Llama 3 with extended context, Gemini with 1M+ token windows) are evaluated primarily on downstream tasks, with language modeling serving as a diagnostic rather than the primary benchmark.

The paper also reconciles a tension between two approaches to handling long documents that had previously been treated as competitors: task-specific architectures (chunking, retrieval-reader pipelines) and efficient attention mechanisms. Longformer shows that efficient attention can eliminate the need for many of these task-specific architectures — "reducing the need for task-specific architectures" (Section 1) — by simply concatenating all available context and processing it in a single pass. This does not make retrieval-reader pipelines obsolete (they remain important for truly massive corpora where even 16K tokens is insufficient), but it resets the default: the simplest model that should be tried first is now a Longformer-style model processing the full context, with chunking or retrieval added only when the context exceeds even the efficient attention model's capacity. The HotpotQA results reinforce this: Longformer achieves 73.2 joint F1 (Table 9) with a model that is "simpler than recent SOTA models that include complex task-specific architectures," and the paper explicitly notes that incorporating entity graph information (as in the top-performing GNN-based models) could improve results further, suggesting a synthesis rather than a competition.

A subtle but important consequence of the paper's approach is that it redirects research attention from attention pattern discovery to attention pattern design. Prior work on sparse attention (Sparse Transformer, Reformer, Routing Transformer) focused on finding a general-purpose sparse pattern that approximates full attention well across all inputs, often through learning (Routing Transformer) or mathematical principles (LSH in Reformer). Longformer's introduction of task-specified global attention reframes the problem: the optimal attention pattern depends on the task, and the practitioner should specify which tokens need global reach based on what the task requires. This is a shift from learned sparsity to designed sparsity, and it opens a productive middle ground between fully dense attention (expensive but general) and fully learned sparse attention (efficient but potentially brittle). The paper's ablation showing an 8.3-point drop when removing global attention (Table 10) demonstrates that this design choice is not a minor optimization — it is the difference between a working system and one that fails.

Finally, the paper makes verifier-free, search-free long-document processing more attractive relative to retrieval-augmented generation (RAG) pipelines for settings where the full context fits within the model's window. At the time of Longformer's publication, the dominant approach for long-document QA was two-stage retrieval-reader (Clark and Gardner, 2017; Chen et al., 2017). Longformer demonstrates that for documents up to 4,096-16,384 tokens, a single-pass model with efficient attention outperforms chunking-based approaches without the cascading error risk of retrieval pipelines. This does not make retrieval obsolete — for corpora of millions of documents, retrieval remains essential — but it expands the range of problems where end-to-end processing is preferable, and the LED results (Figure 3: ROUGE-1 improves from 35.21 at 1K tokens to 46.23 at 16K tokens) quantify exactly how much additional context matters for summarization.

Follow-Up Research This Work Enables

Directly combining Longformer's efficient attention with retrieval for ultra-long corpora. The paper demonstrates that Longformer can process 4,096 tokens (pretrain-finetune) and up to 16,384 tokens (LED) in a single pass, but many real-world corpora — legal document collections, scientific literature databases, multi-book series — contain relevant context spanning hundreds of thousands of tokens. A natural follow-up would combine Longformer's efficient encoder with a retrieval step: a dense retriever selects the top-K most relevant documents or passages, and Longformer processes the concatenated retrieved set (potentially 16K+ tokens of retrieved context) in a single pass. This hybrid would inherit the best of both approaches: retrieval handles the corpus scale, while Longformer's single-pass processing avoids the chunking and cascade errors that occur when the retrieved set is itself longer than a standard Transformer's window. A concrete experiment: on a multi-document QA dataset like Qasper (scientific papers) or a multi-book narrative QA task, compare (a) a retrieval-reader with BERT on individual retrieved documents, (b) the same retriever with Longformer processing the full concatenated retrieval set, and (c) a Longformer-only model limited to whatever fits in its window. The prediction: (b) should outperform (a) on questions requiring cross-document reasoning, while matching (c) on questions answerable from a single document. The paper's WikiHop results — where Longformer outperforms the chunking baseline by 2.6 points — provide a lower bound on what the retrieval + Longformer hybrid could achieve when the relevant context is too large for the window alone.

Training position embeddings from scratch for the copied-initialization scheme to quantify information loss at partition boundaries. The paper's copy-initialization strategy — repeating RoBERTa's 512 position embeddings to cover 4,096 positions — preserves local structure at the cost of introducing artificial boundaries at positions 512, 1024, 1536, etc., where two adjacent tokens (e.g., positions 511 and 512) share the same relative embedding structure as tokens at positions 511 and 0, despite being adjacent in the sequence. The paper shows that continued pretraining largely overcomes this (BPC drops from 1.957 to 1.705 over 65K steps), but a direct measurement of the effect is missing. A concrete experiment: pretrain two Longformer models from the same RoBERTa checkpoint for the same number of steps, one with the copy-initialization strategy and one with a learned interpolation (e.g., sinusoidal position embeddings at 4,096 positions, or learned embeddings extended via a small MLP that maps [0,511] embeddings to [0,4095]). Compare not only aggregate BPC but also per-position BPC at and near the partition boundaries (positions 510-514, 1022-1026, etc.) to quantify whether the copy-initialization introduces boundary artifacts that persisted after pretraining. A negative result — if the boundary artifacts are negligible after 65K steps — would validate the copy-initialization as genuinely lossless; a positive result — if boundary positions show elevated BPC — would motivate more sophisticated initialization schemes and provide an upper bound on how much performance Longformer leaves on the table due to its initialization strategy.

Stress-testing the global attention design pattern on tasks where the "global" tokens are not obvious. The paper demonstrates global attention on tokens whose role is clear from the task structure: [CLS] for classification, question tokens for QA, <s> for summarization. But many important NLP tasks lack an obvious small set of tokens that should attend globally. For multi-label document classification where no single [CLS] token suffices, or for sequence tagging tasks (NER, relation extraction) over long documents, or for pairwise text comparison tasks (natural language inference, paraphrase detection) where both texts are long, the design of global attention is non-obvious. A systematic study would take 3-4 such tasks, propose multiple global attention configurations per task (e.g., for NER over long documents: no global attention, global attention on every 512th token as "anchor" points, global attention on a learned subset of tokens, global attention on all tokens with entity-like capitalization patterns), and measure the performance spread. The paper's coreference resolution result — where no global attention is used and performance matches RoBERTa (Table 7: 78.6 vs. 78.4) — provides one data point suggesting that local windows alone suffice when relevant context is nearby, but a systematic study would establish principles for when global attention helps versus when it's unnecessary or even harmful. A negative result (e.g., learned global token selection underperforms task-specified selection) would reinforce the paper's design-as-inductive-bias framing; a positive result (e.g., learned selection matches or exceeds task-specified) would suggest generalizing the approach beyond human design.

Extending LED with continued seq2seq pretraining to isolate the effect of attention pattern from pretraining signal. The paper's LED results are striking because LED achieves SOTA on arXiv summarization without any additional pretraining — it is "merely initialized from BART" (Section 7). This raises the question: how much of LED's advantage comes from the efficient attention pattern enabling longer inputs, and how much would additional pretraining add? A concrete experiment: continue BART's denoising pretraining on long documents (the same corpus used for Longformer's MLM pretraining, Table 13) with LED's efficient encoder attention, for a comparable number of steps (e.g., 65K), then finetune on arXiv. Compare: (a) BART with 1K input (baseline), (b) LED with 16K input, no continued pretraining (the paper's current result), (c) LED with 16K input, continued pretraining. The prediction: (c) should outperform (b), and the gap between (b) and (c) quantifies the value of pretraining signal versus architectural capacity. The paper's encoder-only result — Longformer's BPC drops from 1.957 to 1.705 with 65K steps of continued MLM pretraining (Table 5) — suggests the gap could be substantial for LED as well. A further ablation: train LED from scratch with the efficient attention pattern and the denoising objective on the long-document corpus, to determine whether BART initialization is necessary or whether the pretraining objective alone is sufficient given enough long-document training data.

Measuring the practical throughput gains of Longformer's chunked implementation versus chunking-based RoBERTa at matched accuracy. Figure 1 demonstrates Longformer's memory scaling advantage, and the paper argues that Longformer can process 4,096-token sequences with "the same amount of computation as RoBERTa" (per token). But the practical alternative to Longformer is not processing a 4,096-token sequence with full self-attention (which runs out of memory); it is processing eight 512-token chunks through RoBERTa and combining the activations. A missing experiment is a direct wall-clock and memory comparison between (a) Longformer processing a single 4,096-token sequence and (b) RoBERTa processing eight 512-token chunks sequentially, at matched batch sizes on identical hardware, measuring both total runtime and peak GPU memory. The prediction: Longformer should be faster because it avoids the overhead of running the full model forward pass eight separate times and avoids the activation concatenation step, and it should use less total memory because intermediate activations from earlier chunks don't need to be stored. Quantifying this speedup would strengthen the practical deployment argument and help practitioners estimate the cost savings of switching to Longformer for long-document pipelines. A variant: measure the throughput of Longformer processing a batch of 4,096-token documents versus RoBERTa processing a batch of 512-token chunks, since batching interacts differently with the two approaches (Longformer can batch full documents; RoBERTa must either batch chunks from different documents or process chunks sequentially).

Benchmarking the pretrain-finetune approach on non-English languages and non-RoBERTa model families. The paper's entire pretrain-finetune evaluation uses English datasets and initializes from English RoBERTa. Both the copy-initialization strategy for position embeddings and the global attention design pattern are language-agnostic, but their effectiveness depends on whether the pretrained model's attention heads exhibit the same strong locality bias that Clark et al. (2019) documented for English BERT. A multilingual replication — e.g., initializing from XLM-RoBERTa (Conneau et al., 2020), extending position embeddings via copying, continuing MLM pretraining on a multilingual long-document corpus, and evaluating on multilingual long-document tasks (e.g., MLQA for cross-lingual QA, multilingual summarization datasets) — would test the generality of the approach. Similarly, replicating with a different English pretrained model family (e.g., ELECTRA, DeBERTa) would test whether the copy-initialization depends on specific properties of RoBERTa's training (e.g., its particular position embedding structure, its pretraining data distribution). A null result — if Longformer-style continued pretraining fails to improve over the baseline for some model families — would bound the method's applicability and motivate investigation into why certain pretrained attention patterns transfer better than others.

Practical Applications and Downstream Use Cases

Long-document question answering over internal knowledge bases. Organizations with large internal document collections — legal firms with case law databases, pharmaceutical companies with research report archives, government agencies with policy documents — need QA systems that can answer questions requiring evidence distributed across an entire 50-page document or across multiple long documents. The standard approach of chunking documents into 512-token segments and running BERT on each segment fails when the answer requires combining information from distant sections (e.g., comparing the methodology in Section 2 with the results in Section 7 of a research paper). Longformer provides a drop-in upgrade: replace RoBERTa with Longformer in the existing QA pipeline, extend the input to 4,096 tokens (or 16,384 with LED), and the model can now find cross-section evidence in a single pass. The paper's WikiHop results (+2.6 accuracy points over the RoBERTa chunking baseline, Table 7) estimate the expected improvement for multi-hop QA, and the TriviaQA results (+0.9 F1) provide a more conservative estimate for single-hop QA where local context often suffices. The global attention on question tokens (described in Section 5 and Appendix D) is a configuration change, not a code change — the task-specific prediction layers remain identical to the RoBERTa baseline.

Scientific literature summarization and review generation. The LED model directly targets the use case of summarizing scientific papers, where the full text (including methods, results, and discussion) can run to 14,500+ tokens (arXiv's 90th percentile). Prior summarization models (Pegasus, BART) were limited to 1,024 input tokens, forcing them to truncate papers — often discarding the methods section entirely — which fundamentally limits summary quality for papers where key findings depend on methodological details. LED-large with 16K input tokens achieves 46.63 ROUGE-1 on arXiv (Table 11), and Figure 3 shows that expanding input from 1K to 4K to 16K monotonically improves ROUGE scores (35.21 → 44.48 → 46.23 on ROUGE-1). The practical implication: for a submission to a scientific venue, LED can read the entire paper and produce a summary informed by all sections, without the information loss that truncation-based approaches incur. The model requires only BART initialization (no additional pretraining), making it immediately deployable with available checkpoints. A deployment setup: an LED-large model behind an API that accepts full-length papers, processes them at 16K input length, and returns abstractive summaries, with the s token receiving global attention to aggregate document-level information for the decoder's cross-attention.

Coreference resolution over long documents without chunking-induced entity fragmentation. The OntoNotes coreference result (Table 7: Longformer 78.6 F1 vs. RoBERTa 78.4 F1) shows a small but positive improvement, which the paper attributes to coreference mentions typically being close together — so the chunking baseline can already connect most mentions within a chunk. However, certain coreference-heavy genres (legal documents, literary fiction with long narrative arcs, multi-party meeting transcripts) contain entity mentions separated by thousands of tokens — e.g., a character introduced on page 1 and referred to by pronoun on page 100. In these settings, chunking-based approaches either place the two mentions in different chunks (breaking the coreference chain) or require complex cross-chunk scoring that Longformer's single-pass processing eliminates. A practical deployment: replace the BERT encoder in the Joshi et al. (2019) coreference pipeline with Longformer, extend the maximum sequence length to 4,096, and evaluate on a genre-stratified coreference benchmark (e.g., the LitBank literary coreference corpus or the LEVICO legal coreference corpus) to quantify the genre-dependent benefit. The paper's finding that no global attention is needed for coreference simplifies the configuration — the sliding window alone handles the local mention-pair scoring, and the full transformer depth provides indirect long-range connectivity.

Long-document classification with limited training data. The Hyperpartisan result (Table 7: Longformer 94.8 F1 vs. RoBERTa 87.4 F1, +7.4 points) is the paper's largest relative improvement, and it occurs on the smallest dataset (645 documents total, split 80/10/10 by the authors). This suggests that Longformer's benefits are amplified in low-data regimes, likely because the model's ability to process the full document in a single pass reduces the need for the finetuning data to teach the model how to stitch together chunk-level predictions. A practical deployment: for any long-document classification task with fewer than ~1,000 labeled examples — policy document categorization, legal brief classification, grant proposal triage — Longformer with global attention on [CLS] and sequence length 4,096 is likely to substantially outperform a RoBERTa chunking baseline. The configuration is minimal (extend position embeddings via copying, add global attention to [CLS], finetune with standard classification loss), making it a low-engineering-cost upgrade over existing BERT-based classifiers.

When to Prefer This Method

The paper explicitly positions Longformer against the dominant alternatives: full self-attention (RoBERTa), chunking-based approaches, and task-specific architectures (retrieval-reader pipelines, graph networks for multi-hop QA). The choice conditions are:

Prefer Longformer over RoBERTa with chunking when: the average document length exceeds 512 tokens and the task requires integrating information across distant parts of the document. The paper's difficulty-dependent results (Table 7) show gains are largest on tasks with long contexts and cross-document reasoning (WikiHop +2.6, Hyperpartisan +7.4) and smallest on tasks with short contexts or local reasoning (IMDB +0.4, OntoNotes +0.2, where only 13.6% of documents exceed 512 tokens and mentions are locally clustered). For tasks where the document fits within 512 tokens, RoBERTa with full attention is preferable — Longformer configured identically (seqlen 512, full attention) slightly underperforms RoBERTa (71.7 vs. 72.4 on WikiHop, Table 10), so there is a small cost to using Longformer unnecessarily.
Prefer Longformer over training a sparse attention model from scratch when: a pretrained RoBERTa checkpoint is available and the target sequence length is 4,096-16,384 tokens. The paper demonstrates that continuing from RoBERTa with copied position embeddings and 65K steps of continued pretraining produces a model that matches or exceeds RoBERTa's downstream performance (Table 7), while training a sparse model from scratch would require orders of magnitude more compute and would not benefit from RoBERTa's pretrained knowledge. The frozen-weights option (Table 10: 73.5 vs. 75.0 WikiHop accuracy for the full model) provides an even lower-cost path when perfect preservation of RoBERTa's short-document behavior is required.
Prefer Longformer's local + global attention pattern over other sparse patterns (Sparse Transformer, Reformer, Routing Transformer) when: the downstream task has a natural set of tokens that should attend globally — [CLS] for classification, question tokens for QA, the start token for summarization. The paper's 8.3-point ablation gap between with and without global attention (Table 10) demonstrates that this inductive bias is not optional for tasks requiring comparison across distant input regions. When no natural global tokens exist (e.g., sequence tagging, language modeling), the dilated sliding window alone (Section 4) is the appropriate configuration, and Longformer's autoregressive LM results (Tables 2, 3) show it is competitive with other sparse patterns in that setting.
Prefer the chunked implementation over the CUDA kernel when: deploying in the pretrain-finetune setting with existing PyTorch infrastructure and sequence lengths up to 4,096. The chunked implementation is "very compute efficient" and integrated with standard PyTorch operations, with the 2× memory overhead being "not a problem for this setting" (Appendix A). The CUDA kernel via TVM is needed for the autoregressive LM setting requiring dilation support and maximum sequence lengths (up to 23,040 tokens in phase 5), but it requires custom compilation and is less portable across GPU architectures.
Prefer LED over encoder-only Longformer when: the task is sequence-to-sequence (summarization, translation, data-to-text) with long input documents but shorter outputs. The decoder's full self-attention to the encoder output is affordable because the output is short, and full cross-attention allows each decoder position to attend to any encoder position without the indirect propagation that a decoder with sparse self-attention would require. The paper's LED results (Table 11) show SOTA arXiv summarization without additional pretraining, making this the lowest-cost path to long-document seq2seq starting from BART.