LongNet: Scaling Transformers to 1,000,000,000 Tokens

🎯 Pitch

Transformers can now process 1 billion tokens at once—not by compressing or forgetting early context, but by using a new dilated attention that gives each token exponentially sparser access to distant parts of the sequence, slashing complexity from quadratic to linear with no loss in short-sequence performance.

1. Executive Summary

This paper introduces LONGNET, a Transformer variant that scales sequence length to more than 1 billion tokens without sacrificing performance on shorter sequences. The core mechanism is dilated attention — a sparse attention pattern where the attentive field expands exponentially as the distance between tokens grows (implemented as a mixture of attention heads with progressively larger segment sizes $w$ and dilation rates $r$ , such as $w = \{2048, 4096, 8192, 16384, 32768\}$ paired with $r = \{1, 2, 4, 6, 12\}$ ). The method achieves linear computation complexity $\mathcal{O}(Nd)$ and logarithmic token dependency $\mathcal{O}(\log N)$ , enabling it to scale to 1 billion tokens with nearly constant runtime on distributed systems while vanilla attention suffers quadratic growth. On language modeling with The Stack dataset using a MAGNETO backbone, LONGNET consistently outperforms both dense Transformers and sparse Transformers at matching FLOPs across sequence lengths from 2K to 32K, establishing that linear-complexity attention can match or exceed dense attention quality — but only when the dilated pattern preserves both local precision (small $w$ , small $r$ ) and global reach (large $w$ , large $r$ ) through its mixture-of-dilations design.

2. Context and Motivation

The Core Problem: Transformer Sequence Length Cannot Scale to Extreme Contexts

The fundamental problem this paper tackles is deceptively simple: standard Transformers cannot handle very long sequences. Since their introduction by Vaswani et al. (2017), Transformers have become the dominant architecture for sequence modeling across virtually every domain — language, vision, speech, code, and multimodal learning. Yet their core mechanism, self-attention, has a well-known Achilles' heel: the attention matrix computes pairwise interactions between every pair of tokens, resulting in quadratic complexity $\mathcal{O}(N^2 d)$ in both computation and memory, where $N$ is the sequence length and $d$ is the hidden dimension.

This quadratic dependency imposes a hard practical ceiling. On a single GPU device, the memory required to store and compute the full attention matrix grows as $N^2$ , meaning doubling the sequence length quadruples the memory footprint. For typical Transformer configurations with hidden dimension $d = 768$ to $d = 2048$ , the maximum feasible sequence length on modern hardware (even with optimizations like FlashAttention) sits somewhere in the tens of thousands of tokens — far short of the millions or billions that would unlock qualitatively new capabilities.

The paper crystallizes this gap in Figure 1, which traces the historical trajectory of Transformer sequence lengths across major systems: from GPT's 512 tokens (Radford et al., 2018) through Sparse Transformer's 12K tokens (Child et al., 2019), Reformer's 64K tokens (Kitaev et al., 2020), Memorizing Transformers' 262K tokens (Wu et al., 2022), and RMT's 1 million tokens (Bulatov et al., 2023). Each step represents incremental progress — a factor of approximately 4–8× per generation — but all remain orders of magnitude below what the authors argue is the ultimate target. LONGNET aims to leap directly to 1 billion tokens, a jump of roughly three orders of magnitude beyond the previous state of the art.

Why Extreme Sequence Length Matters: Three Arguments

The paper's introduction lays out three distinct arguments for why breaking the sequence length barrier is not merely an engineering curiosity but rather a qualitatively important capability with far-reaching implications.

First, it provides larger memory and receptive field for models interacting with humans and the world. This is a practical argument grounded in how real-world information is structured. Consider a model assisting with a complex codebase, a scientific literature review, or a long-form document analysis. Natural interactions involve context that spans hundreds of thousands or millions of tokens — far beyond current limits. Models operating with limited context windows must either truncate input (losing critical information) or employ retrieval-based workarounds (which introduce their own failure modes — the model can only attend to what the retriever finds, and the retriever is imperfect). An architecture that can directly attend across the full context eliminates this fundamental information bottleneck.

Second, longer context contains more complex causality and reasoning paths that models can exploit in training data, while short dependencies exhibit more spurious correlations harmful to generalization. This is a more subtle, theoretically grounded argument. When a model sees only short windows of context during training, it learns statistical patterns that correlate with correct predictions but may not reflect genuine causal relationships — for instance, predicting the next token based on the previous few sentences rather than understanding the broader document structure or argument. Extending the context window exposes the model to longer-range dependencies that are more likely to capture stable, causal patterns. This argument resonates with the broader machine learning principle that models learn spurious shortcuts when their receptive field is artificially limited, and that expanding the field forces them to learn more robust features.

Third, it enables exploration of the limits of in-context learning, with potential as a paradigm shift for many-shot learning, as an extremely long context may help models alleviate catastrophic forgetting. This is the most forward-looking argument and connects to one of the most active areas of LLM research. In-context learning — the ability of language models to adapt to new tasks from examples provided in the prompt — has emerged as a powerful alternative to fine-tuning. However, current models are limited to relatively few examples (dozens or hundreds) by the context window constraint. The authors hypothesize that scaling context to millions of tokens would allow many-shot learning, where models see thousands or tens of thousands of examples in-context. This could fundamentally change how models are deployed: rather than expensive retraining, models might acquire new capabilities purely through prompting with extensive context. The mention of catastrophic forgetting is particularly interesting — it suggests that externalized knowledge stored in context (rather than in model parameters) might provide a mechanism for avoiding the forgetting that typically occurs when models are fine-tuned on new distributions.

Taken together, these three arguments frame sequence length scaling not as incremental optimization but as an architectural constraint preventing Transformers from realizing their full potential. The authors are positioning sequence length as the "last atomic dimension" of neural network scaling that remains to be solved, following the successful scaling of depth (via residual connections and normalization schemes) and width (via sparse mixture-of-experts and model parallelism).

Prior Approaches and Where They Fall Short

The paper identifies five broad categories of prior attempts to scale Transformer sequence length, each with characteristic limitations. Understanding these provides essential context for the design choices in LONGNET.

RNN-style models (e.g., Transformer-XL by Dai et al., 2019) address the sequence length problem by introducing recurrence — processing segments sequentially and passing hidden states across segment boundaries. This reduces per-step computation to $\mathcal{O}(d^2)$ (independent of sequence length) for the recurrent component. However, the sequential nature of RNN processing limits parallelization during training, which is critical for long-sequence modeling. When training on sequences of millions of tokens, the inability to process all tokens in parallel becomes a severe bottleneck — training time scales linearly with sequence length rather than remaining constant. This is a fundamental limitation of the recurrence paradigm, not a matter of implementation optimization.

State space models (e.g., H3 by Fu et al., 2023, S4 by Gu et al., 2022, Hyena by Poli et al., 2023) represent a more recent and promising direction. These models can operate as a CNN during training (enabling parallel computation) and transform to an efficient RNN at test time (enabling fast autoregressive generation). They perform well on long-range benchmarks like the Long Range Arena (Tay et al., 2021), which specifically tests the ability to capture dependencies across thousands of tokens. However, the paper identifies a critical weakness: their performance on regular-length sequences is not as good as Transformers, limited primarily by model expressivity. This is a crucial point — state space models make structural assumptions about how information propagates through time (typically via linear dynamical systems) that are computationally efficient but may lack the flexibility of full attention. The citation to Fathi et al. (2023) suggests that this expressivity gap is an active area of concern in the literature. The implication is clear: for a model to be practically useful, it must not only handle long sequences but also match or exceed Transformer quality on the short and medium sequences that constitute the vast majority of real-world usage.

Sliding window and convolution-based attention (e.g., LogSparse Transformer by Li et al., 2019, Longformer by Beltagy et al., 2020) achieves near-linear complexity by restricting each token's attention to a fixed-size local window. This is conceptually simple and computationally attractive: if each token attends to only $k$ neighbors regardless of sequence length, the complexity becomes $\mathcal{O}(Nkd)$ , which is linear in $N$ . However, the paper identifies a fatal flaw: this sacrifices the ability to recall early tokens, effectively forgetting the prompts at the very beginning of the sequence. For tasks requiring the model to refer back to information presented early in a long context — following instructions given at the start of a conversation, checking facts against a document preamble, or maintaining consistency with an initial plan — sliding window approaches fundamentally break. Information at position 0 can only propagate to position $N$ through a chain of $N/k$ local hops, and each hop is a lossy operation (the attention mechanism must compress information into a fixed-size hidden representation that is then passed to the next window). The paper implicitly positions this as an information-theoretic problem: local connectivity alone cannot maintain global recall.

Sparse attention (e.g., Sparse Transformer by Child et al., 2019, Big Bird by Zaheer et al., 2020, Longformer by Beltagy et al., 2020, Reformer by Kitaev et al., 2020, CoLT5 by Ainslie et al., 2023) is the approach most directly related to LONGNET's dilated attention. Sparse attention reduces computation by sparsifying the attention matrix — each query only attends to a subset of keys and values defined by a sparse attention pattern $S \in \{0, 1\}^{N \times N}$ . The key insight is that if $S$ is carefully designed, a query can still access distant tokens through multiple attention hops (attend to an intermediate token which in turn attended to a distant token in a previous layer), while the per-layer computation is substantially reduced.

The Sparse Transformer (Child et al., 2019) is the canonical example. It uses two complementary fixed patterns: a local pattern where each query attends to tokens within its own block of length $l$ , and a strided pattern where each query attends to the last $c$ tokens of every block. This achieves $\mathcal{O}(N \sqrt{N} d)$ complexity — a substantial improvement over $\mathcal{O}(N^2)$ but still superlinear. The paper identifies two key limitations. First, these are fixed patterns — the sparsity structure is determined by heuristics rather than learned from data, meaning some important attention connections may be structurally excluded while unnecessary connections are retained. Second, and more critically for the scaling argument, none of these sparse attention methods have been scaled to 1 billion tokens. Even $\mathcal{O}(N \sqrt{N})$ becomes intractable at extreme lengths — with $N = 10^9$ , $N \sqrt{N} = 10^{13.5}$ , which is still computationally prohibitive.

Learnable sparse patterns (Reformer by Kitaev et al., 2020, CoLT5 by Ainslie et al., 2023) address the fixed-pattern limitation by allowing the model to learn which connections are important, typically through hashing-based similarity search. This is more expressive than fixed patterns but introduces additional complexity and does not fundamentally change the asymptotic scaling.

Other efficient Transformer variants receive briefer treatment in the paper but are important for completeness. Low-rank attention (Linformer by Wang et al., 2020) projects the key and value matrices to a fixed lower dimension, achieving linear complexity but losing fine-grained attention resolution. Kernel-based methods (Performer by Choromanski et al., 2021, Linear Transformer by Katharopoulos et al., 2020) approximate the softmax attention via kernel feature maps, trading exact computation for efficiency — but the approximation quality degrades for long sequences, and the approach can struggle with the sharp attention distributions typical in language modeling. Downsampling approaches (Set Transformer by Lee et al., 2019, Perceiver by Jaegle et al., 2021, Luna by Ma et al., 2021) compress the sequence into a smaller set of latent tokens that serve as attention bottlenecks, reducing complexity but creating an information compression step that can lose fine-grained details. Recurrent models (Transformer-XL), and retrieval-based methods (Memorizing Transformers by Wu et al., 2022, LongMem by Wang et al., 2023) augment Transformers with external memory accessed through k-nearest-neighbor retrieval, which is effective for long-range recall but adds complexity and does not provide the same end-to-end differentiability as attention.

The paper's Table 1 succinctly captures the asymptotic picture: Recurrent is $\mathcal{O}(Nd^2)$ , Vanilla Attention is $\mathcal{O}(N^2 d)$ , Sparse Attention achieves $\mathcal{O}(N \sqrt{N} d)$ , and LONGNET's Dilated Attention achieves $\mathcal{O}(Nd)$ — the first approach in this taxonomy to reach genuine linear complexity while preserving token dependency at $\mathcal{O}(\log N)$ .

How LONGNET Positions Itself

The paper positions LONGNET at the intersection of three design imperatives that prior approaches fail to simultaneously satisfy. This framing is essential for understanding the architectural choices that follow.

Imperative 1: Linear computation complexity. The method must scale to 1 billion tokens, which fundamentally requires $\mathcal{O}(Nd)$ complexity. Any superlinear dependency — even $N \log N$ or $N \sqrt{N}$ — becomes intractable at this scale. This rules out vanilla attention ( $N^2$ ) and many sparse attention variants ( $N \sqrt{N}$ ).

Imperative 2: Logarithmic token dependency. Every token must be able to access every other token within $\mathcal{O}(\log N)$ attention hops. This is what distinguishes LONGNET from sliding window approaches (which require $\mathcal{O}(N)$ hops for distant tokens) and from compression-based methods (which lose direct access entirely). The logarithmic path length ensures that even for $N = 10^9$ , any token can influence any other in roughly $\log(10^9) \approx 21$ hops — a tractable number of layers for a deep Transformer.

Imperative 3: Drop-in compatibility with existing Transformer infrastructure. This is perhaps the most pragmatically important positioning claim. The paper emphasizes that dilated attention can be implemented as a dense attention between gathered then scattered tokens (Section 2.2), meaning it can directly reuse all existing Transformer optimizations — FlashAttention kernels for memory-efficient exact attention, kernel fusion for reduced overhead, quantization for efficient inference, and distributed training frameworks. This stands in contrast to many prior approaches that require custom CUDA kernels or fundamentally different hardware utilization patterns. The claim is that LONGNET is not just an interesting theoretical architecture but is immediately deployable within the existing Transformer ecosystem.

The paper also positions itself through a distributed training lens that prior efficient attention work largely overlooked. Section 3 directly addresses the reality that even $O(Nd)$ computation on a single GPU cannot handle billion-token sequences — the hidden dimension $d$ is typically hundreds or thousands, so $10^9 d$ operations per attention head is still enormous. The key insight is that the linear complexity structure enables clean parallelization across the sequence dimension with constant communication cost (Section 3.1). In contrast, vanilla attention's quadratic structure means that sequence parallelism is fundamentally bottlenecked by the full attention matrix, which cannot be cleanly partitioned without expensive all-to-all communication. This distributed training capability is not merely an optimization detail — it is what makes the 1 billion token claim realizable on actual hardware, as demonstrated in Figure 5 where dilated attention shows nearly constant runtime from 8K to 1B tokens while vanilla attention explodes.

Finally, the paper positions its contribution historically. Figure 1 places LONGNET at the endpoint of a sequence length scaling trajectory spanning from 2017 to 2023, with a jump from the previous state of the art (1M tokens with RMT) to 1B tokens — a 1000× improvement in a single step. This framing establishes LONGNET not as an incremental refinement but as a qualitative breakthrough in achievable context length, opening up fundamentally new use cases that were previously inaccessible. The concluding phrase — "treating a whole corpus or even the entire Internet as a sequence" — is audacious but logically follows from the claimed capability: at roughly 1.5 tokens per word and 100 billion words, the English-language Internet would require roughly 150 billion tokens, which is within the same order of magnitude as LONGNET's demonstrated 1 billion token capability and theoretically tractable with further engineering.

The Unresolved Tension at the Heart of the Problem

A key intellectual contribution of the paper's motivation section is the explicit identification of a fundamental tension that prior work failed to fully articulate. Any attempt at scaling sequence length must simultaneously satisfy two constraints that pull in opposite directions:

Computational feasibility demands that each token attend to as few other tokens as possible — ideally a constant number, achieving $O(N)$ complexity.
Model expressivity demands that each token be able to attend to any other token — ideally all of them, which is $O(N^2)$ and infeasible.

Prior approaches resolved this tension by choosing one side: sliding windows prioritized computation but sacrificed global recall (favoring feasibility over expressivity), while sparse attention with fixed patterns tried to strike a middle ground but ended up with superlinear complexity and insufficient scaling.

The paper's framing of dilated attention is that it transcends this tension rather than compromising. The exponentially expanding dilation pattern (using geometric sequences for both segment sizes $w$ and dilation rates $r$ , with Equations 11–12 establishing $w$ and $r$ as geometric series) means that tokens at close distances are densely attended (satisfying expressivity for local context, where most relevant information typically resides), while tokens at long distances are sparsely but exponentially covered (satisfying the need for global access while keeping total computation linear). The logarithmic dependency $\mathcal{O}(\log N)$ derived in Equation 20 is the mathematical manifestation of this balance — it shows that the maximum path between any two tokens grows only logarithmically with sequence length, which is the best possible scaling for any architecture that does not maintain a full dense attention matrix.

This resolution of the feasibility-expressivity tension is the intellectual core of the contribution, and the rest of the paper — the dilated attention mechanism, the mixture of dilations, the distributed training algorithm, and the empirical validation — is the concrete embodiment of this resolution.

3. Technical Approach

3.1 Reader Orientation

LONGNET is a modified Transformer architecture where the standard self-attention mechanism is replaced with dilated attention — a family of sparse attention patterns that use exponentially growing gaps between attended tokens, analogous to how dilated convolutions expand receptive fields in CNNs. The system solves the problem that vanilla self-attention costs $\mathcal{O}(N^2)$ to compute and store, which makes sequences beyond ~100K tokens impossible on current hardware, by instead structuring attention so that nearby tokens are densely connected while distant tokens are reached through exponentially sparse sampling, achieving $\mathcal{O}(N d)$ computation and $\mathcal{O}(\log N)$ maximum path length between any two tokens.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components working in concert:

Input segmentation and projection — the incoming sequence of $N$ tokens with hidden dimension $d$ is projected into queries $Q$ , keys $K$ , and values $V$ (standard linear projections), then logically partitioned along the sequence dimension into segments of varying sizes $w$ .
Multi-resolution dilated attention heads — a mixture of $k$ attention patterns operating in parallel, where pattern $i$ uses segment size $w_i$ and dilation rate $r_i$ . Small $w_i$ with $r_i = 1$ captures fine-grained local context; large $w_i$ with large $r_i$ captures long-range context by sampling tokens at exponentially increasing intervals. Each pattern sparsifies its input by selecting every $r_i$ -th token within each segment, computes standard scaled dot-product attention on the sparsified tensors, then scatters results back to the original positions. Patterns are combined via a weighted sum where weights $\alpha_i$ are proportional to the softmax denominator of each pattern.
Multi-head offset mechanism — different attention heads within the same dilation configuration apply different starting offsets $s_j$ when selecting which tokens to sparsify, ensuring that across heads the union of attended positions covers the full sequence rather than always skipping the same tokens.
Distributed sequence parallelism — when $w_i$ exceeds the local sequence length on a single device, keys and values are gathered across devices via an all-gather operation. Because the sparsified $\tilde{K}$ and $\tilde{V}$ have size independent of $N$ (they depend only on $w_i / r_i$ ), the communication cost is constant regardless of total sequence length. Queries remain local; cross-attention is computed between local queries and global keys/values.

Information flows as: input sequence → split across devices → project to Q/K/V → for each dilation pattern $(w_i, r_i)$ and each head offset $s_j$ : sparsify Q/K/V by selecting rows at intervals $r_i$ starting from offset $s_j$ → if $w_i$ exceeds local length, all-gather K and V → compute attention on the gathered tensors → scatter outputs back to original positions → sum across patterns with dynamic weights $\alpha_i$ → concatenate across heads → output.

3.3 Roadmap for the Deep Dive

First, the formal definition of dilated attention (Equations 3–12), because the entire method rests on how sparsification, segment sizing, and dilation rates interact to produce linear complexity with logarithmic dependency.
Second, the mixture-of-dilations mechanism and the dynamic weighting scheme (Equations 9–12), since using a single dilation rate would either lose local precision (large $r$ ) or fail to reach distant tokens (small $r$ ) — the mixture is what resolves this tension.
Third, the multi-head offset mechanism (Equations 13–15), which ensures that across attention heads the union of sparsified positions covers the full sequence, preventing systematic blind spots.
Fourth, the complexity and dependency analysis (Equations 16–20), which proves that the geometric progression of $w$ and $r$ yields $\mathcal{O}(Nd)$ FLOPs and $\mathcal{O}(\log N)$ maximum path length — the paper's central theoretical claim.
Fifth, the distributed training algorithm (Equations 21–25), which explains how linear complexity enables sequence parallelism with constant communication cost, making the 1B token claim realizable on actual multi-GPU systems.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core idea is that exponentially dilated sparse attention patterns can achieve linear complexity while maintaining logarithmic token dependency, and that a mixture of such patterns with different dilation rates simultaneously preserves local precision and global reach — resolving the fundamental feasibility-expressivity tension that limited all prior sequence length scaling approaches.

Standard Self-Attention and Sparse Attention (Preliminaries)

The paper begins by restating the standard self-attention formulation to establish notation and make the transition to dilated attention explicit.

Vanilla self-attention. Given input matrices $Q, K, V \in \mathbb{R}^{N \times d}$ representing queries, keys, and values respectively, standard scaled dot-product attention computes:

$O = \text{softmax}(Q K^T) V$

where $Q$ is the query matrix, $K$ is the key matrix, $V$ is the value matrix, $N$ is the sequence length, and $d$ is the hidden dimension per head.

What it computes: every query token (row of $Q$ ) computes an attention score against every key token (row of $K$ ) via dot product, applies softmax to obtain a probability distribution over all $N$ positions, then uses that distribution to compute a weighted sum of all $N$ value vectors. Each of the $N$ output positions depends on all $N$ input positions.

Why this is problematic: the matrix multiplication $QK^T$ produces an $N \times N$ matrix, requiring $\mathcal{O}(N^2 d)$ FLOPs and $\mathcal{O}(N^2)$ memory. For $N = 1$ billion tokens and $d = 768$ , this is approximately $7.68 \times 10^{20}$ FLOPs per attention layer — completely intractable on any foreseeable hardware. Even memory alone fails: storing a single $10^9 \times 10^9$ attention matrix in float32 requires $4 \times 10^{18}$ bytes = 4 exabytes.

Sparse attention generalizes this by restricting each query to attend to only a subset of keys. Formally, a binary sparsity mask $S \in \{0, 1\}^{N \times N}$ is introduced:

$O = \text{softmax}(Q K^T \odot \mathbf{1}_S) V$

where $S_{i,j} = 1$ if query $i$ is allowed to attend to key $j$ , and $\mathbf{1}_S$ is the indicator (setting masked-out entries to $-\infty$ before softmax, making their attention weights zero).

What it computes: exactly the same operation as vanilla attention, but with most attention scores forcibly set to $-\infty$ before softmax, meaning those positions receive zero weight. The output at each position depends only on the subset of keys permitted by $S$ .

Why this form matters as a starting point: the sparsity mask $S$ encodes the entire design space of efficient attention. Every method — sliding windows, strided patterns, block-sparse, hashing-based, low-rank — corresponds to a different choice of $S$ . The challenge is designing $S$ such that (a) it is sparse (few non-zero entries, giving computational efficiency), (b) it connects every query to every key either directly or through a short path of intermediate tokens (giving expressivity), and (c) it can be implemented efficiently on GPU hardware (giving practical speed).

The Sparse Transformer's fixed pattern (Child et al., 2019), which the paper uses as a baseline, is defined by two complementary masks. The local pattern allows query $i$ to attend to keys within its own block: $S^{(1)}_i = \{j \mid \lfloor j / l \rfloor = \lfloor i / l \rfloor\}$ , where $l$ is the block size. The strided pattern allows query $i$ to attend to the last $c$ tokens of every block: $S^{(2)}_i = \{j \mid j \bmod l \in \{t, t+1, \dots, l-1\}\}$ . Together they achieve $\mathcal{O}(N \sqrt{N} d)$ complexity — better than quadratic but still superlinear and insufficient for billion-token scales.

Dilated Attention: Core Mechanism

Dilated attention is the central architectural contribution. Instead of a single sparsity pattern $S$ , it uses a parameterized family of patterns defined by two numbers per pattern: a segment length $w$ and a dilation rate $r$ .

The sparsification operation. For a single dilation configuration $(w, r)$ , the input sequence is first divided into segments of length $w$ along the sequence dimension. Within each segment, instead of keeping all $w$ positions, only every $r$ -th position is retained. Formally, for the $i$ -th segment:

$\tilde{Q}_i = [Q_{iw}, Q_{iw+r}, Q_{iw+2r}, \dots, Q_{(i+1)w-1}]$

$\tilde{K}_i = [K_{iw}, K_{iw+r}, K_{iw+2r}, \dots, K_{(i+1)w-1}]$

$\tilde{V}_i = [V_{iw}, V_{iw+r}, V_{iw+2r}, \dots, V_{(i+1)w-1}]$

where $\tilde{Q}_i, \tilde{K}_i, \tilde{V}_i \in \mathbb{R}^{\frac{w}{r} \times d}$ are the sparsified query, key, and value matrices for segment $i$ , $w$ is the segment length, $r$ is the dilation rate (the stride between selected positions), $i$ indexes the segment starting at position $iw$ , and the notation $Q_{iw}$ means the row of $Q$ at absolute sequence position $iw$ .

What these equations compute: given a contiguous chunk of $w$ tokens starting at position $iw$ , the sparsification discards all but every $r$ -th token within that chunk. If $w = 16$ and $r = 4$ , positions $\{iw, iw+4, iw+8, iw+12\}$ are kept (4 positions from 16), and the other 12 are dropped from this particular attention computation. The operation is applied independently to queries, keys, and values, producing reduced-size tensors for each of the $N/w$ segments (approximately, accounting for boundary effects).

What happens next — attention on sparsified tensors. Within each segment, standard scaled dot-product attention is computed on the reduced tensors:

$\tilde{O}_i = \text{softmax}(\tilde{Q}_i \tilde{K}_i^T) \tilde{V}_i$

where $\tilde{O}_i \in \mathbb{R}^{\frac{w}{r} \times d}$ is the attention output for segment $i$ , computed from the sparsified queries, keys, and values.

What this computes: each surviving query position (one of the $w/r$ selected positions in segment $i$ ) attends to all surviving key positions within the same segment. The attention matrix is therefore $\frac{w}{r} \times \frac{w}{r}$ rather than $w \times w$ — a reduction by a factor of $r^2$ in FLOPs per segment.

Why sparsify queries too (not just keys)? This is a crucial design choice. If only keys and values were sparsified (keeping full queries), the attention matrix would be $w \times \frac{w}{r}$ , and the output would have $w$ rows — requiring the scattering step to fill in every position. By sparsifying queries as well, the output $\tilde{O}_i$ has only $\frac{w}{r}$ rows, and the positions that were skipped produce no output from this particular dilation pattern. This is what enables the linear complexity: the number of output positions produced is reduced by the same factor $r$ as the keys. The "missing" positions will be covered by other dilation patterns in the mixture (with different offsets or different $r$ values) or by other attention heads.

The scattering operation. After computing attention on the sparsified tensors, the results must be mapped back to the original sequence positions:

$\hat{O}_i = \{\tilde{O}_{i,j} \mid j \bmod r = 0; \; \mathbf{0} \mid j \bmod r \neq 0\}$

$O = [\hat{O}_0, \hat{O}_1, \dots, \hat{O}_{\frac{N}{w}-1}]$

where $\hat{O}_i \in \mathbb{R}^{w \times d}$ is the segment output expanded back to full size (with zeros at positions that were skipped during sparsification), and $O \in \mathbb{R}^{N \times d}$ is the concatenated full-sequence output.

What this computes: for each segment, the $\frac{w}{r}$ computed output vectors are placed back at their original positions (where $j \bmod r = 0$ ), and all other positions receive the zero vector. This means a single dilated attention pattern with parameters $(w, r)$ leaves $\frac{r-1}{r}$ of output positions as zero — it only computes updates for $1/r$ of the tokens.

Why zeros are acceptable: the zero positions will be filled by other dilated attention patterns in the mixture, or by the same pattern with different head offsets. No position is permanently ignored because the mixture of patterns collectively covers all positions. The zeros act as additive identity elements — when outputs from multiple patterns are summed, the zero entries contribute nothing and the computed entries from other patterns fill the gaps.

Implementation as gather-scatter dense attention. The paper notes that this entire procedure (Equations 3–8) can be implemented as: (1) a gather operation that extracts the selected rows from $Q$ , $K$ , and $V$ ; (2) standard dense attention on the gathered tensors; (3) a scatter operation that writes results back to the corresponding positions in the output. This is the basis for the claim that dilated attention is a "drop-in replacement" for standard attention — the core computation is still dense attention, just on a strategically subsampled subset of positions.

Mixture of Dilated Attentions

A single dilation configuration $(w, r)$ cannot simultaneously satisfy the need for local precision and global reach. If $r = 1$ (no dilation, every token within each segment is kept), the attention is exact within segments of size $w$ , but tokens in different segments never directly interact — global information requires stacking many layers, and the receptive field grows only linearly with depth. If $r$ is large, tokens can interact across large distances (a segment of size $w$ with dilation $r$ spans positions covering a range of $w \times r$ ), but local precision is lost because adjacent tokens are often skipped.

The solution is a mixture: run $k$ different dilated attention patterns in parallel with different $(w_i, r_i)$ parameters, then combine their outputs via a weighted sum:

$O = \sum_{i=1}^{k} \alpha_i \, O|_{r_i, w_i}$

where $O|_{r_i, w_i}$ is the output of dilated attention with segment size $w_i$ and dilation rate $r_i$ , and $\alpha_i$ is a scalar weight for pattern $i$ .

Dynamic weight computation. The weights $\alpha_i$ are computed from the softmax denominators of each pattern:

$\alpha_i = \frac{s_i}{\sum_{j} s_j}$

where $s_i$ is the sum of exponentiated attention scores (the softmax denominator) for pattern $i$ 's attention computation.

What this computes: each pattern produces its own attention output with its own softmax normalization. The denominator $s_i = \sum \exp(\text{score})$ measures the total "mass" of the attention distribution for that pattern — roughly, how confident or peaked the attention is. The weight $\alpha_i$ normalizes these masses across patterns, so patterns with sharper attention distributions (larger softmax denominators) receive proportionally more weight in the final output.

Why dynamic weights rather than learned fixed weights: the paper states that "dynamic weights calculated by the denominator of the attention softmax are better than learnable fixed weights." This makes intuitive sense because the importance of different dilation patterns depends on the specific input. A pattern that finds strong key-query matches (producing a peaked softmax with a large denominator) is clearly more relevant for that particular token and should be weighted more heavily. A pattern that finds only weak matches (producing a flat softmax with a small denominator) contributes less useful information. Learned fixed weights cannot adapt to input-dependent variation in pattern relevance. The softmax denominator is a natural, computationally-free measure of pattern quality because it is already computed during the attention operation itself.

The equivalence to gathering keys across patterns. The paper notes an important equivalence: "For a query attends to keys in different dilated attentions, our method to mix dilated attentions is equivalent to gather keys in different parts and calculate softmax together." In other words, you can think of the mixture as: collect all the key positions that would be attended under any of the $k$ patterns, compute one big attention over this union of keys, and then combine. The weighted-sum formulation with $\alpha_i \propto s_i$ is algebraically equivalent to this — the softmax denominators serve as the normalization factors that make the piecewise softmax computation match a single joint softmax over the union of all attended positions.

Geometric progression of segment sizes and dilation rates. The critical design choice is how to select the $k$ pairs $(w_i, r_i)$ . The paper prescribes geometric sequences:

$w = \{w_0, w_1, w_2, \dots, N\}^k \quad (w_i < w_{i+1} < N)$

$r = \{1, r_1, r_2, \dots, r_k\}^k \quad (1 < r_i < r_{i+1})$

where $w_0$ is a predefined constant (typically 2048 in experiments), and both sequences grow geometrically — each subsequent $w_i$ is multiplied by a constant factor $\alpha > 1$ , and each subsequent $r_i$ is similarly multiplied by $\alpha$ .

What these sequences encode: the first pattern $(w_0, r=1)$ is a fine-grained local attention — all tokens within windows of size $w_0$ attend to each other with no dilation, providing exact local computation. The second pattern $(w_1 = \alpha w_0, r_1 = \alpha)$ doubles (if $\alpha = 2$ ) both the window size and the dilation — each segment now spans $2w_0$ tokens, but only every $2$ nd token is kept, so the attended set ( $w_1 / r_1 = w_0$ tokens per segment) has the same size as pattern 1, but those tokens are spread over twice the range. The third pattern $(w_2 = \alpha^2 w_0, r_2 = \alpha^2)$ quadruples both, and so on, until $w_k \approx N$ covers the full sequence.

Why geometric progression matters: this is the mathematical property that produces logarithmic token dependency. With a constant ratio $\alpha$ , the number of patterns needed to reach sequence length $N$ is $k = \lceil \log_\alpha (N / w_0) \rceil$ . For $N = 10^9$ , $w_0 = 2048$ , and $\alpha = 2$ , this gives $k \approx \log_2(10^9 / 2048) \approx 19$ patterns — a small constant. Each pattern adds a roughly equal amount of computation (since $w_i / r_i \approx w_0$ for all patterns), so total computation is $\mathcal{O}(k N d) = \mathcal{O}(N d \log N)$ — but with the geometric sum argument in Equation 18, this tightens to $\mathcal{O}(N d)$ because the sum $\sum w_i / r_i^2$ converges.

The attentive field expands exponentially. A token in pattern $i$ can directly attend to other tokens within its segment, which spans $w_i$ positions. Because the dilation $r_i$ skips intermediate tokens, the maximum distance between two tokens that can directly interact is approximately $w_i \times r_i$ . As $i$ increases, this product grows as $\alpha^{2i}$ , giving exponentially expanding receptive fields. Pattern 1 covers local context (~2048 tokens), pattern 2 covers ~8192 tokens, pattern 3 covers ~32768 tokens, and so on, with pattern $k$ covering the full sequence length $N$ . This is directly analogous to how dilated convolutions in CNNs (e.g., Yu and Koltun, 2016) achieve exponentially expanding receptive fields without increasing the number of parameters or the per-layer computation — a connection the paper's name ("dilated attention") explicitly invokes.

In practice (experimental configuration): the paper uses $w = \{2048, 4096, 8192, 16384, 32768\}$ and $r = \{1, 2, 4, 6, 12\}$ for the language modeling experiments in Section 4. Note that the dilation rates are not exactly a perfect geometric progression — $r_3 = 6$ rather than $8$ , and $r_4 = 12$ rather than $16$ — suggesting some empirical tuning or constraint from matching FLOPs with baselines. The segment lengths $w$ do follow an exact geometric progression with $\alpha = 2$ , starting from $w_0 = 2048$ and going up to $w_4 = 32768$ (matching the maximum sequence length of 32K in the experiments).

Multi-Head Dilated Attention: Offset Mechanism

Standard multi-head attention runs the same attention operation multiple times in parallel with different learned projection matrices, allowing different heads to specialize in different types of relationships. With dilated attention, there is an additional dimension of variation: which specific tokens are sparsified.

If every head used the same sparsification (starting from position $iw$ and selecting every $r$ -th token), then across all heads the same $1/r$ fraction of positions would be computed and the same $(r-1)/r$ fraction would always be zero. The union of attended positions across heads would be identical to a single head — no benefit from multi-head computation.

The solution is to offset the starting position differently for each head. For the $j$ -th head (where $j \in \{0, 1, \dots, h-1\}$ and $h$ is the number of heads), the sparsification uses an offset $s_j = j \bmod r$ :

$\tilde{Q}_i = [Q_{iw + s_j}, Q_{iw + s_j + r}, Q_{iw + s_j + 2r}, \dots, Q_{(i+1)w + s_j - 1}]$

$\tilde{K}_i = [K_{iw + s_j}, K_{iw + s_j + r}, K_{iw + s_j + 2r}, \dots, K_{(i+1)w + s_j - 1}]$

$\tilde{V}_i = [V_{iw + s_j}, V_{iw + s_j + r}, V_{iw + s_j + 2r}, \dots, V_{(i+1)w + s_j - 1}]$

where $s_j$ is the starting offset for head $j$ , computed as $j$ modulo $r$ , and all other notation is as before.

What these equations compute: head 0 starts sparsifying from position $iw + 0$ , selecting positions $\{iw, iw+r, iw+2r, \dots\}$ . Head 1 starts from $iw + 1$ , selecting $\{iw+1, iw+1+r, iw+1+2r, \dots\}$ . Head $r-1$ starts from $iw + r - 1$ , selecting $\{iw+r-1, iw+2r-1, iw+3r-1, \dots\}$ . Across all $r$ possible offsets, every position in the segment is selected by exactly one head. If there are more than $r$ heads ( $h > r$ ), offsets wrap around modulo $r$ , so multiple heads share the same offset — they differ only in their learned projection matrices, as in standard multi-head attention.

Why this matters: the offset mechanism ensures that the union of all heads' attended positions covers the full sequence. No token position is permanently excluded from attention computation — for any given segment and any given position within that segment, there exists at least one head (specifically, the head with $s_j$ equal to that position modulo $r$ ) that includes it in the sparsified set. This eliminates the systematic blind spots that a single dilation pattern would create.

Interaction with the mixture of dilations. The offset mechanism operates independently for each dilation pattern in the mixture. For a given pattern $(w_i, r_i)$ , each head uses an offset $s_j = j \bmod r_i$ . Since different patterns have different dilation rates, the number of distinct offsets varies: pattern 1 ( $r = 1$ ) has only one offset (all heads attend to the same positions — but that's fine because it's local dense attention within each segment), pattern 2 ( $r = 2$ ) has two offsets, pattern 3 ( $r = 4$ ) has four offsets, and so on. Heads are distributed across these offsets, ensuring coverage at every dilation scale.

Output concatenation. As in standard multi-head attention, the outputs of different heads are concatenated along the hidden dimension to form the final output. The computation within each head follows the single-head procedure exactly (Equations 3–8), with the only difference being the offset $s_j$ in the sparsification step.

Computational Complexity and Token Dependency Analysis

The paper provides a formal analysis proving two central claims: (1) dilated attention has linear complexity $\mathcal{O}(Nd)$ , and (2) the maximum path length between any two tokens grows logarithmically with sequence length, $\mathcal{O}(\log N)$ .

FLOPs for a single dilation configuration. For a single pattern with parameters $(r, w)$ , the sparsified tensors $\tilde{Q}, \tilde{K}, \tilde{V}$ each have dimensions $\frac{w}{r} \times d$ . There are $\frac{N}{w}$ segments. The attention computation for each segment involves two matrix multiplications: $\tilde{Q}_i \tilde{K}_i^T$ which costs $2 \cdot \frac{w}{r} \cdot d \cdot \frac{w}{r}$ FLOPs (the factor of 2 accounts for multiply-add), and the subsequent multiplication with $\tilde{V}_i$ which costs another $2 \cdot \frac{w}{r} \cdot d \cdot \frac{w}{r}$ FLOPs. Summing across segments:

$\text{FLOPs} = 2 N \frac{w}{w} \left( \frac{w}{r} \right)^2 d = 2 N \frac{w d}{r^2}$

where the factor 2 comes from the two matrix multiplications (attention scores and value aggregation), $N/w$ is the number of segments, $(w/r)^2 d$ is the FLOPs per segment for the attention matrix multiplication, and the second matrix multiply contributes an equal amount.

Why two matrix multiplies have the same FLOPs count: the attention score computation $\tilde{Q} \tilde{K}^T$ produces a $\frac{w}{r} \times \frac{w}{r}$ matrix, requiring $\frac{w}{r} \cdot d \cdot \frac{w}{r}$ multiply-adds. The value aggregation multiplies the $\frac{w}{r} \times \frac{w}{r}$ attention matrix with $\tilde{V}$ of shape $\frac{w}{r} \times d$ , also requiring $\frac{w}{r} \cdot \frac{w}{r} \cdot d$ multiply-adds. Both are symmetric in FLOPs, hence the factor of 2.

FLOPs for the mixture of $k$ patterns. Summing the per-pattern FLOPs across all $k$ patterns in the mixture:

$\text{FLOPs} = 2 N d \sum_{i=1}^{k} \frac{w_i}{r_i^2}$

where $w_i$ and $r_i$ are the segment size and dilation rate for pattern $i$ , and the sum is over all $k$ patterns.

Plugging in the geometric sequences. With the geometric progression $w_i = w_0 \alpha^{i-1}$ and $r_i = \alpha^{i-1}$ (where $r_1 = 1$ , so the first pattern has no dilation), the ratio $w_i / r_i^2$ becomes:

$\frac{w_i}{r_i^2} = \frac{w_0 \alpha^{i-1}}{\alpha^{2(i-1)}} = \frac{w_0}{\alpha^{i-1}}$

This is a decreasing geometric series with common ratio $1/\alpha$ . Summing over $i = 1$ to $k$ (where $k$ is chosen so that $w_k \approx N$ ):

$\sum_{i=1}^{k} \frac{w_0}{\alpha^{i-1}} = w_0 \sum_{i=0}^{k-1} \left(\frac{1}{\alpha}\right)^i = w_0 \frac{1 - \alpha^{-k}}{1 - \alpha^{-1}}$

For $\alpha > 1$ and $k \to \infty$ , this sum converges to $w_0 \cdot \frac{1}{1 - 1/\alpha} = w_0 \cdot \frac{\alpha}{\alpha - 1}$ . Even for finite $k$ , the sum is bounded by this limit. Therefore:

$\text{FLOPs} = 2 N d w_0 \sum_{i=0}^{k-1} \frac{1}{\alpha^i} \leq 2 N d w_0 \frac{\alpha}{\alpha - 1} = \mathcal{O}(N d)$

where $w_0$ and $\alpha$ are fixed constants independent of $N$ , and the inequality holds for any $\alpha > 1$ .

What this means concretely: the total FLOPs grows linearly with sequence length $N$ . Doubling $N$ doubles the computation — no hidden superlinear factor. The constant factor is $2 w_0 d \cdot \frac{\alpha}{\alpha - 1}$ . For $\alpha = 2$ and $w_0 = 2048$ , this constant is $2 \cdot 2048 \cdot d \cdot 2 = 8192 d$ . For $d = 768$ , this is approximately $6.3 \times 10^6$ FLOPs per token per attention layer — a fixed per-token cost regardless of whether $N = 1000$ or $N = 10^9$ .

Why the sum converges: the key mathematical property is that $w_i / r_i^2$ decreases geometrically. The first pattern (small $w$ , small $r$ ) contributes the most per-token FLOPs because it does dense attention within local windows. Each subsequent pattern adds less computation per token because the dilation rate $r_i$ grows faster than the segment size $w_i$ — specifically, $r_i$ grows quadratically in the denominator ( $r_i^2$ ) while $w_i$ grows only linearly in the numerator. The total is bounded by a constant multiple of the first term.

Token dependency — maximum path length. The paper also analyzes how many attention hops are needed for information to travel from any token to any other token. Within a single pattern $(w_i, r_i)$ , a token can directly attend to other tokens in its segment, spanning a distance of up to $w_i$ positions (the segment length). Across one layer, the maximum distance information can travel is:

$D = \sum_{i=0}^{\ell-1} w_i = w_0 \sum_{i=0}^{\ell-1} \alpha^i \approx \frac{w_0}{\alpha - 1} \alpha^\ell$

where $\ell$ is the number of layers (the propagated path length), and the sum is over the segment sizes of patterns used across layers. Inverting this relationship to find the number of layers needed to span a sequence of length $N$ :

$L \approx \log_\alpha \frac{N (\alpha - 1)}{w_0} = \mathcal{O}(\log N)$

where $L$ is the minimum number of layers required for any token to potentially influence any other token, $N$ is the sequence length, $\alpha$ is the geometric ratio, and $w_0$ is the base segment size.

What this proves: even for a sequence of $N = 10^9$ tokens, with $\alpha = 2$ and $w_0 = 2048$ , the required number of layers is $L \approx \log_2(10^9 \cdot 1 / 2048) \approx \log_2(488281) \approx 19$ layers. This is well within the range of practical Transformer depths (the paper uses 12–32 layers). Information can propagate from the first token to the last token in roughly 19 hops — each layer approximately doubles the effective receptive field radius, so the receptive field grows exponentially with depth.

Why logarithmic dependency is the crucial theoretical property: it distinguishes dilated attention from both sliding window approaches (which require $\mathcal{O}(N)$ hops — linear in sequence length, meaning 1 billion tokens would require ~500 million layers to connect the first and last tokens) and from compression-based approaches (which lose the ability for arbitrary token-to-token communication entirely). Logarithmic dependency means the effective connectivity graph of the Transformer remains a small-world network regardless of sequence length — any two nodes are connected by a short path — which is essential for learning long-range dependencies.

The connection to small-world networks is implicit but important: in graph theory, a small-world network is one where the average shortest path length between nodes grows logarithmically with the number of nodes. The paper's token dependency analysis proves that the attention connectivity graph induced by dilated attention is a small-world network, with the mixture of local dense connections (small $w$ , small $r$ ) and long-range sparse connections (large $w$ , large $r$ ) playing the role of local clustering and random long-range edges respectively.

Distributed Training Algorithm

Even with $\mathcal{O}(Nd)$ complexity, a billion-token sequence with hidden dimension $d$ cannot fit on a single GPU. The paper therefore presents a distributed algorithm that parallelizes computation across the sequence dimension, leveraging the linear complexity structure to achieve constant communication cost per device regardless of sequence length.

The core challenge: standard data parallelism splits the batch dimension (different sequences on different devices), model parallelism splits the hidden dimension (different neurons on different devices), and pipeline parallelism splits the layer dimension (different layers on different devices). None of these address the problem of a single extremely long sequence that cannot fit in one device's memory. Sequence parallelism is needed, but standard attention's quadratic nature makes sequence parallelism expensive — each device would need to communicate its full keys and values to every other device, resulting in $\mathcal{O}(N d)$ communication per device pair, or $\mathcal{O}(P N d)$ total communication for $P$ devices.

LONGNET's distributed algorithm. The input sequence $X \in \mathbb{R}^{N \times d}$ is partitioned along the sequence dimension across $P$ devices (for simplicity, the paper describes the $P=2$ case, with the generalization being straightforward):

$X = [X_1, X_2]$

where $X_1, X_2 \in \mathbb{R}^{N/2 \times d}$ are contiguous halves of the sequence on device 1 and device 2 respectively.

Step 1 — Independent projection. Each device independently projects its local sequence chunk into queries, keys, and values using the shared weight matrices:

$[Q_1, K_1, V_1] = [W_Q, W_K, W_V] X_1$

$[Q_2, K_2, V_2] = [W_Q, W_K, W_V] X_2$

where $W_Q, W_K, W_V \in \mathbb{R}^{d \times d}$ are the learned projection matrices (shared across devices via standard data-parallel synchronization). This step requires no communication — it's purely local computation.

What this computes: after projection, device 1 holds the query, key, and value matrices for the first half of the sequence, and device 2 holds them for the second half. Each device's local matrices have shape $(N/2) \times d$ .

Step 2 — Case split based on segment length. For each dilation pattern with segment length $w_i$ :

If $w_i \leq \ell$ (where $\ell = N/P$ is the local sequence length per device): the attention can be computed entirely locally. All keys and values that queries in segment $w_i$ need to attend to reside on the same device, because the segment is smaller than the per-device chunk. No communication is needed. This is the case for the smaller dilation patterns (e.g., $w_0 = 2048$ when $\ell = 16384$ on each of two devices for a 32K sequence).
If $w_i > \ell$ : the segment spans across device boundaries. A query on device 1 may need to attend to keys and values on device 2 (and vice versa). Communication is required.

Step 3 — Sparsification (local) and all-gather (cross-device). First, each device sparsifies its local $Q$ , $K$ , and $V$ using Equations 3–5, producing $\tilde{Q}_1, \tilde{K}_1, \tilde{V}_1$ on device 1 and $\tilde{Q}_2, \tilde{K}_2, \tilde{V}_2$ on device 2. Then, for the cross-device case, an all-gather operation collects the sparsified keys and values:

$\tilde{K} = [\tilde{K}_1, \tilde{K}_2], \quad \tilde{V} = [\tilde{V}_1, \tilde{V}_2]$

where $\tilde{K}, \tilde{V}$ are now the global sparsified key and value tensors, replicated on both devices after the all-gather.

Critical property — constant communication size. Because $\tilde{K}_i$ and $\tilde{V}_i$ have dimensions $\frac{w_i}{r_i} \times d$ (independent of sequence length $N$ ), the amount of data communicated per device per pattern is $\frac{w_i}{r_i} d$ elements. With the geometric progression $w_i / r_i \approx w_0 / \alpha^{i-1}$ , the total communication across all patterns is:

$\text{Communication} = 2 d \sum_{i: w_i > \ell} \frac{w_i}{r_i} \approx 2 d w_0 \sum_{i: w_i > \ell} \frac{1}{\alpha^{i-1}}$

where the sum is only over patterns where $w_i$ exceeds the local sequence length $\ell$ . This sum is bounded by the same geometric series argument, giving constant total communication per device regardless of $N$ (it depends only on $d$ , $w_0$ , $\alpha$ , and the number of patterns exceeding $\ell$ ). In contrast, vanilla attention would require communicating $\mathcal{O}(N d)$ keys and values — growing linearly with sequence length.

Why this constant communication cost is possible: the sparsification step compresses the keys and values before communication. Instead of sending all $N/P$ key vectors from each device (which would be $\mathcal{O}(N d)$ communication), each device sends only the strategically sampled subset defined by the dilation pattern. Because the number of sampled keys per pattern is constant (roughly $w_0$ ), the communication is constant. The queries are not communicated — they remain local — because the cross-attention is computed as local queries attending to global keys and values.

Step 4 — Cross-attention and output. Each device computes attention using its local queries $\tilde{Q}_1$ (or $\tilde{Q}_2$ ) against the global keys and values $\tilde{K}, \tilde{V}$ :

$\tilde{O}_1 = \text{softmax}(\tilde{Q}_1 \tilde{K}^T) \tilde{V}$

$\tilde{O}_2 = \text{softmax}(\tilde{Q}_2 \tilde{K}^T) \tilde{V}$

What this computes: device 1's queries can now attend to keys from both device 1 and device 2 (since $\tilde{K}$ contains both), and similarly for device 2. The attention outputs $\tilde{O}_1$ and $\tilde{O}_2$ are the computed updates for tokens on each device, incorporating information from the full sequence. No further communication is needed — the outputs are already on the correct devices.

Step 5 — Concatenation. The outputs are concatenated along the sequence dimension to form the full attention output:

$\tilde{O} = [\tilde{O}_1, \tilde{O}_2]$

Backward pass. The paper notes that the all-gather in the forward pass becomes a reduce-scatter in the backward pass — the gradients with respect to $\tilde{K}$ and $\tilde{V}$ need to be summed across devices and redistributed to their respective owners. This is a standard distributed communication primitive with the same complexity as all-gather.

Orthogonality to other parallelism strategies. The paper emphasizes that this sequence parallelism is "orthogonal to other parallelisms" — it can be combined with data parallelism (splitting the batch), model parallelism (splitting hidden dimensions), and pipeline parallelism (splitting layers). This is important for practical deployment: a large model with a long sequence might use model parallelism to handle the parameter size, pipeline parallelism to handle the depth, data parallelism to handle the batch, and LONGNET's sequence parallelism to handle the length — all simultaneously without conflicts.

Scaling to 1B tokens — empirical verification (Figure 5). The paper reports that starting from 8K tokens and scaling up to 1 billion tokens, dilated attention with FlashAttention shows "almost constant latency" — the runtime increases only marginally from ~1000 ms at 8K to ~2000 ms at 1B tokens, a factor of 2× for a 125,000× increase in sequence length. Vanilla attention with FlashAttention, in contrast, explodes from ~1000 ms at 8K to ~5000 ms at 128K (5× for only 16× length increase) and cannot scale beyond that due to memory constraints. Each model has up to 3 segment lengths: 2048, the number of tokens per device ( $\ell$ ), and the full sequence length $N$ . The batch size is reduced as sequence length increases to keep the total number of tokens per batch constant at 1 billion — so the runtime comparison is at constant total FLOPs, isolating the effect of sequence partitioning and communication.

Summary of Design Choices and Their Justifications

Dilation rather than striding or blocking: dilation exponentially expands the receptive field with depth, achieving $\mathcal{O}(\log N)$ path length, whereas fixed local windows achieve only $\mathcal{O}(N)$ path length. The exponential expansion is critical for reaching billion-token scales with realistic model depths (tens of layers).
Mixture of dilations rather than a single dilation rate: a single rate would force a tradeoff between local precision (small $r$ ) and global reach (large $r$ ). The mixture with geometric progression resolves this by having patterns at all scales operating in parallel, with dynamic weighting to emphasize the most relevant scale for each token.
Sparsifying queries (not just keys and values): this is what enables the FLOPs reduction by a full factor of $r^2$ (rather than just $r$ for key-only sparsification) and ensures the output size matches the sparsified input size, making the gather-scatter implementation clean.
Geometric sequences for $w$ and $r$ : this is the mathematical property that makes the series sum converge, yielding $\mathcal{O}(Nd)$ rather than $\mathcal{O}(Nd \log N)$ . An arithmetic progression would produce $\mathcal{O}(Nd \log N)$ complexity.
Dynamic softmax-denominator weights rather than learned weights: adaptivity to input-dependent pattern relevance, with zero additional parameters or computation (the softmax denominators are already computed).
Offset mechanism for multi-head attention: ensures full sequence coverage across heads without requiring more heads or more computation, by simply staggering which tokens each head sparsifies.
Distributed training via sequence parallelism with all-gather: exploits the constant-size sparsified representations to achieve constant communication cost, making the distributed algorithm scalable to arbitrary numbers of devices — a property that vanilla attention's quadratic structure fundamentally prevents.
Gather-scatter implementation compatible with FlashAttention: rather than requiring custom sparse attention kernels (which are notoriously difficult to optimize for GPU memory hierarchies), dilated attention is implemented as dense attention on gathered tensors, immediately benefiting from highly optimized dense attention kernels like FlashAttention. This is a pragmatic choice that prioritizes deployability over theoretical purity.

4. Key Insights and Innovations

Innovation 1: Dilated Attention Resolves the Feasibility-Expressivity Tension Through Exponential Receptive Field Expansion — Not Through Compromise

The dominant paradigm in efficient attention prior to LONGNET was compromise: every method traded some amount of model expressivity for computational feasibility. Sliding window approaches (Longformer, Beltagy et al., 2020) sacrificed global recall entirely — information from the first token could only reach the last token through a chain of $N/w$ local hops, each one a lossy compression through a fixed-size hidden state. Sparse attention methods (Sparse Transformer, Child et al., 2019; Big Bird, Zaheer et al., 2020) preserved some long-range connectivity but at superlinear cost ( $\mathcal{O}(N \sqrt{N})$ ), which still becomes intractable at extreme scales. Compression-based methods (Linformer, Wang et al., 2020; Perceiver, Jaegle et al., 2021) projected sequences into fixed-size bottlenecks, permanently discarding fine-grained token-level information. In every case, the designer had to choose which capability to sacrifice — local precision, global recall, or computational tractability — and live with the consequences.

Dilated attention's conceptual breakthrough is that it refuses this tradeoff entirely. Instead of asking "which connections should we drop?", it asks "how can we structure connections so that every token can reach every other token in $\mathcal{O}(\log N)$ hops while still doing only $\mathcal{O}(N)$ total computation?" The answer — exponential dilation with geometric progression of both segment sizes and dilation rates — is not a compromise but a structural property that emerges from a specific mathematical design: the attentive field doubles (or grows by factor $\alpha$ ) with each additional pattern in the mixture and with each additional layer of the Transformer. This means the receptive field grows exponentially with depth while the per-token computation remains constant.

To understand why this is a genuine conceptual shift rather than a clever engineering trick, consider the analogy the paper's name explicitly invokes: dilated convolutions in CNNs (Yu and Koltun, 2016). In the CNN context, dilated convolutions were revolutionary because they allowed a network with a fixed number of layers to have an exponentially larger receptive field — enabling dense prediction tasks like semantic segmentation that require seeing both fine detail and global context. But the CNN version of dilation only expanded the receptive field; it didn't address the fundamental quadratic scaling problem because CNNs already had linear complexity in sequence length. Transferring the dilation concept to attention is nontrivial because attention's quadratic complexity must be tackled simultaneously with receptive field expansion. The paper's key move is recognizing that the same geometric progression that gives exponential receptive field growth also makes the FLOPs series converge — the two properties are mathematical duals, not separate design goals.

This is what makes the $\mathcal{O}(\log N)$ token dependency result (Equation 20) the theoretical centerpiece of the paper rather than just a nice property. It proves that dilated attention achieves the information-theoretic lower bound for connectivity in a sparse graph with constant-degree nodes: you cannot connect $N$ nodes with paths shorter than $\mathcal{O}(\log N)$ unless each node has $\Omega(N)$ edges (which would be quadratic). Dilated attention achieves this lower bound while using only $\mathcal{O}(1)$ edges per node (amortized across patterns), making it information-theoretically optimal for the class of architectures with constant per-token computation. No prior efficient attention method had provided such a bound — sparse Transformers' $\mathcal{O}(N \sqrt{N})$ complexity meant their path length scaled as $\mathcal{O}(\sqrt{N})$ , which is far from optimal. The logarithmic dependency is not just asymptotically elegant; at $N = 10^9$ , it means information propagates end-to-end in roughly 19 layers, versus ~31,600 layers for a $\sqrt{N}$ -dependency method — a difference between feasible and impossible.

The practical manifestation of this insight appears in Figure 5's scaling behavior: dilated attention maintains nearly constant runtime from 8K to 1B tokens because the per-token computation truly is constant once the geometric series converges, while vanilla attention's quadratic growth makes it unusable beyond ~128K. But the deeper significance is that Figure 5 validates a theoretical prediction — it's not just "our method is fast," it's "our method's runtime follows the $\mathcal{O}(N)$ scaling law we derived, confirming that the geometric design achieves what the math promises."

This is a fundamental shift, not an incremental refinement. Prior work improved the constant factors in attention complexity (e.g., reducing memory via FlashAttention, Dao et al., 2022) or reduced the exponent (e.g., from $N^2$ to $N \sqrt{N}$ via sparsity) but never achieved genuine linear scaling with logarithmic connectivity. LONGNET's dilated attention is the first to simultaneously satisfy both the computational constraint ( $\mathcal{O}(Nd)$ ) and the connectivity constraint ( $\mathcal{O}(\log N)$ path length) with mathematical guarantees — and to demonstrate that satisfying both yields a model that matches or exceeds dense attention quality (Table 2) while scaling to lengths that were previously science fiction.

Innovation 2: The Mixture-of-Dilations Design Introduces Multi-Scale Attention as a First-Class Architectural Principle

Prior work on efficient attention treated the choice of attention sparsity pattern as a monolithic design decision: pick one pattern (local windows, or strided, or hashing-based, or low-rank) and apply it uniformly across all layers and all heads. If you chose local windows, every head in every layer used local windows. If you chose a combination (like Sparse Transformer's local + strided), the combination was fixed and applied uniformly. There was no notion that different scales of attention might be appropriate for different parts of the input, or that the model should dynamically route information through different scales depending on content.

LONGNET's mixture-of-dilations design (Equations 9–12) introduces multi-scale attention as a first-class architectural principle. Instead of one pattern, there are $k$ patterns operating in parallel at different spatial scales — from fine-grained local attention ( $w_0 = 2048, r = 1$ , every token attends to every other token within 2048-length windows) to coarse global attention ( $w_k \approx N$ , large $r$ , tokens are sparsely sampled across the entire sequence). Each scale captures dependencies at a characteristic distance, analogous to how the visual cortex processes information at multiple spatial frequencies simultaneously, or how wavelet transforms decompose signals into frequency bands.

The key conceptual advance is not the existence of multiple patterns — sparse Transformers already had local and strided patterns — but rather three design choices that elevate it from "having multiple patterns" to "multi-scale computation as an architectural primitive":

First, the geometric progression guarantees complete scale coverage. With $w$ and $r$ growing as $\{1, \alpha, \alpha^2, \dots\}$ , the scales are not arbitrary but form a complete exponential hierarchy. Every distance from 1 to $N$ is within a factor of $\alpha$ of being directly captured by some pattern. This ensures there are no "scale gaps" — no distance range where the model must rely entirely on multi-hop propagation rather than direct attention. A sparse Transformer with only local and strided patterns has a gap between the local block size (e.g., 2048) and the strided reach (e.g., 32768 via skipping) where intermediate-range dependencies are poorly modeled.

Second, the dynamic weighting via softmax denominators (Equation 10) introduces content-dependent scale selection. This is a subtle but important innovation. Rather than using learned fixed weights that would treat all tokens identically, the weight $\alpha_i \propto s_i$ (where $s_i$ is the softmax denominator for pattern $i$ ) means that for each token, the mixture automatically emphasizes whichever scale finds the strongest key-query matches. A token in the middle of a local syntactic pattern (e.g., completing a noun phrase) will likely have its local pattern ( $r=1$ ) dominate because the relevant keys are nearby and produce high attention scores. A token that needs to resolve a long-range coreference ("the company" referring back to "Microsoft" mentioned 10,000 tokens ago) will have its long-range pattern dominate because the relevant key is distant and only accessible through patterns with large $w$ and $r$ . This is a form of learned sparse routing that costs zero additional parameters — the softmax denominators are already computed as part of attention, so the dynamic weighting is essentially free.

Third, the paper's equivalence proof that the dynamic weighting equals computing one joint softmax over the union of all attended keys provides theoretical grounding for what would otherwise be an arbitrary heuristic. The statement "our method to mix dilated attentions is equivalent to gather keys in different parts and calculate softmax together" (Section 2.2) shows that the mixture is not an approximation or an ensemble — it is mathematically equivalent to standard attention where the set of attended keys is the union of all patterns' keys, with the softmax computed jointly. This means that from the perspective of any query token, the dilated attention mixture is computing exactly the same operation as dense attention, just with a strategically subsampled key set that preserves local density and global coverage. This is a stronger guarantee than most sparse attention methods can provide — they typically approximate or truncate the attention distribution, introducing bias. LONGNET's mixture is unbiased in the sense that it computes exact attention over a carefully chosen subset of keys, with the subset designed to minimize the information loss from not attending to the dropped keys.

This framing — multi-scale computation with content-dependent routing and exact-attention guarantees — represents a fundamental advance in how we think about attention sparsity. It moves the field from "sparsity as a necessary evil to reduce computation" to "sparsity as a mechanism for multi-scale representation learning." The parallel to multi-scale processing in CNNs (feature pyramids, U-Nets) and in signal processing (wavelets, filter banks) suggests that this architectural principle may generalize beyond Transformers to other sequence models — and the paper's brief mention of future applications to multimodal modeling, BEiT pretraining, and genomic data (Section 5) hints at this broader ambition.

The empirical validation that this matters — LONGNET outperforming sparse Transformers at matched FLOPs in Table 2, and LONGNET achieving lower perplexity than dense Transformers at equal compute in Figure 6 — confirms that the multi-scale design isn't just theoretically elegant but yields genuine quality improvements. The fact that LONGNET surpasses dense Transformers on some metrics is particularly revealing: it suggests that forcing the model to process information at multiple scales simultaneously acts as a beneficial architectural inductive bias, not just a compute-saving approximation. Dense attention provides every connection equally, which may actually be harmful — it allows the model to overfit to spurious local correlations without being forced to learn the long-range dependencies that generalize better (the paper's argument about "short dependency has more spurious correlations" from Section 1). Multi-scale attention, by making long-range dependencies explicit and structurally accessible, may guide the model toward more robust features.

Innovation 3: Distributed Sequence Parallelism with Constant Communication Cost — Proving That Architecture Design and Systems Design Are the Same Problem

A recurring pattern in deep learning systems research is the separation between model architecture (what operations to compute) and distributed systems (how to parallelize those operations across hardware). Researchers typically design architectures assuming infinite single-device capacity, then systems engineers retrofit parallelism strategies (data parallelism, model parallelism, pipeline parallelism) as afterthoughts. This separation has been workable because most architectures have natural parallelism dimensions: the batch dimension for data parallelism, the hidden dimension for model parallelism, the layer dimension for pipeline parallelism.

Sequence length, however, has been the impossible parallelism dimension. For vanilla attention, parallelizing across the sequence dimension is fundamentally expensive because the attention matrix $QK^T$ requires all-to-all communication: every device holding a chunk of the sequence needs access to every other device's keys and values. The communication cost scales as $\mathcal{O}(P N d)$ for $P$ devices — linear in both sequence length and device count, making it a non-starter for long sequences on large clusters. This is why prior sequence parallelism approaches (e.g., Megatron-LM's tensor parallelism, Shoeybi et al., 2019; sequence parallelism, Li et al., 2021) have been limited to moderate sequence lengths where the quadratic attention cost hurts more than the communication cost.

LONGNET's distributed training algorithm (Section 3, Equations 21–25) represents a conceptual fusion of architecture design and systems design. The insight is that the same mathematical property that yields linear computation complexity — the sparsified key-value representations having size $\frac{w_i}{r_i} \times d$ , independent of $N$ — also yields constant communication cost. Because each dilation pattern only needs to communicate its sparsified keys and values (a constant-sized tensor), the all-gather operation that collects global keys and values transmits a fixed amount of data regardless of total sequence length. The communication cost is dominated by the sum $\sum w_i / r_i$ , which converges geometrically, exactly like the FLOPs sum.

This is not an optimization that could be retrofitted to any architecture — it is a structural property of dilated attention specifically. Sparse attention patterns that achieve $\mathcal{O}(N \sqrt{N})$ complexity would require communicating $\mathcal{O}(\sqrt{N})$ keys per device, which still grows with sequence length (albeit sublinearly) and would become the bottleneck at extreme scales. Sliding window attention could be parallelized trivially (no communication needed for cross-device attention since each device's windows are local) but suffers the expressivity limitations discussed in Innovation 1. The fact that LONGNET's architecture simultaneously solves the computation, memory, and communication bottlenecks — and that the solution to all three emerges from the same geometric series — is evidence of a deep connection between the architecture's mathematical structure and its systems properties.

The significance of this goes beyond the specific distributed algorithm presented. It suggests a new design principle: architectures should be evaluated not just on their single-device FLOPs but on their distributed communication complexity. An architecture that is $\mathcal{O}(N d)$ in computation but $\mathcal{O}(N d)$ in communication does not actually scale to large clusters because communication bandwidth, not FLOPs, becomes the bottleneck. LONGNET's $\mathcal{O}(1)$ communication complexity (per device, in the sequence dimension) means it can scale to arbitrary numbers of devices without hitting a communication wall — a property that should be a design target for future long-sequence architectures.

The practical validation in Figure 5 — nearly flat runtime scaling from 8K to 1B tokens on multi-GPU systems — is more significant than a typical speed benchmark because it demonstrates that the constant-communication property holds in real systems, not just in asymptotic analysis. The fact that each model of different sequence lengths used "up to 3 segment lengths" (2048, per-device length $\ell$ , and full sequence length $N$ ) shows that the implementation dynamically switches between local computation (no communication, for $w_i \leq \ell$ ) and cross-device computation (all-gather, for $w_i > \ell$ ) — seamlessly handling the transition from single-device to multi-device regimes without any architectural changes or hyperparameter tuning.

This is a fundamental advance in the theory of distributed sequence processing. It establishes that the problem of scaling sequence length is not just a matter of reducing FLOPs (which many prior methods addressed) but of designing connectivity patterns that are inherently partitionable across devices with bounded communication. The paper does not belabor this point theoretically, but the implication is clear: the search for efficient long-sequence architectures should prioritize constant-communication designs, and dilated attention provides a template for how such designs can be constructed from geometric sparsification patterns.

Innovation 4: Empirical Validation That Sparse Attention Can Match and Exceed Dense Attention Quality — Overturning the Implicit Assumption That Sparsity Necessarily Degrades Performance

A persistent, often unstated assumption in the efficient Transformers literature is that sparse attention is a performance-cost tradeoff: you accept some degradation in model quality (higher perplexity, lower accuracy) in exchange for the ability to process longer sequences or reduce computation. This assumption is so deeply embedded that papers on efficient attention routinely report metrics showing their method "approaches" or "is competitive with" dense attention, implying that matching dense attention is the ceiling. Even the original Sparse Transformer paper (Child et al., 2019), which demonstrated that sparse patterns could achieve impressive results on density estimation and raw audio generation, showed performance comparable to dense Transformers at similar parameter counts — not better.

LONGNET's experimental results (Table 2, Figure 6, Figure 7) challenge this assumption directly. Across multiple sequence lengths (2K, 8K, 16K, 32K) and model sizes (up to 2.7B parameters), LONGNET does not merely match the baselines — it consistently outperforms both dense Transformers and sparse Transformers at matched computational cost. In Table 2, LONGNET achieves lower perplexity than Sparse Transformer with the same FLOPs at every tested sequence length (e.g., 3.01 vs. 3.64 at 32K). In Figure 6, LONGNET achieves lower test perplexity than dense Transformers at equal or lower total FLOPs — the LONGNET curve lies below the Transformer curve across the entire compute range. This is not "sparse attention that doesn't hurt too much" — it's "sparse attention that is genuinely better."

The significance of this finding extends beyond the specific numbers. If sparse attention can outperform dense attention at equal compute, it means that the inductive biases of the sparsity pattern are actually beneficial for learning, not just computationally convenient. The paper's motivation section hints at why: "short dependency has more spurious correlations, which is harmful to generalization." Dense attention allows a model to learn brittle local patterns — for instance, predicting the next token based on the previous sentence's phrasing rather than understanding the broader argument structure. By forcing the model to integrate information across multiple spatial scales simultaneously, dilated attention may act as a regularizer that discourages overfitting to local shortcuts and encourages learning more robust, long-range features.

This interpretation is supported by the scaling behavior in Figure 6: the gap between LONGNET and dense Transformers appears to widen as the training sequence length increases (from 2K to 32K). At longer training lengths, the benefit of multi-scale attention over dense attention grows, consistent with the hypothesis that dense attention increasingly suffers from spurious local correlations as the context available for overfitting expands. LONGNET's multi-scale structure — where each token is forced to route information through patterns at different spatial scales — may prevent the model from collapsing into a purely local prediction strategy.

The scaling law results in Figure 7(a) add further weight: LONGNET follows the same power-law relationship between compute and loss that has been observed for dense Transformers (Kaplan et al., 2020), indicating that the sparse attention pattern does not introduce any fundamental learning dynamics issues. The 2.7B parameter model trained on 300B tokens continues to improve along the predicted power-law trajectory, with no evidence of saturation or degradation relative to what would be expected from a dense model of the same size. This is a crucial validation because some prior efficient attention methods (particularly kernel-based approximations and low-rank methods) have shown degraded scaling behavior at larger model sizes, where the approximation errors compound across layers.

This is a fundamental empirical finding rather than an incremental improvement. It reframes the efficient attention problem from "how much quality must we sacrifice to reduce computation?" to "what attention connectivity patterns are better for learning, independent of computational considerations?" This reframing opens up a new research direction: designing attention sparsity patterns as architectural inductive biases rather than as compute-saving approximations. The dilated attention pattern — with its multi-scale structure, logarithmic connectivity, and dynamic scale selection — may represent a particularly good inductive bias for sequence modeling, and the paper's results suggest that exploring this design space is likely to yield further improvements beyond what dense attention can achieve.

5. Experimental Analysis

Evaluation Methodology

Dataset. Language modeling experiments use The Stack dataset (Kocetkov et al., 2022), a collection of permissively licensed source code in over 300 programming languages, tokenized with the tiktoken tokenizer using cl100k_base encoding. The training and test splits follow the standard partitioning from the dataset release. The paper also performs long-context prompting experiments using a held-out test set where prefixes of varying lengths (2K to 32K tokens) are prepended to fixed suffixes.
Base model(s). The backbone architecture is MAGNETO (Wang et al., 2022) with XPOS relative position encoding (Sun et al., 2022), a foundation Transformer variant designed for stability and length extrapolation. The base configuration uses 12 decoder layers, hidden dimension 768, 12 attention heads, and FFN size 3072 (approximately 125M parameters). For scaling experiments, models range from 125M to 2.7B parameters, with the 2.7B model using 32 layers, hidden dimension 2560, and 32 heads. The MAGNETO backbone is chosen because its architecture (pre-normalization, relative position encoding rather than absolute) is representative of modern Transformer designs and provides a strong baseline for evaluating the attention mechanism in isolation — only the attention layers differ between LONGNET and baseline configurations.
Metrics. The primary metric is test perplexity (PPL) on held-out data from The Stack. Perplexity is the exponentiated average negative log-likelihood per token: lower is better, with each unit reduction representing a meaningful improvement in predictive quality. For the scaling curves, test loss (cross-entropy) is also reported directly. For the long-context prompting experiments, perplexity is measured specifically on the suffix tokens (not the prompt), isolating the effect of additional context on prediction quality for the remaining text.
Baselines. Three baselines are compared: (1) Vanilla Transformer (Vaswani et al., 2017) with standard dense self-attention — all queries attend to all keys, giving quadratic complexity. Due to computational constraints, it is only scaled to 32K sequence length. (2) Sparse Transformer (Child et al., 2019) with the fixed local + strided attention pattern. The block size is set to 2048, and the sparse ratios are adjusted to match the computation FLOPs of LONGNET so comparisons are fair — this means the Sparse Transformer uses an attention pattern that is sparser (fewer attended keys per query) to equalize the per-token computational budget. Multiple heads attend to distinct subblocks as in the original Sparse Transformer design. (3) Block-wise causal attention (BCA) (Sun et al., 2022) is used during inference for extrapolation beyond the maximum training length for all models. All attention implementations, including custom kernels for sparse and dilated attention, are built on FlashAttention (Dao et al., 2022) for memory efficiency and speed.
Generation budget / compute accounting. The paper establishes fair comparison through careful FLOPs matching. For Table 2, each model configuration operates at a constant number of tokens per batch (0.5M tokens in the base experiments, 4M for the 2.7B model) across all sequence lengths — as the sequence length increases, the batch size decreases proportionally, so the total computation per training step remains comparable. The Sparse Transformer's attention sparsity is explicitly adjusted to match LONGNET's FLOPs, ensuring that both models do approximately the same amount of computation per token. For the scaling curves in Figure 6, compute is measured as total FLOPs of matrix multiplication during training, and the comparison sweeps across different training sequence lengths while holding other hyperparameters constant. In the scaling law experiments (Figure 7a), the amount of compute is estimated similarly, tracking total matrix multiplication FLOPs across models of different sizes trained on different numbers of tokens (from ~40B tokens for smaller models to 300B tokens for the 2.7B model).
Cross-validation / statistical protocol. The paper does not describe explicit cross-validation or multiple random seeds for the main results. The test set used for evaluation is held-out data from The Stack, evaluated once after training completion. Hyperparameters are listed in Appendix Tables 3 and 4. For the runtime measurements in Figure 5, results are averaged over 10 different forward propagation runs to account for system-level variance. The paper does not report confidence intervals, standard deviations, or statistical significance tests for perplexity differences between models.

Main Quantitative Results

Language Modeling with Increasing Sequence Length (Table 2, Figure 6)

Headline result: LONGNET consistently achieves lower perplexity than both dense Transformers and sparse Transformers at matched sequence lengths and matched computational cost, with the advantage persisting or growing as sequence length increases from 2K to 32K.

Table 2 reports perplexity for LONGNET, Transformer, and Sparse Transformer across three evaluation sequence lengths (2K, 8K, 32K) for models trained at four different maximum sequence lengths (2K, 8K, 16K, 32K). The results demonstrate several key findings simultaneously:

Increasing training sequence length improves language model quality regardless of architecture. For all models, perplexity at a given evaluation length generally decreases as the training sequence length increases. For example, LONGNET evaluated at 32K drops from 11.29 PPL when trained at 2K to 3.36 PPL when trained at 8K, to 3.31 PPL at 16K, to 3.01 PPL at 32K. This validates the core motivation: longer context during training yields better language models.
Extrapolation beyond training length fails when the gap is large. When evaluated at 32K, models trained at 2K (using block-wise causal attention for extrapolation) show dramatically higher perplexity: Transformer at 11.29 PPL, Sparse Transformer at 8.79 PPL, LONGNET at 11.29 PPL. The fact that all three show similar degradation at extreme extrapolation distances (~16x beyond training length) suggests that block-wise causal attention, while state-of-the-art for length extrapolation, cannot bridge an order-of-magnitude gap — the model must be trained at the target length to perform well. This justifies the need for architectures that can actually train at extreme lengths rather than relying on inference-time extrapolation.
LONGNET outperforms Sparse Transformer at matched FLOPs across all configurations. At 2K training length (evaluation at 2K): LONGNET achieves 4.23 PPL vs. Sparse Transformer's 4.39 and Transformer's 4.24 — essentially tied with dense attention but better than sparse. At 8K training length (evaluation at 8K): LONGNET 3.24 vs. Sparse Transformer 3.35 vs. Transformer 5.07. The dense Transformer's performance degrades notably here — likely because at 8K sequence length with batch size reduced to maintain constant tokens, the dense attention's quadratic cost forces other compromises. At 32K training length (evaluation at 32K): LONGNET 3.01 vs. Sparse Transformer 3.64 — a substantial 0.63 PPL gap (approximately 17% relative improvement). The dense Transformer could not be evaluated at the 16K and 32K training lengths due to computational constraints (noted by the missing entries in the table).
LONGNET's advantage over Sparse Transformer grows with training sequence length. The gap between LONGNET and Sparse Transformer at 2K evaluation is small (4.23 vs. 4.39, a gap of 0.16). At 8K evaluation, the gap grows to 0.11 (3.24 vs. 3.35). At 32K evaluation, the gap is 0.63 (3.01 vs. 3.64). This progressive widening suggests that dilated attention's multi-scale design provides increasing benefits as the context length grows — the Sparse Transformer's fixed two-pattern sparsification (local + strided) may miss intermediate-range dependencies that LONGNET's geometric progression of dilation rates captures.

Figure 6 — Test perplexity vs. FLOPs curves. The figure plots test perplexity against total FLOPs for LONGNET and dense Transformers at different training sequence lengths. The key observations:

LONGNET achieves lower perplexity than dense Transformers at equal or lower compute across the entire FLOPs range tested. The LONGNET curve sits below the Transformer curve — often by a substantial margin — meaning that for any given compute budget, LONGNET is the more efficient choice.
LONGNET trained at 32K reaches lower perplexity than dense Transformers trained at any length (2K, 8K, 16K), despite using significantly fewer FLOPs. At approximately 6 × 10^16 FLOPs, LONGNET-32K achieves test PPL around 3.0–3.5, while dense Transformer-16K at the same FLOPs achieves PPL around 4.5–5.0.
The scaling behavior of LONGNET with respect to training sequence length is more efficient: the 32K LONGNET curve reaches lower perplexity with less compute than the 16K LONGNET curve, which in turn is better than the 8K curve. This demonstrates that the longer context translates to better predictive performance per unit of computation.

The paper interprets this as evidence that "longer training input [is better] over extrapolation" — models trained with longer sequences learn better representations than models that see the same number of tokens but in shorter chunks. LONGNET makes this feasible because its linear complexity allows scaling training length without proportional increases in per-token computation, unlike dense Transformers.

Scaling Model Size (Figure 7a)

Headline result: LONGNET follows the same power-law scaling relationship between compute and loss that has been established for dense Transformers (Kaplan et al., 2020), demonstrating that sparse attention does not disrupt the fundamental scaling properties of the Transformer architecture.

Figure 7(a) plots test loss against total training FLOPs for LONGNET models ranging from 125M to 2.7B parameters. The 2.7B model was trained on 300B tokens, while smaller models (125M, 350M, 760M) were trained on approximately 40B tokens each. The resulting curve shows a smooth power-law decrease in loss as compute increases, with no visible discontinuities or saturation effects as model size scales up. The paper states: "It proves that LONGNET can still follow the power law. This implies that the dense Transformer is not a prerequisite for scaling the language models."

This is a critical empirical finding because some prior efficient attention methods have shown degraded scaling behavior. Kernel-based approximations (Performer, Linear Transformer) and low-rank methods (Linformer) introduce biases in the attention computation that can compound across layers and training steps, causing the model's effective capacity to saturate earlier than a dense Transformer of the same parameter count. The fact that LONGNET's scaling curve remains smooth up to 2.7B parameters and 300B training tokens (a non-trivial scale, though smaller than the largest dense LMs) suggests that dilated attention's exact-attention-over-subsampled-keys design does not introduce systematic bias that limits model quality.

The implication is significant for practitioners: one can replace dense attention with dilated attention and follow the same model scaling recipes (same hyperparameters, same training procedures) without needing to recalibrate scaling law coefficients or worry about hitting a quality ceiling at larger scales. The "scalability and the efficiency are both obtained by LONGNET," as the paper phrases it — no tradeoff required.

Long Context Prompting (Figure 7b)

Headline result: LONGNET's test loss on fixed suffix tokens decreases monotonically as the context window (prefix length) grows from 1K to 32K, demonstrating that the model can effectively leverage additional context to improve predictions.

Figure 7(b) measures perplexity on a fixed set of suffix tokens while varying the length of the preceding context from 1K to 32K tokens. The test loss drops from approximately 2.2 at 1K context to approximately 1.6 at 32K context — a substantial improvement of roughly 0.6 in cross-entropy loss, or roughly a 27% relative reduction.

This result is important because it distinguishes LONGNET from sliding-window approaches, which cannot effectively use context beyond their window size (the model simply doesn't have access to tokens more than $w$ positions away). It also distinguishes LONGNET from compression-based approaches, which might retain some information from distant context but in a lossy compressed form that degrades for early tokens. LONGNET's logarithmic token dependency means that information from the very beginning of a 32K sequence is accessible through at most a handful of attention hops (with $w_0 = 2048$ and $\alpha = 2$ , roughly $\log_2(32768/2048) = 4$ hops), and the model demonstrably uses this access to improve predictions.

The monotonic improvement also suggests that the model is not saturating its ability to use context at 32K — the curve is still declining at the right edge of Figure 7(b). This implies that further increasing the context length would yield additional prediction improvements, consistent with the paper's broader claim that arbitrary-length context scaling is valuable. However, this is an extrapolation beyond the tested range and cannot be confirmed from the reported data.

Distributed Training Scalability (Figure 5)

Headline result: Dilated attention with FlashAttention maintains nearly constant runtime (~1000–2000 ms per forward pass) as sequence length scales from 8K to 1 billion tokens, while vanilla attention's runtime explodes from ~1000 ms at 8K to ~5000 ms at 128K and cannot scale further.

Figure 5 reports average forward propagation runtime for 10 runs across sequence lengths spanning five orders of magnitude. The key data points:

At 8K: both dilated attention and vanilla attention run at approximately 1000 ms.
At 16K: dilated attention remains at ~1000 ms; vanilla attention increases to ~1200 ms.
At 32K: dilated attention ~1000 ms; vanilla attention ~1300 ms.
At 64K: dilated attention ~1000 ms; vanilla attention ~2000 ms.
At 128K: dilated attention ~1000 ms; vanilla attention ~5000 ms.
Beyond 128K: vanilla attention cannot run due to memory constraints. Dilated attention continues to scale: ~1500 ms at 512K, ~2000 ms at 2M, and essentially flat from 8M through 1B tokens at approximately 2000 ms.

The experimental details are crucial: to keep the total number of tokens per batch constant at 1 billion, the batch size is reduced as sequence length increases. Each model uses up to 3 segment lengths (2048, the number of tokens per device, and the full sequence length). The runtime measurements include the distributed communication costs (all-gather operations for cross-device attention) and the FlashAttention kernel execution. The near-constant latency validates the theoretical analysis: the geometric series bounding per-token FLOPs (Equation 18) and the constant communication cost (Section 3.1) translate to real hardware performance.

The practical significance: a model processing 1 billion tokens runs only about 2× slower than the same model processing 8K tokens, despite a 125,000× increase in sequence length. This is possible because the per-token computation is genuinely constant — the model processes each token with the same number of FLOPs regardless of total sequence length. The small increase from ~1000 ms to ~2000 ms likely stems from the transition from purely local computation (all segment lengths fit on a single device at 8K) to cross-device communication (larger segment lengths require all-gather operations) and the fixed overhead of the additional dilation patterns needed for longer sequences.

Ablation Studies and Robustness Checks

Sparse ratio fairness (Table 2): The Sparse Transformer's attention sparsity is explicitly adjusted to match LONGNET's computation FLOPs. Without this adjustment, a direct comparison would be confounded — if Sparse Transformer did more FLOPs per token (by attending to more keys), it might achieve better perplexity purely through additional computation rather than through a better attention pattern. The fact that LONGNET outperforms sparse Transformers at matched FLOPs isolates the architectural advantage of dilated attention over fixed local + strided patterns. The paper does not report what Sparse Transformer perplexity would be with its default (non-FLOPs-matched) sparsity configuration.

Sequence length generalization via block-wise causal attention (Table 2, across rows): All models use block-wise causal attention (Sun et al., 2022) during inference to handle sequences longer than their training length. The results show that this extrapolation method works reasonably for moderate extensions (e.g., models trained at 8K evaluated at 32K show manageable perplexity degradation), but fails completely for large extensions (models trained at 2K evaluated at 32K show 2–3× higher perplexity than models trained at 32K). This validates that BCA is not a substitute for training at the target length, and supports the paper's central claim that architectures enabling long-sequence training are necessary.

Dynamic vs. learned fixed weights for mixture combination (Section 2.2, stated without table): The paper asserts that "dynamic weights calculated by the denominator of the attention softmax are better than learnable fixed weights" for the mixture-of-dilations combination (Equation 10). However, no experimental comparison or quantitative results are provided to support this claim — no table or figure contrasts dynamic weighting against learned scalar weights or uniform averaging. This is a notable omission, as the dynamic weighting mechanism is a key architectural claim (Innovation 2 in Section 4 of this analysis), and empirical validation would strengthen it considerably.

Geometric progression of $w$ and $r$ (Equations 11–12, implemented in experiments): The paper prescribes geometric sequences for segment sizes and dilation rates and derives the linear complexity guarantee from this choice. The experiments in Table 2 use $w = \{2048, 4096, 8192, 16384, 32768\}$ (exact geometric progression with ratio 2) and $r = \{1, 2, 4, 6, 12\}$ (approximately geometric, with deviations at $r=6$ instead of $8$ and $r=12$ instead of $16$ ). No ablation compares geometric vs. arithmetic progression, nor explains why the exact values $r=\{1,2,4,6,12\}$ were chosen over the cleaner $\{1,2,4,8,16\}$ . This is a minor transparency issue — the theoretical analysis depends on exact geometric sequences, but the implementation uses a slight variant without justification.

Number of dilation patterns (implicit in configuration): The experiments use $k=5$ dilation patterns. No ablation studies varying $k$ (e.g., using 3 patterns or 7 patterns) are reported. The theoretical analysis suggests that $k = \lceil \log_\alpha (N / w_0) \rceil$ patterns are needed to span the sequence length, and that additional patterns add minimal computation (due to the converging geometric series), but it would be informative to see whether fewer patterns (reducing parallel computation) or more patterns (finer scale coverage) affect perplexity.

Position encoding interaction: The models use XPOS relative position encoding (Sun et al., 2022) rather than absolute position encodings. The paper does not ablate this choice — for instance, comparing dilated attention with absolute position encodings, RoPE, or ALiBi — so it is unclear whether the performance of dilated attention depends on the specific position encoding scheme or generalizes across encoding methods. This matters because the sparsification pattern interacts with how the model represents distance: a relative encoding like XPOS may be more compatible with the multi-scale structure (since it naturally handles varying distances) than an absolute encoding (which might struggle with the irregular sparsification pattern).

Comparison with other linear-complexity methods (not performed): The paper compares against dense Transformers and Sparse Transformers but does not benchmark against other linear-complexity attention methods such as Performer (Choromanski et al., 2021), Linear Transformer (Katharopoulos et al., 2020), Linformer (Wang et al., 2020), or state space models (S4, H3, Hyena). The motivation section critiques these methods for expressivity limitations, but no head-to-head experimental comparison is provided. This is a significant omission because the paper's core claim is that LONGNET resolves the feasibility-expressivity tension that prior linear-complexity methods failed on — without empirical evidence that LONGNET actually outperforms those methods on short-sequence quality (where they allegedly struggle), the claim rests on the theoretical argument and the favorable comparison to dense Transformers, but not on direct competitive benchmarking.

Extrapolation to 1B tokens (Figure 5 only): The paper demonstrates runtime scaling to 1B tokens in Figure 5, but does not report language modeling perplexity or any quality metric at this scale. The largest training experiments use 32K sequence length (Table 2), and the largest inference experiments use 32K context (Figure 7b). The claim of "scaling Transformers to 1,000,000,000 tokens" is therefore a systems claim (the distributed algorithm can process sequences of that length) but not a quality claim (the model achieves good perplexity at that length). The paper acknowledges this implicitly by focusing the experimental section on moderate lengths where quality can be measured, but the gap between the 32K training experiments and the 1B systems demonstration is roughly four orders of magnitude, and it is unknown whether LONGNET's perplexity would continue to improve with context length at those scales or whether new issues (e.g., optimization difficulties, gradient propagation problems, or PRM over-optimization analogous to what Section 4 of this analysis discussed for other architectures) would emerge.

FLOPs accounting for distributed communication (not included): The FLOPs comparisons in Table 2 and Figure 6 count only the matrix multiplication operations in the attention computation. The communication cost of the all-gather operations in the distributed setting is not included in the FLOPs accounting, which is standard practice (communication bandwidth is a separate resource from compute throughput) but means that the true end-to-end comparison on multi-GPU systems may differ from the single-device FLOPs comparison. Given that the paper emphasizes distributed training as a key contribution (Section 3, Figure 5), this is worth noting.

Critical Assessment

Does LONGNET Scale Sequence Length to 1 Billion Tokens? — Supported as a Systems Claim, Unsupported as a Quality Claim

The paper's title promises "Scaling Transformers to 1,000,000,000 Tokens." Figure 5 convincingly demonstrates that the dilated attention mechanism, combined with the distributed training algorithm, can process sequences of this length with nearly constant runtime. This is a genuine systems achievement — the architecture, implemented with FlashAttention kernels and distributed across multiple GPUs, handles forward propagation through attention layers on 1B-token inputs without running out of memory or experiencing catastrophic slowdown.

However, this is a runtime and memory scalability claim, not an empirical modeling quality claim. The actual language modeling experiments (Table 2, Figure 6, Figure 7) train and evaluate at sequence lengths up to 32K — a factor of roughly 30,000× shorter than 1B. The reader is asked to extrapolate from demonstrated quality improvements at 2K–32K (where LONGNET outperforms baselines) to the expectation that quality would continue to improve at 1B, but no evidence supports this extrapolation. The paper's own finding that block-wise causal attention fails for large extrapolation gaps (Table 2: models trained at 2K fail badly at 32K) ironically underscores why such extrapolation is dangerous.

Several mechanisms could cause LONGNET's quality to plateau or degrade at extreme lengths even if runtime scaling remains linear: (1) optimization difficulties from propagating gradients across millions of tokens and ~20 attention hops (the number of hops grows only logarithmically, but 20 hops through softmax attention may still attenuate gradient signals); (2) the capacity of the model to actually use information from 1B tokens ago, which depends on the hidden dimension and number of layers, not just the attention connectivity pattern; (3) training data constraints — The Stack dataset may not contain meaningful dependencies at 1B-token distances (most code files are far shorter), making the architectural capability irrelevant for this training distribution; (4) the revision-model-style correct-to-incorrect reversion problem (analogous to what was discussed in Section 4 of the prior analysis for other architectures) where attention to very distant tokens introduces noise rather than useful signal.

The paper would be strengthened by at least one quality measurement at a substantially longer scale — for instance, training at 128K or 512K and showing that perplexity continues to improve, or fine-tuning a pre-trained LONGNET on a synthetic long-range dependency task where the correct answer requires attending to information 100K+ tokens back. Without such evidence, the "1 billion tokens" claim should be understood as applying to computational feasibility, not modeling capability.

Does LONGNET Match or Exceed Dense Transformer Quality? — Supported at Tested Scales

Table 2 and Figure 6 provide clear evidence that LONGNET achieves lower perplexity than dense Transformers at matched FLOPs for sequence lengths up to 32K. The results are consistent across multiple training lengths (2K, 8K, 16K, 32K) and evaluation lengths (2K, 8K, 32K), with LONGNET either matching or outperforming dense Transformers in every configuration where both could be run.

The evidence is strongest at 8K and 32K, where the dense Transformer baselines exist and the gaps are substantial (e.g., LONGNET 3.24 vs. Transformer 5.07 at 8K evaluation, Table 2). At the shortest length (2K), LONGNET essentially ties dense Transformers (4.23 vs. 4.24 PPL), suggesting that dilated attention does not impose a quality penalty even when sequence length is small and the full dense attention matrix is computationally feasible.

However, the comparison at longer lengths is incomplete because dense Transformers cannot be trained at 16K or 32K in this configuration. The paper reports dense Transformer perplexity at these evaluation lengths only for models trained at shorter lengths (using block-wise causal attention for extrapolation), which is not a fair comparison to LONGNET trained at the target length. The missing entries in Table 2 for dense Transformers at 16K and 32K training lengths mean we cannot directly observe whether dense attention would outperform dilated attention if computational constraints were removed — the observed LONGNET superiority at 32K evaluation (3.01 PPL) is relative to a dense Transformer trained only at 8K (5.07 PPL extrapolated to 32K), not to a dense Transformer trained at 32K. The paper is transparent about this constraint but the comparison is nonetheless confounded.

Does LONGNET Scale According to Power Laws? — Supported at Tested Scales, Missing Critical Comparison

Figure 7(a) demonstrates that LONGNET models from 125M to 2.7B parameters follow a power-law scaling curve consistent with established findings for dense Transformers. This is a meaningful result that addresses a genuine concern: do linear-complexity attention approximations degrade model scaling behavior? The smooth curve up to 2.7B parameters and 300B training tokens suggests they do not, at least at these scales.

However, the paper does not plot a dense Transformer scaling curve on the same axes for direct comparison. Without the dense Transformer curve, we cannot assess whether LONGNET's power-law exponent (the slope of the log-log relationship) differs from dense Transformers, or whether LONGNET achieves the same loss for the same total FLOPs at equivalent model sizes. The paper says the curve "follows a similar law to the vanilla Transformers" but provides no quantitative comparison — the reader must take on faith that the exponent and constant factor are comparable. A direct overlay of LONGNET and dense Transformer scaling curves (or at minimum a statement of the fitted power-law coefficients for both) would make this claim testable rather than qualitative.

Additionally, 2.7B parameters is relatively small by contemporary LLM standards (2023–2024). Whether the power-law relationship holds through the 10B, 100B, and 1T parameter scales — where optimization dynamics, attention head specialization, and multi-scale interaction effects might change — is an open question that the current experiments cannot address.

Does LONGNET Outperform Sparse Transformers? — Supported but Scoped Narrowly

Table 2 provides a clean comparison: LONGNET vs. Sparse Transformer at matched FLOPs, with the same backbone architecture (MAGNETO), same training data, and same hyperparameters — differing only in the attention pattern. LONGNET consistently achieves lower perplexity, with a growing advantage as sequence length increases. This is the most internally valid comparison in the paper because it isolates the attention mechanism with all other variables controlled.

The limitation is that only one sparse attention pattern is tested: the fixed local + strided pattern from Child et al. (2019). The broader class of sparse attention methods — Big Bird's random + local + global pattern, Longformer's dilated sliding window, Reformer's LSH-based pattern, or learned sparsity like CoLT5 — are not compared. The claim that dilated attention outperforms "sparse attention" writ large would require testing against multiple sparse patterns; the current experiments only support the narrower claim of outperforming one specific fixed pattern at matched FLOPs.

Does LONGNET Leverage Long Context to Improve Predictions? — Supported at 32K

Figure 7(b) shows that LONGNET's test loss on fixed suffix tokens decreases monotonically as the context window increases from 1K to 32K. This is a clean demonstration that the model can effectively use additional context — unlike sliding-window approaches that would show no improvement beyond their window size. The monotonic improvement without saturation at 32K is encouraging.

However, this result is on the test set of The Stack (code data), where dependencies are often local (within a function or file) and the benefit of very long context may be limited. The paper does not evaluate on a benchmark specifically designed to test long-range dependency utilization, such as the Long Range Arena (Tay et al., 2021), which would provide a more controlled assessment of whether LONGNET actually captures dependencies at the maximum distances its architecture theoretically supports (e.g., 32K-token dependencies). The code domain's natural locality means that even a model with poor long-range capabilities might show improving perplexity with longer context simply because more local context is available — the suffix tokens might benefit from seeing additional nearby code, not from truly long-range dependencies.

What Experiments Would Strengthen the Paper?

Several experiments that are notably absent would significantly increase confidence in the claims:

Quality measurement at lengths beyond 32K. Training at 128K or 256K and reporting perplexity would bridge the enormous gap between the 32K training experiments and the 1B runtime demonstration. Even if full training at 1B is impractical, fine-tuning a pre-trained model with 128K context and measuring on a long-range dependency task would provide evidence that quality scales with context length.
Head-to-head comparison with other linear-complexity methods. Benchmarking against Performer, Linear Transformer, or state space models (S4/H3) on the same dataset and model size would test the paper's claim that prior linear-complexity methods sacrifice short-sequence expressivity while LONGNET does not. If LONGNET matched or beat these methods on short sequences while also scaling to extreme lengths, that would strongly support the core contribution.
Synthetic long-range dependency task. A controlled experiment where the correct answer depends on information at a known, variable distance (e.g., "retrieve the token at position N-K and output it") would directly measure the effective context utilization as a function of distance, revealing whether the logarithmic connectivity actually translates to usable information propagation at billions of tokens.
Ablation of mixture components. Testing with 1, 2, 3, 5, and 7 dilation patterns; with geometric vs. arithmetic progression of $w$ and $r$ ; with dynamic vs. learned vs. uniform weights — these would illuminate which aspects of the design matter most and whether the geometric progression is genuinely necessary or just one of many workable configurations.
Multiple random seeds and confidence intervals. The paper reports single-run perplexity values without variance estimates. Given that the gaps between methods are sometimes small (e.g., 4.23 vs. 4.39 at 2K evaluation), it is impossible to assess whether these differences are statistically significant or within the noise of training stochasticity.
Training set dependency measurement with controlled distance. An experiment measuring perplexity specifically on tokens whose most informative context token is at a known distance (close, medium, far) would reveal how LONGNET's multi-scale attention differentially handles local vs. long-range prediction, and whether the dynamic weighting mechanism successfully routes information through the appropriate scale.
Evaluation on a standard long-range benchmark. The Long Range Arena (LRA) benchmark is the standard evaluation suite for efficient Transformers. Testing LONGNET on LRA tasks (ListOps, text classification, retrieval, image, Pathfinder) would provide comparable metrics to a large body of prior work and situate LONGNET's performance in the broader efficient Transformers literature. The paper's exclusive use of language modeling perplexity on The Stack limits comparability.

6. Limitations and Trade-offs

The "1 Billion Token" Claim Is a Systems Demonstration, Not a Quality Guarantee

The assumption or constraint. The paper's title and central claim — "Scaling Transformers to 1,000,000,000 Tokens" — is supported by runtime measurements in Figure 5 showing that dilated attention with FlashAttention maintains nearly constant latency when processing forward passes on sequences up to 1B tokens. However, every language modeling quality experiment in the paper (Table 2, Figure 6, Figure 7) trains and evaluates at sequence lengths between 2K and 32K — a factor of roughly 30,000× shorter than the claimed maximum. The paper acknowledges this implicitly by separating the systems results (Section 3.2, Figure 5) from the modeling results (Section 4) and never claiming quality at 1B tokens. Yet the title and abstract ("can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences") create the impression that the model works well at these lengths, not merely that it can process them without crashing.

The consequence. A practitioner cannot determine from this paper whether LONGNET would actually produce useful predictions when given 1B tokens of context. Several failure modes become plausible at extreme lengths that would not appear at 32K. Gradient propagation across ~20 attention hops (the ~log₂(10^9 / 2048) layers needed for end-to-end connectivity) may attenuate training signals, making it difficult to learn dependencies that span the full context. The model's fixed hidden dimension (768 in the base configuration) may lack the capacity to store and integrate information from 1B tokens of context, regardless of attention connectivity — the bottleneck shifts from access to representation. The training data itself (The Stack, a code dataset) may simply not contain meaningful dependencies at billion-token distances (most source files are far shorter), meaning the architectural capability has no training signal to exploit. The paper's own evidence that block-wise causal attention fails catastrophically for large extrapolation gaps (Table 2: models trained at 2K reach ~11 PPL when evaluated at 32K, versus ~3 PPL when trained at 32K) ironically demonstrates why extrapolating from 32K to 1B is unreliable.

What evidence exists in the paper. Figure 5 demonstrates runtime scaling to 1B tokens — but this is a forward-pass latency measurement, not a quality measurement. The paper reports no perplexity, accuracy, or any task metric at sequence lengths beyond 32K. The distributed training algorithm (Section 3.1, Equations 21–25) is described in general terms but is only validated as a runtime benchmark, not as part of an end-to-end training run with loss curves or convergence analysis. The paper's statement that "each model of different sequence lengths has up to 3 segment lengths, which are 2,048, the number of tokens per device, and the sequence length" (Section 3.2) confirms that the 1B-token configuration was tested structurally, but not trained to convergence.

Mitigation status. The paper does not address this gap. The title and framing present the 1B-token scaling as an achieved capability, but the experiments only validate it as a computational feasibility result. Future work on training LONGNET at 128K, 512K, or 1M tokens with reported perplexity would bridge this gap, as would a synthetic long-range dependency task (e.g., retrieve a token from 100K positions back) demonstrating that the logarithmic connectivity actually enables usable information propagation at scale.

Difficulty Estimation Cost Is Entirely Externalized from the Headline Efficiency Numbers

The assumption or constraint. LONGNET's mixture-of-dilations design requires selecting $k$ dilation patterns with specific segment sizes $w_i$ and dilation rates $r_i$ . The paper prescribes geometric sequences (Equations 11–12) and uses $w = \{2048, 4096, 8192, 16384, 32768\}$ and $r = \{1, 2, 4, 6, 12\}$ for the 32K experiments. These choices are presented as architectural constants, but they implicitly depend on knowing the maximum sequence length $N$ in advance — the number of patterns $k$ must satisfy $w_k \approx N$ , requiring $k = \lceil \log_\alpha (N / w_0) \rceil$ . For $N = 10^9$ and $w_0 = 2048$ , this means $k \approx 19$ patterns must be configured before training. More subtly, the per-device communication pattern in the distributed algorithm (Section 3.1) depends on knowing which segment lengths are locally computable ( $w_i \leq \ell$ ) and which require cross-device all-gather ( $w_i > \ell$ ). If a practitioner deploys LONGNET on a different hardware configuration or with a different maximum sequence length, these choices must be recomputed — but the paper provides no method for doing so automatically or for adapting them at runtime.

The consequence. This is not merely a hyperparameter tuning concern. The dilation pattern configuration fundamentally determines the tradeoff between local precision and global reach, and there is no guarantee that the geometric progression with $\alpha = 2$ and $w_0 = 2048$ is optimal for all sequence lengths, all datasets, or all model sizes. A sequence length of 1M tokens might benefit from a smaller $w_0$ (finer local attention) and more patterns, while a sequence length of 1B might need a larger $w_0$ to keep $k$ manageable. The paper provides no guidance on how to set these parameters for a new deployment scenario — they are treated as fixed design choices validated only at the specific tested configuration (MAGNETO base model, The Stack dataset, 2K–32K sequence lengths). A practitioner scaling to a new domain (e.g., genomic data with different dependency structures, or multi-modal data with heterogeneous token types) has no principled way to select $w_0$ , $\alpha$ , or $k$ .

What evidence exists in the paper. The experiments use exactly one dilation configuration: $w = \{2048, 4096, 8192, 16384, 32768\}$ and $r = \{1, 2, 4, 6, 12\}$ . The paper notes that $r$ deviates from a perfect geometric progression ( $r=6$ instead of $8$ , $r=12$ instead of $16$ ) but does not explain why or ablate alternatives. No experiments vary $w_0$ (e.g., testing 1024 or 4096 as the base segment size), vary $\alpha$ (e.g., testing ratio 1.5 or 3), vary $k$ (e.g., using 3 or 7 patterns), or compare geometric against arithmetic progressions. The theoretical analysis in Equations 18–20 proves that geometric sequences guarantee $\mathcal{O}(Nd)$ complexity and $\mathcal{O}(\log N)$ dependency, but does not characterize how sensitive quality is to deviations from exact geometric progression or to the specific choice of ratio.

Mitigation status. The paper acknowledges none of these configuration dependencies as limitations. The geometric progression is presented as part of the method definition, not as a hyperparameter requiring tuning. Future work could develop an automated method for selecting $w_0$ and $\alpha$ based on sequence length and available compute, or could demonstrate that model quality is robust across a wide range of these parameters.

The Method Is Validated on a Single Model Family, Single Dataset, and Single Domain

The assumption or constraint. All experiments use the MAGNETO backbone architecture (Wang et al., 2022) with XPOS relative position encoding (Sun et al., 2022), trained on The Stack code dataset (Kocetkov et al., 2022). The model architecture, position encoding scheme, training data distribution, and task domain (code language modeling) are held fixed. The paper's motivation (Section 1) argues for extreme sequence length scaling across broad application domains — "interact with human and the world," "complex causality and reasoning paths," "in-context learning as a paradigm shift," and the conclusion mentions multimodal large language modeling, BEiT pretraining, and genomic data modeling — but none of these are evaluated.

The consequence. Several aspects of LONGNET's design could be sensitive to domain, model architecture, or position encoding choices. The geometric dilation pattern presumes that dependencies in the data follow a roughly scale-invariant structure where relevant information is distributed across distances from local to global — this may hold for code (where a function call may reference a definition thousands of lines away, but most dependencies are within-file) but could fail for domains with fundamentally different dependency structures. Genomic data, for instance, has regulatory elements acting over characteristic genomic distances (enhancers can regulate genes millions of base pairs away, but the distribution is not scale-free in the same way). The XPOS relative position encoding was specifically designed for length extrapolation and may interact favorably with dilated attention's multi-scale structure in ways that absolute position encodings or RoPE would not — the paper provides no ablation to assess this dependency. The MAGNETO architecture's pre-normalization and DeepNorm initialization may affect gradient flow through dilated attention paths differently than post-normalization architectures like the original Transformer. Without experiments across model families and domains, the paper's claim that dilated attention is a "drop-in replacement for standard attention" (abstract) is supported only for the specific combination of MAGNETO + XPOS + The Stack.

What evidence exists in the paper. All experiments (Table 2, Figures 6, 7) use the identical backbone architecture and training data. The paper does not report results on standard long-range benchmarks (Long Range Arena; Tay et al., 2021), which would provide comparability to the broader efficient Transformers literature. It does not test on natural language (e.g., WikiText, PG-19, Books3), multimodal data, or genomic sequences. It does not ablate the position encoding choice or the backbone architecture. The claim of being a "drop-in replacement" is supported by the implementation compatibility argument (gather-scatter dense attention reuses FlashAttention kernels) but not by empirical evidence that simply swapping standard attention for dilated attention works across diverse architectures and tasks without additional tuning.

Mitigation status. The paper does not acknowledge this as a limitation. The conclusion mentions future work extending LONGNET to "multimodal large language modeling, BEiT pretraining, and genomic data modeling" but frames these as straightforward extensions rather than validations that the current paper lacks. A practitioner considering LONGNET for a non-code domain or a different Transformer variant would need to conduct their own validation from scratch.

The Dynamic Weighting Claim Is Theoretically Interesting but Empirically Unvalidated

The assumption or constraint. Section 2.2 states that the mixture-of-dilations weights $\alpha_i = s_i / \sum_j s_j$ (where $s_i$ is the softmax denominator for pattern $i$ ) are "better than learnable fixed weights." This claim is central to Innovation 2 in this analysis (dynamic, content-dependent scale selection with zero additional parameters), and the paper provides a theoretical equivalence argument: the weighted sum with these $\alpha_i$ is equivalent to computing one joint softmax over the union of all patterns' keys. The mechanism is presented as a key architectural contribution distinguishing LONGNET from prior fixed-pattern sparse attention methods.

The consequence. The paper provides no experiment comparing dynamic softmax-denominator weighting against any alternative — not learned scalar weights, not uniform averaging, not attention-based gating, not hard selection (picking the single best pattern per token). Without such a comparison, the reader cannot assess whether the dynamic weighting mechanism actually improves quality, how much it contributes relative to other design choices (geometric progression, multi-head offsets, distributed training), or whether simpler alternatives would work equally well. The theoretical equivalence to a joint softmax is elegant but does not by itself prove that this particular weighting scheme is better than alternatives — it only proves that the chosen weights have a clean mathematical interpretation. The claim that dynamic weights "are better than learnable fixed weights" is an empirical assertion presented without empirical evidence.

What evidence exists in the paper. None. No table, figure, or ablation compares weighting schemes for the mixture-of-dilations combination. The dynamic weighting is described in Equations 9–10 and justified with the equivalence argument, but its contribution to model quality is never isolated or measured.

Mitigation status. The paper does not acknowledge this gap. The claim is presented as a design choice with a theoretical motivation, not as a hypothesis requiring validation. An ablation comparing dynamic weights against learned weights, uniform weights, and attention-based gating would directly test whether this mechanism matters — and would either strengthen the paper's contribution (if dynamic weights win) or simplify the method (if a simpler alternative works equally well).

Communication and Difficulty Estimation Overhead for Distributed Training Are Not Factored into Efficiency Comparisons

The assumption or constraint. The FLOPs comparisons in Table 2 and Figure 6 account only for the matrix multiplication operations in attention computation, following the derivation in Equation 18. The distributed training algorithm (Section 3.1) requires all-gather operations to collect sparsified keys and values across devices whenever $w_i > \ell$ (where $\ell$ is the per-device sequence length). The paper argues that these operations have constant communication cost because $\tilde{K}_i$ and $\tilde{V}_i$ have size $\frac{w_i}{r_i} \times d$ , independent of $N$ . However, this argument addresses only the per-pattern per-device communication volume — it does not account for the latency of the all-gather operations, the synchronization overhead across devices, or the fact that communication bandwidth, not FLOPs, often bottlenecks distributed training. The Figure 5 runtime measurements include communication costs and show near-constant latency, but these are forward-pass measurements on a specific hardware configuration, not training measurements that include backward-pass communication (reduce-scatter) and optimizer synchronization.

The consequence. A practitioner deploying LONGNET on a multi-GPU system for training (not just inference) needs to understand the end-to-end training throughput, not just the forward-pass latency. The backward pass requires reduce-scatter operations for gradients with respect to $\tilde{K}$ and $\tilde{V}$ , which have the same communication volume as the forward-pass all-gather but add additional synchronization points. At very large device counts, the all-gather latency scales with the number of devices (typically logarithmically for tree-based algorithms, but with non-trivial constant factors). The paper's FLOPs-based comparison to dense Transformers therefore understates the real-world training cost difference, particularly at scale — the constant communication cost is constant in $N$ but still grows (sublinearly) with the number of devices. Moreover, the paper's own difficulty estimation is effectively free in the reported comparisons because the dilation pattern configuration ( $w$ , $r$ , $k$ ) is chosen manually before training — but in any deployment where these parameters need to be tuned per-dataset or per-hardware-configuration, the cost of that tuning is externalized from the reported efficiency numbers.

What evidence exists in the paper. Figure 5 provides forward-pass runtime measurements including communication, showing nearly flat scaling from 8K to 1B tokens. However, these are measured on a system where the total tokens per batch is held constant at 1B (batch size decreases as sequence length increases), meaning the device count and per-device computation change across the x-axis in ways that are not fully specified. The paper does not report backward-pass runtime, end-to-end training step time, or training throughput (tokens per second) for the distributed configuration. Table 2 and Figure 6 report only FLOPs-based compute comparisons, not wall-clock training time comparisons — so the claim that LONGNET is "more efficient" than dense Transformers (Figure 6) is a FLOPs claim, not necessarily a time-to-convergence or cost-to-train claim.

Mitigation status. The paper is partially transparent — Figure 5 does include communication costs in the runtime measurements, and the text in Section 3.2 notes that "both of them are implemented with FlashAttention Kernel for saving memory and improving speed." However, the FLOPs comparisons in the quality experiments (Section 4) do not account for communication, and the paper does not discuss the practical implications for training cost on large clusters. A practitioner would need to benchmark LONGNET on their specific hardware configuration to determine actual training throughput.

Hard Problems at Extreme Lengths — the Gap Between Logarithmic Connectivity and Usable Context Remains Uncharacterized

The assumption or constraint. The paper proves that dilated attention achieves $\mathcal{O}(\log N)$ maximum path length between any two tokens (Equation 20), establishing that information can propagate across the full sequence in roughly $\log_\alpha (N / w_0)$ layers. For $N = 10^9$ and $\alpha = 2$ , $w_0 = 2048$ , this gives approximately 19 layers — within practical Transformer depths. The paper's language modeling experiments (Table 2, Figure 7b) show that perplexity improves as sequence length increases from 2K to 32K, consistent with the model successfully using longer context.

The consequence. Logarithmic connectivity is a necessary condition for long-range information propagation, but it is not sufficient. The paper does not characterize how much information actually propagates across the maximum path length, or whether the signal-to-noise ratio of long-range dependencies degrades with distance. Each attention hop involves a softmax over a set of keys — information from a token 19 hops away has been re-weighted, mixed with other tokens' values, and passed through nonlinearities 19 times. The effective information that survives this process may be negligible even though the connectivity graph theoretically supports it. This is analogous to the vanishing gradient problem in very deep networks: connectivity exists (every layer is connected to the next), but useful gradients may not propagate. For LONGNET, the question is whether attention-weighted information propagates effectively across many hops, particularly when each hop's attention distribution is computed over a sparsified key set that may not include the optimal intermediate routing tokens.

The paper provides no measurement of effective context utilization as a function of distance. The language modeling perplexity improvements in Figure 7b could be driven entirely by tokens within the first few thousand positions, not by genuinely long-range dependencies. A model that effectively uses only the nearest 8K tokens would still show improving perplexity as context grows from 1K to 8K (because it gains relevant local context), but would plateau beyond 8K. Figure 7b shows improvement from 1K to 32K, but 32K is only 1–2 effective hops beyond the base segment size of 2048 — this does not stress-test multi-hop propagation.

What evidence exists in the paper. The paper reports no experiment that measures dependency on information at controlled distances. There is no synthetic retrieval task ("output the token at position N−K"), no probing analysis of attention head behavior at different distance ranges, and no evaluation on a benchmark like Long Range Arena that specifically tests long-range dependency capture. The token dependency analysis (Equation 20) is purely theoretical. The runtime scaling in Figure 5 demonstrates that the architecture can process 1B-token sequences, but provides zero evidence that any useful information propagates across those distances.

Mitigation status. The paper does not address this gap. The theoretical $\mathcal{O}(\log N)$ result is presented as a guarantee that the architecture can model long-range dependencies, but the distinction between connectivity and usable information propagation is not discussed. Future work could address this with controlled-distance probing tasks, gradient flow analysis across attention hops, or comparison of perplexity improvements stratified by the distance of the most informative context token.

7. Implications and Future Directions

How This Work Changes the Landscape

LONGNET represents a genuine architectural breakthrough rather than an incremental refinement, but its impact is more precisely characterized as opening a new design space than as immediately reshaping practice. The paper's core contribution is not a faster attention kernel or a better approximation to softmax — it is the demonstration that exponential dilation can resolve the fundamental feasibility-expressivity tension that had constrained every prior attempt at sequence length scaling. This shifts the conversation from "how much quality must we sacrifice to reduce attention complexity?" to "what connectivity patterns are better for learning, independent of computational cost?"

The most consequential reframing is the elevation of multi-scale attention from an implementation trick to an architectural first-class principle. Prior sparse attention methods (Sparse Transformer's local + strided, Big Bird's random + local + global, Longformer's sliding window with global tokens) combined patterns heuristically, typically with one pattern for local precision and one for global coverage — a two-scale design chosen by the engineer. LONGNET's geometric progression of dilation rates ( $w$ and $r$ doubling with each pattern, producing a complete exponential hierarchy of scales from local to global) demonstrates that there exists a continuous spectrum of attention scales, and that covering this spectrum systematically — with dynamic, content-dependent weighting — yields models that match or exceed dense attention quality while maintaining linear complexity. This is analogous to the shift in computer vision from single-scale ConvNets to feature pyramid networks (FPNs): the recognition that information exists at multiple spatial frequencies and that architectures should process all of them simultaneously, not pick one.

The paper also resolves a latent contradiction in the efficient Transformers literature. Several prior linear-complexity methods (Performer, Linear Transformer, Linformer) showed competitive perplexity on short sequences but degraded on long-range tasks like LRA, while state space models (S4, H3, Hyena) excelled on long-range benchmarks but underperformed Transformers on standard language modeling. The implicit narrative was that linear complexity and Transformer-quality expressivity were mutually exclusive — you could have one or the other, not both. LONGNET's results in Table 2 and Figure 6 demonstrate that this tradeoff is not fundamental: the same architecture matches dense Transformer perplexity at 2K sequence length (4.23 vs. 4.24 PPL) while simultaneously scaling to 32K with improving quality, following power-law scaling behavior (Figure 7a) indistinguishable from dense Transformers up to 2.7B parameters. The contradiction in prior work reflected inadequate attention pattern design, not an inherent limitation of sparse attention. This finding redirects research attention from alternative sequence modeling paradigms (state space models, recurrent architectures) back toward sparse attention — but with the crucial insight that the pattern matters more than the sparsity ratio.

The paper's distributed training contribution (Section 3) is equally landscape-shifting, though in a different register. It establishes that the sequence dimension can be parallelized with constant communication cost — a property that vanilla attention's quadratic structure fundamentally prevents, and that no prior sparse attention method had exploited. This is not merely an optimization; it is a proof of concept that architecture design and distributed systems design are the same problem. The linear complexity structure that yields $\mathcal{O}(Nd)$ FLOPs also yields $\mathcal{O}(1)$ communication per device because the sparsified key-value representations depend only on $w_i / r_i$ , not on $N$ . This insight — that the geometric series bounding computation also bounds communication — provides a template for future long-sequence architectures: design connectivity patterns whose distributed implementation has bounded communication, not just bounded FLOPs. In practical terms, this means that sequence length scaling is no longer gated by single-device memory; the bottleneck shifts to total cluster throughput, which is far more scalable.

However, the paper's impact is constrained by a significant gap between its demonstrated capability and its claimed capability. The 1B-token scaling is validated as a runtime and memory claim (Figure 5) but not as a quality claim — no perplexity, accuracy, or task metric is reported beyond 32K. The paper's own evidence that block-wise causal attention fails for large extrapolation gaps (Table 2: models trained at 2K catastrophically degrade at 32K) ironically underscores why the reader should not extrapolate from 32K quality to 1B quality. This gap means that LONGNET currently opens a design space but does not populate it with evidence — the architecture can process billion-token sequences, but whether it can learn from them, and whether training at such lengths yields proportional quality improvements, remains unproven. The landscape shift is therefore foundational but incomplete: the community now knows that linear-complexity attention with logarithmic connectivity is achievable without quality degradation, and that distributed sequence parallelism with constant communication is possible, but does not know what happens to model quality when these capabilities are pushed to their limits.

Follow-Up Research This Work Enables

Training LONGNET at 128K–1M tokens with perplexity measurement and controlled-distance probing. The most urgent gap in the paper is the absence of any quality metric beyond 32K sequence length. A direct follow-up would train LONGNET (identical architecture, same MAGNETO backbone, same Stack dataset) at 128K, 256K, 512K, and 1M tokens — sequence lengths where the multi-hop logarithmic connectivity starts to matter. At 128K tokens, the maximum path length is $\log_2(128K / 2048) \approx 6$ hops; at 1M tokens, $\approx 9$ hops. Measuring perplexity as a function of training length would reveal whether the quality improvements observed from 2K→32K (Table 2, Figure 6) continue or plateau. More importantly, this experiment should include controlled-distance probing: construct a synthetic evaluation set where the correct next-token prediction depends on a token at a known distance $d$ (e.g., retrieve a variable name defined $d$ tokens earlier), and measure accuracy as a function of $d$ . This would directly test the paper's central theoretical claim — that $\mathcal{O}(\log N)$ connectivity translates to usable information propagation — and would reveal the effective context utilization horizon, which may be substantially shorter than the architectural maximum.

Head-to-head comparison with state space models on both short-sequence quality and long-range benchmarks. The paper's motivation (Section 1) explicitly critiques state space models (S4, H3, Hyena) for limited expressivity on regular-length sequences while acknowledging their strong long-range performance. However, no experimental comparison is provided. A rigorous follow-up would train LONGNET, H3, and a dense Transformer baseline at matched parameter counts (125M, 350M, 760M) on The Stack and evaluate on: (a) standard language modeling perplexity at 2K, 8K, 32K; (b) Long Range Arena (LRA) tasks, particularly the retrieval task (which requires matching tokens across 4K positions) and the Pathfinder task (which requires detecting long-range spatial structure); (c) a synthetic extreme-distance retrieval task (matching tokens at 8K, 16K, 32K, 64K distances) that stresses the architectural limits. The key question: does LONGNET's exact-attention-over-subsampled-keys design actually deliver better short-sequence quality than state space models (as the paper's motivation implies), while matching or exceeding their long-range capabilities? If LONGNET wins on both fronts, it establishes dilated attention as the dominant paradigm; if state space models win on long-range tasks despite their alleged expressivity limitations, the paper's theoretical advantage does not translate to practice.

Ablation of the geometric progression: does exact geometric spacing matter, or only multi-scale coverage? The paper's theoretical analysis (Equations 18–20) proves that geometric sequences guarantee $\mathcal{O}(Nd)$ complexity and $\mathcal{O}(\log N)$ dependency, but the experiments use slightly non-geometric dilation rates ( $r = \{1, 2, 4, 6, 12\}$ instead of $\{1, 2, 4, 8, 16\}$ ) without explanation or ablation. A systematic follow-up would vary the dilation pattern along three axes: (a) geometric vs. arithmetic vs. random progression of $w$ and $r$ , while keeping total FLOPs constant; (b) the geometric ratio $\alpha$ (testing 1.5, 2, 3, 4) to assess whether the convergence speed of the FLOPs series matters in practice; (c) the number of patterns $k$ (from 2 to 10) to identify the point of diminishing returns. The hypothesis is that any multi-scale coverage (not necessarily exact geometric spacing) yields most of the benefit, and that the geometric progression is sufficient but not necessary — but the paper provides no evidence either way. A negative result (random pattern assignment works equally well) would simplify LONGNET substantially; a positive result (geometric spacing matters significantly) would validate the theoretical apparatus as practically important.

Dynamic weighting ablation: softmax denominator vs. learned vs. uniform vs. hard selection. The paper claims that weighting dilation patterns by their softmax denominators $\alpha_i \propto s_i$ is "better than learnable fixed weights" (Section 2.2) but provides zero experimental comparison. A clean ablation would compare four weighting schemes at identical FLOPs and model size: (a) softmax-denominator weighting (the paper's method); (b) learned scalar weights per pattern (one learnable parameter per pattern, applied uniformly across all tokens); (c) uniform averaging ( $\alpha_i = 1/k$ ); (d) hard selection via gating (a small learned router that selects the single best pattern per token, trained with straight-through estimation). The experiment should also analyze the per-token variation in $\alpha_i$ under the softmax-denominator scheme — do different tokens genuinely use different patterns, or does the weighting collapse to a nearly uniform distribution? If uniform averaging performs equally well, the dynamic weighting mechanism is unnecessary complexity; if learned weights perform better, the paper's claim is false. Either outcome is informative.

Scaling law comparison: LONGNET vs. dense Transformer power-law coefficients. Figure 7(a) shows LONGNET following a power-law scaling curve for models from 125M to 2.7B parameters, but no dense Transformer curve is plotted for direct comparison. A rigorous follow-up would train matched dense Transformer and LONGNET models at 125M, 350M, 760M, 1.5B, and 2.7B parameters on identical data, then fit power laws of the form $L(C) = a C^b + c$ to both sets of loss curves. The key comparison is the exponent $b$ : if LONGNET's exponent is less steep (closer to zero) than dense Transformers', it means LONGNET scales worse with compute — the sparse attention imposes a scaling penalty that becomes visible only at larger scales. If the exponents are equal but LONGNET has a lower constant factor $a$ , it means LONGNET is strictly more compute-efficient at all scales. If LONGNET's exponent is more favorable (more negative), it means sparse attention actually improves scaling behavior — perhaps by preventing the overfitting to local correlations that the paper's motivation hypothesizes. This experiment would directly address the paper's unsubstantiated claim that "the dense Transformer is not a prerequisite for scaling the language models" and would provide actionable guidance for practitioners deciding whether to adopt dilated attention at billion-parameter scales.

Synthetic extreme-length task: can information propagate across 20+ attention hops? The paper proves $\mathcal{O}(\log N)$ connectivity but provides no evidence that useful information survives multi-hop propagation through dilated attention. A targeted stress test would construct a synthetic dataset where each sequence is 1M tokens long, and the correct prediction at position $N$ depends on a single token at position $N - d$ , with $d$ varied from 1K to 500K. The input between the informative token and the prediction position is filled with random tokens, so the model cannot use local context — it must propagate information across the specified distance. Train LONGNET on this task and measure accuracy as a function of $d$ . The architectural prediction is that accuracy remains high for $d$ up to the effective receptive field of the model (roughly $w_0 \times \alpha^L \approx 2048 \times 2^{12} \approx 8.4 \times 10^6$ for 12 layers with $\alpha=2$ ), and degrades beyond that. The practical finding might be that effective propagation is much shorter — perhaps 50K–100K tokens — due to softmax attention's tendency to produce diffuse attention distributions over large key sets, or due to gradient attenuation across many hops. This experiment would calibrate the gap between architectural connectivity and usable context, providing a realistic expectation for what "1 billion token context" means in practice.

Practical Applications and Downstream Use Cases

Long-document and codebase-level understanding for developer tools. LONGNET's demonstrated ability to process sequences up to 32K tokens with quality matching or exceeding dense Transformers (Table 2: 3.01 PPL at 32K vs. 4.24 at 2K for dense Transformer) directly enables models that can attend across entire code files, documentation, and multi-file contexts without truncation or retrieval. A developer tool using LONGNET could ingest a complete codebase snapshot — source files, documentation, issue tracker context, and conversation history — as a single sequence, with the model attending directly to relevant definitions and discussions regardless of their position. The 4× improvement in perplexity from 2K to 32K training length (11.29 → 3.01 PPL, Table 2) suggests substantial quality gains from this expanded context. The linear complexity means that doubling the context window from 32K to 64K approximately doubles the inference cost (rather than quadrupling it, as with dense attention), making it economically feasible to offer context-sensitive code completion and bug detection across entire repositories. The key practical advantage over retrieval-augmented approaches is that attention is end-to-end differentiable and can learn which context is relevant, rather than relying on a separate retrieval model that may miss critical dependencies.

Many-shot in-context learning for specialized domains. The paper's motivating argument — that extreme context enables "many-shot learning" as an alternative to fine-tuning — becomes practically actionable with LONGNET's linear scaling. Consider a medical question-answering system that needs to adapt to a new hospital's patient record format, clinical guidelines, and formulary. Rather than fine-tuning (which requires curated training data, GPU resources, and validation), a LONGNET-based model could ingest thousands of example QA pairs, clinical notes, and guideline documents directly in the context window. The paper's Figure 7(b) shows that test loss drops from ~2.2 at 1K context to ~1.6 at 32K context on code data, demonstrating that the model can leverage additional context for improved predictions. Extrapolating this trend (with appropriate caution), a 128K context window might yield further gains. The economic calculus shifts: the cost of generating a 128K-token prompt with LONGNET is roughly 4× the cost of a 32K-token prompt (linear scaling), while fine-tuning a comparable model might cost 100–1000× more in compute and engineering time. For organizations that need to customize models to rapidly changing or highly specialized data distributions, LONGNET makes in-context adaptation the default strategy rather than the fallback.

Genomic sequence modeling with single-base-pair resolution across entire chromosomes. The paper's conclusion mentions genomic data modeling as a future direction, and the architectural properties align well with the domain's needs. The human genome is approximately 3 billion base pairs — within LONGNET's demonstrated runtime scaling range (Figure 5: 1B tokens at ~2000 ms). Genomic modeling requires detecting regulatory elements (enhancers, promoters) that can act over distances of millions of base pairs, making it a canonical long-range dependency problem. Current approaches typically use convolutional architectures with limited receptive fields or bin the genome into coarse regions, losing single-base-pair resolution. LONGNET trained with a 4-base vocabulary (A, T, G, C) at 1M–10M token sequence lengths could ingest entire gene clusters or chromosomal arms, attending across regulatory elements and their target genes in a single forward pass. The geometric dilation pattern is particularly well-suited to genomic data: most regulatory interactions are local (within a few thousand base pairs), but critical long-range interactions (enhancer-promoter loops) span hundreds of thousands to millions of base pairs — exactly the multi-scale structure that the mixture of dilations captures. The key validation needed is whether the logarithmic connectivity actually propagates regulatory signals across megabase distances; a successful demonstration would likely become the standard architecture for genomic sequence models, displacing current convolution-based and binning-based approaches.

Cost-efficient batch inference for document processing pipelines. Organizations that process large volumes of long documents — legal document review, scientific literature screening, financial report analysis — currently use chunking strategies that split documents into fixed-size windows with overlap, process each chunk independently, and merge results heuristically. This approach loses cross-chunk context (a key definition in chunk 1 might be essential for understanding chunk 50) and wastes computation on overlapping regions. LONGNET enables processing each document as a single sequence, with the attention mechanism naturally handling cross-reference resolution. The paper's FLOPs comparison (Figure 6: LONGNET at 32K achieves lower perplexity than dense Transformers at any length, with fewer FLOPs) translates directly to cost savings: for a fixed quality target, LONGNET requires fewer total FLOPs per document, and for a fixed FLOPs budget, LONGNET produces higher-quality predictions. The practical deployment is a batch inference pipeline where documents of varying lengths (from 1K to 100K+ tokens) are processed with a single model, with per-document cost scaling linearly rather than quadratically with length — enabling processing of documents that would be economically infeasible with dense attention.

When to Prefer This Method

The paper explicitly positions LONGNET against vanilla dense attention and Sparse Transformer (fixed local + strided), offering clear conditions for preference grounded in the experimental results.

Prefer LONGNET over vanilla dense attention when the target sequence length during training exceeds ~8K tokens. At 2K sequence length, LONGNET and dense Transformers achieve essentially identical perplexity (4.23 vs. 4.24 PPL, Table 2), so there is no quality penalty for the architectural change. At 8K and beyond, LONGNET's quality advantage widens while its FLOPs per token remain constant, whereas dense attention's FLOPs grow quadratically. The cross-over point in practice depends on hardware memory constraints: on single-GPU setups where 8K dense attention still fits in memory, the FLOPs savings alone may not justify switching, but on any multi-GPU or memory-constrained deployment targeting >8K context, LONGNET is strictly better on both quality and efficiency metrics.
Prefer LONGNET over Sparse Transformer (fixed local + strided pattern, Child et al., 2019) when the training sequence length exceeds 8K and the domain involves multi-scale dependencies (code, long-form text, genomic data) rather than purely local structure. Table 2 shows LONGNET outperforming Sparse Transformer across all sequence lengths at matched FLOPs, with the gap growing from 0.16 PPL at 2K to 0.63 PPL at 32K. The geometric progression of dilation patterns provides complete scale coverage (from 2048-token local windows to full-sequence global attention) that Sparse Transformer's two-pattern design (local + strided) misses — particularly at intermediate distances where neither local nor strided patterns are optimal.
Prefer LONGNET over state space models (S4, H3, Hyena) when short-sequence quality cannot be sacrificed for long-range capability — for instance, in a general-purpose language model that must perform well on both short prompts (chat, QA) and long-document tasks. The paper does not experimentally compare against state space models, but its motivation (Section 1) argues that state space models "perform well at long-range benchmarks [but] their performance on regular lengths is not as good as Transformers, limited mainly by the model expressivity." If this claim holds (which future work should verify), LONGNET offers the best of both worlds: Transformer-quality short-sequence modeling with linear-complexity long-sequence scaling.
Prefer LONGNET over retrieval-augmented approaches (Memorizing Transformers, kNN-LM) when the task requires end-to-end differentiable attention across the full context, rather than discrete retrieval from an external memory. Retrieval-based methods can scale to arbitrary context sizes by storing the entire corpus in a vector database, but the retrieval step is non-differentiable and may miss relevant context that falls outside the similarity search radius. LONGNET's attention mechanism is fully differentiable and can learn which distant context is relevant through gradient-based training — important for tasks where "relevance" is not well-captured by embedding similarity (e.g., reasoning tasks where the connection between a premise and conclusion is logical rather than semantic).
Do not prefer LONGNET when the maximum sequence length is guaranteed to be short (<2K tokens) and the overhead of implementing the mixture-of-dilations is not justified — dense attention is simpler and equally performant at this scale. Do not prefer LONGNET (in its current, unvalidated state beyond 32K) when the application requires guaranteed usable context at million-token scales — the paper demonstrates runtime scaling to 1B tokens but provides zero quality evidence beyond 32K, and practitioners needing reliable billion-token context should await validation at those lengths or invest in their own large-scale training experiments.