Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

ArXiv: 2605.16928

🎯 Pitch

Standard full-attention LLMs already hide internal sparsity: only a few retrieval heads need long context, and they serve a low-dimensional subspace that can be queried with a tiny 16-dim indexer. RTPurbo surgically unmasks this sparsity with just a few hundred training steps, giving you up to 9.36× prefill speedup at 1M length while perfectly matching the dense model on hard reasoning benchmarks.

---

1. Executive Summary

This paper proposes RTPurbo, a method that transforms standard full-attention LLMs into highly sparse models with only a few hundred training steps, achieving near-lossless long-context inference without native sparse pretraining. Evaluated on Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think across long-context benchmarks (LongBench, RULER) and reasoning tasks (AIME24/25, MMLU-PRO), RTPurbo exploits three intrinsic properties: head specialization into retrieval vs. local groups (retaining full KV cache only for retrieval heads), a low-dimensional retrieval subspace induced by RoPE (a 16-dimensional projector achieving over 90% recall), and dynamic top-p thresholding (adapting per-query token budgets, avoiding the recall-failure of fixed top-k). The method delivers up to a 9.36× prefill speedup at 1M context and roughly a 2.01× decode speedup, while preserving near-lossless accuracy—perfectly matching the dense baseline on AIME and sustaining robust accuracy on ultra-long multi-hop tasks at 512K context. The results establish that strong sparse inference can be obtained from standard full-attention training with minimal post-hoc adaptation, but only when the model exhibits stable head specialization that survives sparsification.

2. Context and Motivation

The Core Problem: Full Attention Is Quadratic, and Sparse Alternatives Demand Painful Trade-offs

The fundamental challenge RTPurbo addresses is the quadratic cost of full attention in long-context inference. In a standard transformer, computing attention scores between a query and all keys in a sequence of length $L$ requires $O(L^2)$ time and memory. When $L$ reaches tens or hundreds of thousands of tokens—as demanded by modern applications like multi-turn dialogue, document understanding, and long-horizon reasoning—this quadratic growth makes inference prohibitively expensive. The prefill phase (processing the entire prompt) becomes a computational bottleneck, and the decode phase (generating tokens autoregressively) remains memory-bound because the full KV cache must be accessed at every step.

The practical significance of this problem is immense. As the paper notes (Section 1), long-context capability has become a core requirement for modern LLMs. Models are routinely deployed with context windows of 128K, 256K, or even 1M tokens (Qwen2.5-1M, Gemini 2.5, Kimi K2). Without efficient attention mechanisms, the cost of serving these models at scale—both in GPU hours and in latency—becomes unsustainable. A method that can reduce attention cost while preserving output quality directly translates to lower infrastructure costs, faster response times, and the ability to deploy long-context models on more modest hardware.

The Existing Landscape: Three Families of Sparse Attention, Each with a Different Weakness

Prior approaches to efficient long-context attention fall into three broad categories, each with a characteristic limitation that RTPurbo is designed to overcome.

1. Pattern-Based Sparsity (MInference, FlexPrefill, DuoAttention, RazorAttn)

These methods exploit the observation that different attention heads exhibit different behavior: some heads attend broadly, others focus narrowly. MInference assigns each head an offline-discovered sparse pattern (e.g., vertical-slash, block-sparse). FlexPrefill makes the pattern selection context-aware. DuoAttention and RazorAttention explicitly partition heads into retrieval (requiring long-range access) and streaming (local-only) groups and treat them differently.

The key limitation is that these methods, while partially effective, lack a mechanism for fine-grained, query-dependent token selection within retrieval heads. They make a binary decision—this head is "retrieval" or "local"—but within a retrieval head, they either retain all tokens (DuoAttention, RazorAttn) or apply a static pruning rule that doesn't adapt to the specific query. As RTPurbo's Figure 3 dramatically demonstrates, the same retrieval head can need ~8,500 tokens for one query ("Galápagos" inducing diffuse retrieval across a long passage) and only 2 tokens for another (a needle-in-a-haystack query). A static budget either wastes computation on concentrated queries or loses information on diffuse ones. This is the paper's central critique of the pattern-based family: the sparsity level is not a fixed attribute of the head; it changes with the query.

2. Token-Wise Sparse Attention (DSA, FASA, SnapKV, Quest)

These methods estimate token relevance on-the-fly and apply exact attention only to the retained tokens. SnapKV compresses the KV cache using relevance to recent local queries—an approach that works well when the relevant information is near the end of the context, but fails catastrophically on retrieval-heavy tasks where distant tokens carry the answer. The paper's results confirm this: SnapKV drops to 16.11% on LongBench's g-report and 11.03% on vcsum (Table 3), tasks requiring synthesis of dispersed information.

Quest uses query-aware page ranking based on min-max key statistics and operates at the block level. While it avoids the local-query bias of SnapKV, its coarse block-level sparsity (entire blocks of 64 tokens are either kept or discarded) introduces a granularity mismatch: a block may contain one relevant token and 63 irrelevant ones, but Quest must retain the entire block. This yields a general accuracy loss across LongBench (Table 3: Quest averages 50.69% vs. 53.80% for full attention).

The deeper issue with token-wise methods is that they lack a principled subspace for efficient relevance estimation. Full-dimensional key-query dot products are expensive to compute for filtering. DSA uses a lightweight learned indexer—the approach closest to RTPurbo—but then applies a static top-k selection, inheriting the same fixed-budget problem. FASA exploits RoPE's frequency structure for compression, but also uses static thresholding.

3. Block-Sparse Attention (MoBA, BLASST, SpargeAttention, Prism)

These methods select a subset of key-value blocks rather than individual tokens. MoBA treats sparse attention as block-level routing. BLASST and SpargeAttention use softmax-contribution estimates for block selection. Prism employs spectral criteria.

The granularity problem is acute here: blocks are typically 64–128 tokens. Even if only one token in a block is relevant, the entire block's computation and memory are paid. For tasks requiring precise token-level retrieval (e.g., finding a specific password in a haystack of text), this overhead is substantial. Moreover, block-sparse methods generally have no mechanism for dynamic, per-query budget adaptation—the number of selected blocks is either fixed or determined by a global threshold that doesn't account for query-dependent variation in attention concentration.

The Overlooked Alternative: Full-Attention Models Are Already Intrinsically Sparse

The paper's motivating observation—and its most significant conceptual contribution—is that these painful trade-offs between training cost, inference efficiency, and accuracy may be unnecessary. Full-attention LLMs already exhibit substantial intrinsic sparsity that can be exploited without expensive native sparse pretraining.

This insight is supported by converging evidence from prior work that the paper synthesizes:

• Head-level sparsity: Only a small subset of attention heads—the so-called retrieval heads or induction heads—actually perform long-range information retrieval. The majority of heads (often 80–85%) attend primarily to local context or to attention sinks (initial tokens that absorb excess attention mass). This was established by work on induction heads (Olsson et al., 2022) showing that certain heads implement a "retrieve previously similar tokens" mechanism, and by RazorAttention (Tang et al., 2025) and DuoAttention (Xiao et al., 2025) demonstrating that this specialization persists in long-context settings.

• Token-level sparsity within retrieval heads: Even for the heads that do perform long-range retrieval, the attention distribution is highly concentrated. For any given query, only a small subset of tokens receives substantial attention mass—often 2–10% of the context captures 90%+ of the probability mass. This is visible in the paper's Figure 3 and quantified in Table 1: top-2K tokens (5.6% of a 35K context) capture 64.2% of attention mass for the "Galápagos" query, while top-2 tokens capture 96.6% for the NIAH query.

The critical implication is that the model has already learned a sparse structure during full-attention pretraining. The sparsity is not something that must be imposed through a new training objective or architecture; it emerges naturally from the optimization process. RTPurbo's core claim is that only minimal adaptation—a few hundred training steps—is needed to surface this latent sparsity into an efficient inference mechanism.

Why Hasn't This Been Done Before? Three Unsolved Challenges

If full-attention models are already sparse, why does RTPurbo represent a novel contribution? The paper identifies three specific technical challenges that prior work failed to address simultaneously:

Challenge 1: Robust head selection. Identifying which heads genuinely require full-context access is not trivial. A head that appears "local" on one input might perform retrieval on another. The paper's solution (Section 3.1) is a calibration procedure using a synthetic "needle" insertion that directly measures retrieval capability. Crucially, the paper demonstrates that head behavior is highly stable and input-agnostic (Figure 8, Appendix A.1)—running calibration on a single long sequence is sufficient. This empirical finding is what makes offline head partitioning viable.

Challenge 2: Efficient token indexing within retrieval heads. Once a head is identified as a retrieval head, how do you efficiently select which tokens it should attend to? Computing full-dimensional attention scores for all tokens to decide which to keep would defeat the purpose—you'd pay the quadratic cost upfront. The paper's key theoretical insight (Section 2.2) is that RoPE's frequency structure makes long-range retrieval compressible: high-frequency RoPE components vary rapidly with distance and become noisy at long range, while low-frequency components change smoothly and preserve retrieval signals. This means relevance can be estimated in a much lower-dimensional space (16 dimensions), enabling cheap filtering before full-dimensional attention is applied to the selected tokens.

Challenge 3: Adaptive sparsity budgets. The amount of sparsity needed varies dramatically across queries (Figure 3) and across retrieval heads (Appendix A.2, Table 7: L43H31 retains 21 tokens while L24H25 retains 24,621 tokens at 64K, both under the same top-p threshold). A fixed top-k budget—the default in most prior work—forces a one-size-fits-all compromise that either over-retains tokens for concentrated queries/heads or under-retains for diffuse ones. RTPurbo replaces fixed top-k with dynamic top-p, where the threshold is on cumulative attention probability rather than token count, naturally adapting the budget to the query's attention concentration.

How RTPurbo Positions Itself

RTPurbo does not propose an entirely new attention mechanism. Instead, it combines and refines elements from multiple prior lines of work into a unified framework that addresses all three challenges simultaneously:

• From pattern-based sparsity: It inherits the retrieval/local head partition from RazorAttention and DuoAttention, but goes further by adding low-dimensional token indexing and dynamic thresholding within retrieval heads rather than treating them as monolithic.

• From token-wise sparsity: It inherits the idea of learned token indexing from DSA, but uses a principled low-frequency subspace induced by RoPE rather than a generic learned indexer, and replaces static top-k with dynamic top-p.

• From block-sparse attention: It inherits the hardware-awareness (custom CUDA kernels for efficient sparse execution), but operates at token granularity rather than block granularity, avoiding the precision loss from coarse blocking.

• From the training efficiency perspective: It positions itself as a post-hoc adaptation method that requires only a few hundred training steps (~1M label tokens for self-distillation), contrasting sharply with native sparse pretraining approaches (Kimi Delta Attention, DeepSeek Sparse Attention) that require training the model from scratch with sparse attention. This is a crucial practical distinction: organizations with existing full-attention models can apply RTPurbo without retraining from the ground up.

The paper's relationship to prior work is not adversarial but integrative. It synthesizes observations about head specialization (Olsson et al., 2022; Tang et al., 2025; Xiao et al., 2025), RoPE geometry (Su et al., 2024; Wang et al., 2026), and dynamic sparsity (Jiang et al., 2024; Lai et al., 2025) into a coherent design where each component addresses a specific limitation of its predecessors. The key novelty is not any single technique in isolation, but the combination of head-wise partitioning, low-dimensional RoPE-based retrieval indexing, and dynamic top-p thresholding—along with the empirical demonstration that this combination can be trained with minimal data and compute.

The Broader Significance: Challenging the Native Sparse Pretraining Narrative

The paper's motivation extends beyond the technical solution to a broader methodological claim. The prevailing narrative in efficient attention research has been that achieving strong sparse inference requires native sparse pretraining—training the model from scratch with sparse attention patterns, as done by Kimi Delta Attention (Team, 2025), DeepSeek Sparse Attention (DeepSeek-AI, 2025), and others. This is expensive and locks organizations into specific attention architectures from the start of training.

RTPurbo challenges this narrative by demonstrating that full-attention training remains a highly competitive and practical choice. A model trained with standard dense attention can be sparsified post-hoc with minimal additional cost (hundreds of steps, not full retraining), achieving near-lossless accuracy while delivering substantial speedups. This finding has significant practical implications: it means organizations can train models with standard, well-optimized full-attention kernels (like FlashAttention-2), benefit from the established training stability and convergence properties of dense attention, and then apply RTPurbo as a lightweight inference-time optimization.

The paper frames this as an "overlooked point" (Section 1): the field has been so focused on designing new sparse attention architectures that it has underestimated how much sparsity already exists in standard models. RTPurbo's contribution is as much about reframing the problem—from "how do we train sparse models from scratch?" to "how do we extract the sparsity already present in dense models?"—as it is about any specific technical innovation.

3. Technical Approach

3.1 Reader Orientation

RTPurbo is a post-hoc sparsification system that converts a standard full-attention LLM into a highly sparse model for efficient long-context inference. It solves the problem of quadratic attention cost by exploiting intrinsic sparsity already present in the pretrained model—identifying which attention heads genuinely need full-context access, compressing the retrieval relevance computation into a tiny 16-dimensional subspace, and dynamically adapting the token budget per query—all trained with only a few hundred steps of lightweight adaptation rather than expensive native sparse pretraining.

3.2 Big-Picture Architecture (Diagram in Words)

The RTPurbo system has five major components that work together across offline calibration, training, and inference:

1. Offline Head Calibrator: Runs once before any training. Inserts a synthetic "needle" into one long document, measures how much attention each head directs from the later needle to the earlier one, and partitions all query heads into a retrieval set (top ~15% by retrieval score, requiring full-context access) and a local set (remaining ~85%, which can safely discard remote tokens). This partition is frozen for the lifetime of the sparsified model.

2. Low-Dimensional Projection Weights: For each retrieval head, a pair of small matrices (each $128 \times 16$ or similar) that project the pre-RoPE query and key vectors into a 16-dimensional subspace where token relevance can be estimated cheaply. These weights are trained in Stage 1 while the backbone LLM is frozen, using KL-divergence between the original full-dimensional attention distribution and the projected approximation.

3. Dynamic Top-p Selector: At decode time, uses the low-dimensional projected scores to compute approximate relevance for every token in the KV cache, then accumulates attention mass from highest to lowest score until a cumulative fraction $p = 0.9$ is reached. The set of tokens reaching this threshold becomes the active set for that query. This replaces fixed top-k with a query-adaptive budget.

4. End-to-End Self-Distilled Model: After inserting the trained projections and switching to sparse attention mode, the full model weights are fine-tuned in Stage 2 to match the original dense model's next-token predictions on long-context data. Only top-10 logits are aligned via KL-divergence, keeping training lightweight (~1.2M label tokens, ~600 steps).

5. Hardware-Aware Sparse Kernels: Custom CUDA kernels that implement the top-p selection and sparse attention efficiently. A sort-free histogram-based top-p (fused into a single kernel launch) and a bandwidth-optimized sparse decode kernel (single-warp CTAs with no shared memory, 2-token unrolled loops, vectorized loads) that together deliver the reported speedups.

Information flows as follows during inference: an input prompt arrives → during prefill, local heads attend only to sink tokens + a sliding window of 8192 tokens, while retrieval heads compute full dense attention to build a complete KV cache → during decode, local heads continue with the static window + sink pattern, while retrieval heads project each new query token through the 16-dimensional indexer, compute approximate scores against all cached keys, apply dynamic top-p to select the active token set, and compute exact full-dimensional attention only on those selected tokens.

3.3 Roadmap for the Deep Dive

• First, the offline head calibration procedure (Section 3.1), because the retrieval/local partition is the foundation that every subsequent component depends on—it determines which heads get the low-dimensional indexer and which get the static local pattern.
• Second, the RoPE geometry analysis (Section 2.2, operationalized in Section 3.2), because it provides the theoretical justification for why a 16-dimensional subspace suffices for retrieval and dictates the design of the projection weights.
• Third, the low-dimensional projection mechanism and its Stage-1 training (Sections 3.2 and 3.3), because the projections are the core learnable component that enables efficient token indexing within retrieval heads.
• Fourth, the dynamic top-p selection mechanism (Section 3.2, Eq. 5), because it is the inference-time policy that uses the projections to decide which tokens to attend to, and its design is motivated by the query-dependent sparsity patterns documented in Section 2.3.
• Fifth, the Stage-2 self-distillation procedure (Section 3.3), because it closes the accuracy gap introduced by sparsification and its design choices (top-10 logit alignment, small learning rate, short training) reflect the paper's "minimal surgery" philosophy.
• Sixth, the hardware-aware kernel design (Section 3.4), because it translates the algorithmic sparsity into actual wall-clock speedups through careful GPU engineering that addresses the specific bottlenecks of long-context sparse attention.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical analysis paper whose core idea is that full-attention LLMs can be sparsified with minimal post-hoc adaptation by exploiting three intrinsic properties: head specialization, RoPE-induced compressibility of long-range retrieval, and query-dependent attention concentration.

---

Offline Head-Wise Calibration

The retrieval/local head partition is the foundation of RTPurbo's design, determining which heads receive the full treatment (low-dimensional indexer, dynamic top-p, dense KV cache) and which are collapsed to a simple static local pattern. The calibration must be both accurate (correctly identifying heads responsible for long-range retrieval) and practical (cheap enough to run once and forget, without per-input recomputation).

Calibration procedure. The authors construct a single long document by sampling from FineWeb. They then insert an identical "needle" span at both the beginning and the end of this document. Let the token indices of the earlier needle be the set $N_{\text{pre}}$, and the token indices of the later needle be the set $N_{\text{post}}$. For each attention head $h$, they compute a retrieval score $R_h$ as:

$R_h = \frac{1}{|N_{\text{post}}|} \sum_{t \in N_{\text{post}}} \sum_{j \in N_{\text{pre}}} A_h(t, j)$

where $A_h(t, j)$ is the post-softmax normalized attention score from query token $t$ (in the later needle) to key token $j$ (in the earlier needle), $|N_{\text{post}}|$ is the number of tokens in the later needle span, and the double sum runs over all pairs where the query is in the later needle and the key is in the earlier needle.

What it computes: For each head, this score measures the average attention mass that tokens in the later needle direct backward to tokens in the earlier needle. A high score means the head consistently attends to the earlier occurrence of the same content when it appears again later—the defining behavior of a retrieval head. A low score means the head either does not attend backward (focusing on local context) or does not distinguish the repeated content from surrounding tokens. The score is normalized by $|N_{\text{post}}|$ so it represents the average per-query-token retrieval mass, making it comparable across heads regardless of needle length.

Why this form: The paired-needle design directly tests the capacity that matters for long-context inference: can this head find semantically identical content across an arbitrarily long gap? This is more targeted than measuring general long-range attention (which could be diffuse and non-specific) and avoids confounding factors like document structure or topic shifts. Using post-softmax attention ensures the score is on a fixed scale and accounts for competition from other tokens. The normalization by $|N_{\text{post}}|$ makes the score interpretable as a probability mass per token, independent of how many query tokens happen to fall in the needle.

Partitioning heads. After computing $R_h$ for every query head across all layers, the authors sort heads by this score and select the top-scoring heads to form the retrieval set $\mathcal{H}_{\text{ret}}$. The remaining heads form the local set $\mathcal{H}_{\text{loc}}$. In the Qwen3-Coder-30B-A3B model used for long-context experiments, the total number of query heads is 1536. The authors select the top 210 heads (approximately 15%) as retrieval heads, and the remaining 1326 heads (approximately 85%) as local heads.

Justification for 15%. The ablation in Appendix B.1 (Tables 8–10) shows that increasing the ratio to 30% brings "almost no accuracy improvement" while reducing overall sparsity and roughly doubling the number of trainable projection parameters in Stage 1. Reducing to 10% causes a "substantial accuracy drop" because too few heads are available for long-range retrieval. The 15% figure is therefore an empirically determined sweet spot for this specific model. For Qwen3-30B-A3B-Think, the reasoning-specialized model, the authors report that the head distribution pattern is "largely consistent" with a similar concentration of retrieval ability in later layers (Appendix A.1).

Stability of calibration. The paper makes a strong empirical claim: head behavior is "highly stable and largely input-agnostic," and therefore "running this calibration on just one single long text sequence is sufficient to robustly score and partition all query heads." This claim is supported by the heatmap in Figure 8 (Appendix A.1), which shows retrieval scores for all 1536 heads (48 layers × 32 heads per layer) and reveals that retrieval heads are not scattered randomly but cluster in the latter half of the model, a pattern consistent with the known layer-wise computation pipeline of LLMs (early layers perform local contextualization; later layers produce stable semantic representations suitable for retrieval). The stability is crucial because it means the partition is a one-time offline cost that does not need to be recomputed per input or updated during training.

Concentration of retrieval heads in later layers. The observation that retrieval heads appear "almost exclusively in the latter half of the model" (Appendix A.1) is consistent across both Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think. This has a natural interpretation: early layers are still building up token representations from local context; only once representations are sufficiently rich and stable does it make sense to perform long-range matching. This layer-wise distribution is what makes the 15% ratio feasible—if retrieval heads were uniformly distributed, many more would need to be retained to cover all layers.

---

RoPE-Induced Compressibility of Long-Range Retrieval

The theoretical basis for using a 16-dimensional indexer rather than full-dimensional scoring comes from an analysis of how Rotary Position Embedding (RoPE) interacts with long-range token matching. This analysis (Section 2.2) shows that high-frequency RoPE components degrade the signal for distant token pairs, meaning the retrieval-relevant information is concentrated in a low-dimensional subspace.

RoPE recap. For a query token at position $m$ and a key token at position $n$, with head dimension $d = 2D$ (i.e., $D$ pairs of dimensions processed together), RoPE applies a rotation matrix to each pair of dimensions:

$R_i(m) = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}$

$\mathbf{q}_m = \mathbf{R}(m)\mathbf{q}, \quad \mathbf{k}_n = \mathbf{R}(n)\mathbf{k}$

where $\mathbf{R}(m) = \text{diag}(R_1(m), \dots, R_D(m))$ is a block-diagonal matrix applying a 2D rotation to each consecutive pair of dimensions, $\mathbf{q}$ and $\mathbf{k}$ are the pre-RoPE query and key vectors, and $\theta_i$ is a frequency that decreases with the channel index $i$ (typically $\theta_i = \text{base}^{-2i/d}$ with base often equal to 10,000 or larger). Low-index dimension pairs (small $i$) have high frequencies and rotate rapidly with position; high-index pairs (large $i$) have low frequencies and rotate slowly.

Score decomposition. The pre-softmax attention score between query at $m$ and key at $n$ depends only on the relative offset $\Delta = m - n$:

$s(m, n) = \mathbf{q}_m^\top \mathbf{k}_n = \sum_{i=1}^{D} \left[ a_i(\mathbf{q}, \mathbf{k}) \cos(\theta_i \Delta) + b_i(\mathbf{q}, \mathbf{k}) \sin(\theta_i \Delta) \right]$

where $a_i$ and $b_i$ are bilinear coefficients induced by the $i$-th rotary pair—they are computed from the pre-RoPE query and key vectors for that specific pair of dimensions and are independent of position, depending only on the semantic content of the tokens. The position-dependent part of the score comes entirely from the $\cos(\theta_i \Delta)$ and $\sin(\theta_i \Delta)$ terms.

Why this form matters. This decomposition separates position effects from content effects in a precise way. Each rotary pair $(2i-1, 2i)$ contributes a term that oscillates with $\Delta$ at frequency $\theta_i$. For high-frequency pairs (small $i$, large $\theta_i$), even a small change in relative distance causes the cosine and sine terms to oscillate rapidly, meaning the contribution to $s(m, n)$ becomes highly sensitive to the exact distance. At long range (large $\Delta$), this rapid oscillation effectively acts as noise: two tokens with identical content but at slightly different distances could get very different attention scores from these components.

For low-frequency pairs (large $i$, small $\theta_i$), the cosine and sine terms vary slowly with $\Delta$, meaning the content-dependent coefficients $a_i$ and $b_i$ dominate the score. These low-frequency components thus preserve the retrieval signal across long distances without being corrupted by position-dependent oscillation.

The compressibility insight. The paper's key theoretical claim is that for retrieval heads—heads specialized at finding semantically related tokens across arbitrary distances—the attention score is governed primarily by the low-frequency RoPE components. The high-frequency components not only contribute little useful signal but can actively interfere with long-range retrieval by introducing distance-dependent noise. This means the full $d$-dimensional query-key dot product contains substantial redundancy for the specific task of estimating long-range token relevance.

Operational consequence. If retrieval quality is driven by a low-dimensional subspace of the query and key vectors (specifically, the dimensions associated with low-frequency RoPE components), then we can estimate token relevance in a much lower-dimensional space without significant accuracy loss. This is the theoretical justification for the 16-dimensional projection in RTPurbo: instead of computing $d = 128$ dimensional dot products to score every token pair, we can project to $r = 16$ dimensions, compute approximate scores cheaply, and then apply the expensive full-dimensional attention only to the top-$p$ tokens selected by the approximate scores.

Empirical validation. The paper reports that "with our trained low-dimensional projector, we achieve over 90% recall using only 16 dimensions" (Section 1). The ablation in Appendix B.2 (Table 12) shows that dimension 16 achieves the smallest recalled-token budget across all sequence lengths (32K, 64K, 128K) among tested dimensions (4, 16, 32), indicating it best captures the retrieval structure. Dimension 4 is too small—it fails to model the full attention distribution, forcing the top-p selector to retain many more tokens to reach the same attention mass, reducing effective sparsity. Dimension 32 provides no benefit over 16 and actually requires more recalled tokens, suggesting it introduces unnecessary flexibility without improving selection quality.

---

Low-Dimensional Projection Mechanism and Stage-1 Training

The low-dimensional projector is the component that makes efficient token indexing possible. It translates the theoretical compressibility of retrieval-head attention into a learned mapping that can be evaluated cheaply at inference time.

Projection architecture. For each retrieval head $h \in \mathcal{H}_{\text{ret}}$, the paper introduces two small projection matrices:

• $\mathbf{W}^Q_h \in \mathbb{R}^{r \times d_h}$: projects the pre-RoPE query vector from dimension $d_h$ (the head dimension) down to $r$ (the low dimension)
• $\mathbf{W}^K_h \in \mathbb{R}^{r \times d_h}$: projects the pre-RoPE key vector from dimension $d_h$ down to $r$

For Qwen3-Coder-30B-A3B, $d_h = 128$ and $r = 16$, so each head introduces $128 \times 16 = 2048$ parameters per projection matrix, for a total of $2 \times 2048 = 4096$ trainable parameters per retrieval head. With 210 retrieval heads (15% of 1536), the total number of trainable parameters in Stage 1 is $210 \times 4096 \approx 8.6 \times 10^5$, or approximately 840K parameters—a tiny fraction of the model's total parameters.

Crucial design choice: pre-RoPE projection. The projections are applied to the query and key vectors before RoPE injection. This is deliberate. If applied after RoPE, the projections would have to deal with position-dependent rotations that couple content and position information. By applying projections to the pre-RoPE representations, the low-dimensional space captures pure content-based similarity, while the position information is handled separately by the full-dimensional attention (which uses RoPE normally) on the selected tokens. The projected scores provide a position-independent relevance estimate.

Score computation. Given a query token at position $m$ and a key token at position $n$, the low-dimensional approximate relevance score is:

$s_h(m, n) = (\mathbf{W}^Q_h \mathbf{q}^{\text{pre}}_{m,h})^\top (\mathbf{W}^K_h \mathbf{k}^{\text{pre}}_{n,h})$

where $\mathbf{q}^{\text{pre}}_{m,h}$ is the pre-RoPE query vector for head $h$ at position $m$, and $\mathbf{k}^{\text{pre}}_{n,h}$ is the pre-RoPE key vector for head $h$ at position $n$. The output is a single scalar representing the estimated relevance of key token $n$ to query token $m$ under head $h$.

What it computes: Each projection first compresses the $d_h$-dimensional pre-RoPE vector into an $r$-dimensional embedding (via matrix multiplication), then computes the dot product between the compressed query and key embeddings. The result is a cheap-to-compute approximate attention score that captures content-based similarity while ignoring position-dependent RoPE effects. Critically, this computation costs $O(r \cdot d_h)$ for the projections plus $O(r)$ for the dot product, compared to $O(d_h)$ for the full-dimensional score—when $r = 16$ and $d_h = 128$, the projection dominates but is a fixed per-token cost, and the per-pair comparison cost is $8\times$ smaller in dimension.

Why pre-RoPE: RoPE injects position information through rotation, which couples semantic similarity (captured by the raw vectors) with relative position. At long range, the high-frequency RoPE components introduce position-dependent noise that degrades the retrieval signal (as analyzed in Section 2.2). By projecting pre-RoPE vectors, the low-dimensional scores estimate pure semantic similarity without this noise. The full-dimensional attention later reintroduces RoPE for the exact computation on selected tokens, so position information is not lost—it's just excluded from the approximate filtering stage where it would hurt accuracy.

Why a learned projection rather than fixed low-frequency selection: A natural alternative would be to simply take the dimensions corresponding to the lowest RoPE frequencies and discard the rest—a fixed, non-learned subspace. However, the paper's learned projection approach is more flexible: it can learn which linear combinations of dimensions best capture retrieval-relevant information, which may not align exactly with individual dimensions ordered by RoPE frequency. The learned projections can also adapt to head-specific retrieval patterns. The cost of learning is tiny (840K parameters, trained with ~30M tokens) and requires no special architecture modifications.

Stage-1 training objective. The projections are trained while keeping the backbone LLM completely frozen. For each retrieval head $h$ and each query position $m$, let $\mathbf{a}^{\text{full}}_h(m)$ be the original attention distribution produced by the full-dimensional attention (a probability vector over all key positions), and let $\mathbf{a}^{\text{proj}}_h(m; \mathbf{W}^Q_h, \mathbf{W}^K_h)$ be the attention distribution derived by computing softmax over the low-dimensional projected scores. The training objective minimizes the KL-divergence between these distributions:

$\mathcal{L}_{\text{proj}} = \sum_{h \in \mathcal{H}_{\text{ret}}} \text{KL}\left( \mathbf{a}^{\text{full}}_h(m) \;\|\; \mathbf{a}^{\text{proj}}_h(m; \mathbf{W}^Q_h, \mathbf{W}^K_h) \right)$

where $\text{KL}(P \| Q) = \sum_i P_i \log(P_i / Q_i)$ is the Kullback-Leibler divergence, $\mathbf{a}^{\text{full}}_h(m)$ is the target distribution (computed once from the frozen full-attention model and held fixed), and $\mathbf{a}^{\text{proj}}_h(m; \mathbf{W}^Q_h, \mathbf{W}^K_h)$ is the model's predicted distribution parameterized by the projection weights. The sum runs over all retrieval heads and all query positions in the training sequences.

What it computes: For each query position in each training sequence, the full model produces a probability distribution over all key tokens (via softmax over full-dimensional dot products). The projected model produces its own distribution (via softmax over low-dimensional dot products). KL-divergence measures how much information is lost when using the projected distribution to approximate the full distribution. Minimizing this sum forces the projected scores to produce attention patterns that closely match the original.

Why KL-divergence: The attention distribution is a proper probability distribution (non-negative, sums to 1), and KL-divergence is the natural information-theoretic measure of discrepancy between two distributions. It is asymmetric ($\text{KL}(P \| Q)$ penalizes $Q$ for placing low probability where $P$ places high probability more than the reverse), which is the right asymmetry here: we want the projected distribution to cover all the tokens the full model considers important, even if it also includes some tokens the full model ignores. Using MSE or cosine similarity on the raw scores would not account for the softmax normalization and would weight all token pairs equally, including the vast majority that receive near-zero attention.

Training data and hyperparameters. Stage 1 uses 8,000 sequences sampled from FineWeb, each with length between 32K and 80K tokens. The training hyperparameters (Table 13 in Appendix C.1) are:

• Maximum learning rate: $1 \times 10^{-3}$
• Learning rate schedule: linear warmup from 0 to peak over the first 100 steps, then cosine annealing decay
• Weight decay: 0.01
• Maximum gradient norm: 1.0 (clipped)

The loss converges well within about 600 steps (Figure 9a in Appendix C). With an average sequence length of 48K tokens and 600 training steps, the total token budget for Stage 1 is approximately $48\text{K} \times 600 \approx 30\text{M}$ tokens—roughly 30 million tokens, a tiny amount by modern pretraining standards.

Per-head convergence. Figure 9a shows the training loss for a representative retrieval head (Layer 24, Head 25), and the authors note "highly similar convergence behavior for the other retrieval heads." This uniformity is important because it means the 600-step budget is sufficient for all heads—there are no stragglers that would require extended training.

Why freeze the backbone: The backbone LLM is frozen during Stage 1 for two reasons. First, it ensures the "target" full-attention distributions are stable—if the model weights changed, the target distributions would shift, turning the optimization into a moving-target problem. Second, it keeps the training extremely lightweight: only the small projection matrices receive gradient updates, meaning memory usage is dominated by the forward pass activations, not optimizer states for the full model.

---

Dynamic Top-p Selection Mechanism

Once the low-dimensional projections are trained, they serve as an efficient scoring function for selecting which tokens each retrieval head should attend to during decoding. The selection mechanism uses dynamic top-p thresholding rather than fixed top-k, adapting the token budget to the concentration of the projected attention scores for each query.

The problem with top-k. Fixed top-k selection retains exactly $k$ tokens regardless of the query. As demonstrated in Figure 3 and Table 1, the concentration of attention mass varies dramatically: a diffuse query might spread 90% of its mass over 8,504 tokens (24% of a 35K context), while a concentrated query might place 96.6% of its mass on just 2 tokens. A fixed $k=2048$ would capture only 64.2% of mass for the diffuse query (losing substantial information) while retaining 2048 mostly-irrelevant tokens for the concentrated query (wasting computation). No single $k$ works for all queries, and the paper's results confirm that static top-k "performs poorly because it recalls too few tokens to preserve sufficient attention mass" (Section 4.1, discussion of RULER 64K results).

Top-p selection. Instead of fixing a token count, top-p fixes a cumulative probability threshold $p$. Given the low-dimensional projected scores $s_h(m, \cdot)$ for query position $m$ under head $h$, the tokens are sorted by descending score, and the smallest set of tokens whose cumulative softmax probability reaches $p$ is selected. Formally, the active set $\mathcal{S}_h(m)$ is:

$\mathcal{S}_h(m) = \text{Top-P}(s_h(m, \cdot), p)$

where $\text{Top-P}$ takes the vector of projected scores for all cached key tokens, applies softmax to convert them to a probability distribution, sorts tokens by descending probability, and accumulates from highest to lowest until the cumulative sum reaches or exceeds $p$. The default threshold is $p = 0.9$, meaning the selected tokens capture at least 90% of the projected attention mass for that query.

What it computes: For each query token during decoding, the top-p mechanism takes the approximate relevance scores from the 16-dimensional projection, converts them to a probability distribution over all cached key tokens via softmax, and selects the minimal set of tokens needed to cover fraction $p$ of that probability mass. The output $\mathcal{S}_h(m)$ is a set of token indices that varies in size depending on how concentrated or diffuse the projected attention scores are.

Why top-p: The cumulative probability threshold directly encodes what we care about—preserving a specified fraction of the attention mass—rather than an arbitrary token count. It naturally adapts: diffuse attention distributions require more tokens to reach the threshold; concentrated distributions require fewer. The threshold $p$ is a single hyperparameter that controls the sparsity-accuracy tradeoff without baking in assumptions about the distribution's shape. This makes it robust to the dramatic query-dependent variation documented in Figure 3.

How attention is computed on the active set. Once $\mathcal{S}_h(m)$ is determined, the sparse attention output for head $h$ at position $m$ is:

$\mathbf{O}_h(m) = \sum_{n \in \mathcal{S}_h(m)} \frac{\exp(\mathbf{q}_{m,h}^\top \mathbf{k}_{n,h} / \sqrt{d_h})}{\sum_{j \in \mathcal{S}_h(m)} \exp(\mathbf{q}_{m,h}^\top \mathbf{k}_{j,h} / \sqrt{d_h})} \mathbf{v}_{n,h}$

where $\mathbf{q}_{m,h}$ and $\mathbf{k}_{n,h}$ are the full-dimensional post-RoPE query and key vectors (dimension $d_h$), $\mathbf{v}_{n,h}$ is the value vector for head $h$ at position $n$, and the softmax is computed only over the tokens in $\mathcal{S}_h(m)$, not over all tokens.

What it computes: This is standard scaled dot-product attention, but evaluated only over the subset of key-value pairs selected by the low-dimensional indexer. The query and key vectors used here are the original full-dimensional, post-RoPE vectors—the same ones the original model uses—so the attention computation on the selected tokens is mathematically exact (up to the missing normalization from excluded tokens). The only approximation is the selection of $\mathcal{S}_h(m)$ itself.

Why this two-stage approach: The low-dimensional projections serve strictly as a routing mechanism, not as a replacement for the attention computation. This is a critical design choice. If the low-dimensional scores were used directly for attention weighting, any approximation error in the projections would directly degrade the attention output quality. By using the projections only to select which tokens to attend to, and then computing exact full-dimensional attention on those tokens, the system decouples the indexing accuracy from the attention accuracy. The indexing needs to be "good enough" to include the truly relevant tokens in the active set; the attention computation then determines the exact weights using the full representational capacity of the head.

Interaction with GQA and MQA. For models using Grouped Query Attention (GQA) or Multi-Query Attention (MQA)—both Qwen3 variants used in the paper employ GQA—the head partition is defined over query heads, because these are the heads whose behavior is analyzed during calibration. However, in GQA, multiple query heads share the same key-value head. The paper distinguishes two types of sparsity (Section 3.2):

• Compute sparsity: measured at the query-head level, representing the average number of attended tokens across all query heads. When a query head selects a sparse set $\mathcal{S}_h(m)$, only those tokens incur the $O(|\mathcal{S}_h(m)| \cdot d_h)$ cost for that head.

• Memory sparsity: measured at the KV-head level. For each KV head, the actual set of retained tokens is the union of the token sets selected by all query heads mapped to that KV head. If query heads A and B both map to the same KV head, and A selects tokens {1, 5, 10} while B selects {1, 3, 10}, the KV head must retain tokens {1, 3, 5, 10} to serve both. This union operation reduces memory sparsity relative to compute sparsity—the more query heads share a KV head, the more overlapping their selected sets must be for high memory sparsity.

Table 6 provides concrete numbers: at 32K context on the niah-S task, compute sparsity is 78.7% (on average, only 21.3% of tokens are attended per query head) while memory sparsity is 76.2% (at the KV-head level, 23.8% of tokens are retained). The 2.5 percentage point gap reflects the union overhead.

The 16-dimensional scoring cost. The low-dimensional scoring itself requires computing, for each query token during decoding:

1. Project the query: $\mathbf{W}^Q_h \mathbf{q}^{\text{pre}}_{m,h}$, costing $O(r \cdot d_h)$
2. For each cached key token: compute the dot product with the projected key (already computed and stored during prefill or previous decode steps), costing $O(r)$ per token
3. Total per-token scoring cost: $O(L \cdot r)$ where $L$ is the context length

At $r = 16$, this is $8\times$ cheaper than the equivalent full-dimensional scoring ($O(L \cdot d_h)$ with $d_h = 128$), and the projection cost is amortized over all key tokens. The selected-token attention cost is $O(|\mathcal{S}_h(m)| \cdot d_h)$, where $|\mathcal{S}_h(m)|$ is typically 5–25% of $L$, giving an overall decode cost per head of roughly $0.125 \cdot L \cdot d_h + 0.2 \cdot L \cdot d_h = 0.325 \cdot L \cdot d_h$ compared to $1.0 \cdot L \cdot d_h$ for full attention — approximately a $3\times$ reduction in FLOPs per retrieval head per decode step.

Why 0.9 for $p$: The paper does not provide an extensive ablation over $p$ values, but the threshold $p = 0.9$ (as listed in Table 2) represents the standard choice from the nucleus sampling literature (Holtzman et al., 2020) applied to attention selection. At $p = 0.9$, Table 6 shows that the actual preserved attention mass exceeds 0.93–0.96 on all tested tasks (niah-S: >0.95, multi-K: >0.96 at 32K), indicating that the projected scores slightly underestimate the true attention concentration—a conservative bias that is actually desirable because it means the selected set is slightly larger than strictly necessary, providing a safety margin against projection errors.

---

Local Head Attention Pattern

While retrieval heads receive the full treatment of low-dimensional indexing and dynamic selection, local heads ($\mathcal{H}_{\text{loc}}$) use a much simpler, static sparse pattern throughout both prefill and decode.

Pattern definition. Each local head attends to:

1. Sink tokens: The first 4 tokens of the sequence are always retained. This is motivated by the "attention sink" phenomenon (Xiao et al., 2024), where initial tokens absorb disproportionate attention mass and serve as a kind of "null" attention target. Retaining them prevents the softmax from being computed over an artificially truncated distribution where the missing mass would distort the remaining probabilities.

2. Sliding window: The 8192 tokens immediately preceding the current query token (causal attention, so only past tokens are considered). This is a standard local attention window that captures the recent context.

3. All tokens before the window are discarded—the local head never attends to them.

During prefill. In the prefill phase, local heads also use this pattern. During prefill, the query token is at the end of the prompt (all tokens are processed in one forward pass), so the sliding window covers the last 8192 tokens of the prompt plus any tokens before that which fall within the window. Sink tokens are the first 4 tokens of the sequence. This means local heads never build a full KV cache—only the sink tokens and the most recent 8192 tokens are stored.

During decode. As new tokens are generated autoregressively, the sliding window advances: each new query token attends to the 4 sink tokens (fixed, always at the start of the sequence) and the 8192 most recent tokens before it. Tokens that fall outside this window are never accessed.

Why this pattern: The sliding window with attention sinks is a minimal, well-established sparse pattern that captures the behavior of local heads. Prior work (Xiao et al., 2024, 2025; Tang et al., 2025) has shown that most attention heads in pretrained LLMs primarily attend to nearby tokens and to initial sink tokens, with only a small minority performing long-range retrieval. For these local heads, the window size of 8192 is generous—it far exceeds the typical local attention span—and serves as a conservative setting that "just works" without per-head tuning. The sink tokens handle the edge case where a head needs to distribute some attention mass away from local tokens (the sinks act as an escape valve).

Contribution to overall sparsity. Since 85% of heads are local, and each local head attends to at most $4 + 8192 = 8196$ tokens regardless of total context length, the prefill FLOPs for these heads scale as $O(L \cdot 8196)$ instead of $O(L^2)$. At $L = 1\text{M}$, local heads attend to ~0.8% of the context, contributing substantially to the reported 9.36× prefill speedup at that length (Figure 1).

---

Stage-2 End-to-End Self-Distillation

After the low-dimensional projections are trained and inserted, and the sparse attention patterns are activated (dynamic top-p for retrieval heads, sink+window for local heads), the model undergoes a second training stage to recover any accuracy lost due to sparsification. This stage uses self-distillation: the sparsified model learns to match the original dense model's predictions rather than training on ground-truth labels.

Why self-distillation over standard fine-tuning. The paper argues that self-distillation "bypasses the negative impact of specific dataset distributions, thereby eliminating the tedious need to ablate and tune data mixtures" (Section 3.3). Standard supervised fine-tuning on a particular dataset can inadvertently shift the model's behavior toward that dataset's distribution, causing regression on other tasks. By using the original model's predictions as targets, self-distillation preserves the model's full output distribution—it learns to match what the original model would have predicted rather than what a specific dataset says is correct. This is aligned with RTPurbo's "minimal surgery" philosophy: the goal is to make the sparse model behave like the dense model, not to teach it new capabilities or preferences.

Target construction. Before Stage-2 training begins, the authors run a forward pass of the original (dense, unmodified) model over the entire training corpus and cache its next-token prediction logits at every position. These cached logits serve as fixed distillation targets during training. Crucially, to reduce computational overhead, they retain only the top-10 logits (the 10 highest-scoring tokens) for each position, discarding the remaining vocabulary logits.

Why top-10 alignment: Full-vocabulary logit alignment would require storing and computing KL-divergence over $V \approx 150{,}000$ tokens for every training position, which is memory-intensive and computationally wasteful since the vast majority of tokens receive near-zero probability. The top-10 logits capture the dominant modes of the teacher's predictive distribution—the model's top guesses for what comes next—and aligning these is sufficient to preserve the model's behavior while drastically reducing the memory and compute cost of distillation. The authors do not report ablation over the number of aligned logits, so the choice of 10 is empirical.

Distillation objective. Let $\mathbf{z}^{\text{dense}}_{(10)}$ be the teacher's logits restricted to the top-10 token IDs it predicted (all other logits are set to $-\infty$ before softmax), and $\mathbf{z}^{\text{sparse}}_{(10)}$ be the student sparse model's logits at the same positions restricted to those same token IDs. The loss is:

$\mathcal{L}_{\text{distill}} = \text{KL}\left( \text{softmax}(\mathbf{z}^{\text{dense}}_{(10)}) \;\|\; \text{softmax}(\mathbf{z}^{\text{sparse}}_{(10)}) \right)$

where $\text{KL}(P \| Q)$ is the Kullback-Leibler divergence, $\text{softmax}(\mathbf{z})$ converts logits to a probability distribution over the selected token IDs, and the student's logits $\mathbf{z}^{\text{sparse}}_{(10)}$ are the raw model outputs at the same token positions for the same subset of vocabulary entries.

What it computes: For each position in the training sequence, the teacher model's probability distribution over its top-10 predicted tokens is compared to the student model's probability distribution over those same tokens using KL-divergence. The student is penalized if it assigns low probability to tokens the teacher considered likely, or high probability to tokens the teacher considered unlikely (within the top-10 set). Tokens outside the top-10 are ignored entirely.

Why KL-divergence again: Same motivation as Stage 1—it measures distributional discrepancy in a probabilistically principled way, heavily penalizing the student for assigning near-zero probability to a token the teacher considered important. The restriction to top-10 is a practical approximation that captures the most important aspects of the teacher's predictive distribution while making the loss cheap to compute and store.

Training data. Stage 2 uses 8,000 long reasoning examples in dialogue format from the Dolma 3 Longmimo Mix dataset. Each sequence is longer than 32K tokens, with an average length of approximately 48K tokens. Importantly, the average number of training label tokens—positions where the model's prediction is trained—is only about 300 per sequence. This is because in dialogue-format data, most tokens belong to the user input (which the model does not need to predict) and only the assistant turns contribute label tokens. The total training corpus contains approximately 180M tokens, but the actual number of label tokens involved in learning is only about $8{,}000 \times 300 \approx 2.4\text{M}$, and the paper reports "only about 1.2M" label tokens were used (this discrepancy likely reflects that not all positions are used as training targets—some positions may be masked or only a subset of the 8,000 sequences may be used, or the 300 average may include the full training set while only 600 steps are taken, covering a fraction of the epochs).

Training hyperparameters. Table 14 (Appendix C.2) lists:

• Maximum learning rate: $3 \times 10^{-6}$ (substantially lower than Stage 1)
• Learning rate schedule: linear warmup from 0 to peak over 200 steps, then constant (no decay)
• Weight decay: 0.01
• Maximum gradient norm: 1.0
• Global batch size: 8
• Micro-batch size: 1 (gradient accumulation over 8 micro-batches)

Why such a small learning rate: The learning rate of $3 \times 10^{-6}$ is approximately 300× smaller than the Stage 1 learning rate of $1 \times 10^{-3}$. This is deliberate: Stage 2 fine-tunes the entire model (not just the projection weights), and a large learning rate could cause the model to drift away from its original capabilities, undoing the benefits of pretraining. The small learning rate ensures that the model adapts to the sparse attention patterns without forgetting its general knowledge. The constant schedule (no decay after warmup) also reflects this—there is no assumption that the model needs to converge to a sharp minimum, just that it needs gentle adjustment.

Training duration. Figure 9b shows the Stage 2 training loss converging within about 600 steps. With a global batch size of 8 and sequences averaging 48K tokens, the token throughput per step is $8 \times 48\text{K} \approx 384\text{K}$ tokens. Over 600 steps, the model sees approximately 230M tokens total, though only about 1.2M are label tokens used for the distillation loss.

What happens to the projection weights in Stage 2: The low-dimensional projection weights trained in Stage 1 are attached to the model for Stage 2 but kept frozen—they are not updated during self-distillation. This is because the projections define the token selection mechanism, and changing them during self-distillation would create a moving target: the selection set would change as the projections change, and the model would be trying to adapt to a shifting sparsity pattern. By freezing the projections, the sparsity pattern is fixed, and the model learns to produce accurate outputs given that fixed sparsity.

Why two stages rather than joint training: The authors do not explicitly justify the two-stage separation, but there are clear practical reasons. Jointly training the projections and the backbone would require the projections to simultaneously learn to approximate the full attention distribution (Stage 1 objective) while the backbone adapts to produce outputs that work well under the projected sparsity (Stage 2 objective). These objectives could conflict: the best projections for approximating the original attention might not be the best for enabling accurate final predictions. By separating the stages, each optimization has a clear, stationary target: Stage 1 targets the original model's attention distributions (which are fixed), Stage 2 targets the original model's output logits (which are fixed once precomputed and cached). This decoupling makes the training more stable and interpretable.

---

Hardware-Aware Fast Top-p Decoding Kernel

The algorithmic sparsity of RTPurbo must be translated into actual wall-clock speedups, which requires custom GPU kernels that efficiently implement the top-p selection and sparse attention operations. The paper's kernel design addresses two specific engineering bottlenecks: fast top-p thresholding without expensive sorting, and memory-efficient sparse decoding over long contexts.

The engineering challenge. At decode time, for each new query token and each retrieval head, the system must: (1) score all cached key tokens using the 16-dimensional projections, (2) identify the set of tokens that cumulatively reach probability $p$, and (3) compute full-dimensional attention on the selected tokens. The first step is $O(L \cdot r)$ where $L$ can be 128K or more; the second step, if done naively with sorting, is $O(L \log L)$; the third step is $O(|\mathcal{S}| \cdot d_h)$. The sorting step is particularly problematic—at $L = 128\text{K}$, sorting 128K scores per head per query token would dominate the computation and eliminate most of the benefit from sparse attention.

Solution: sort-free top-p via histogram. The key idea is to avoid sorting entirely by leveraging a coarse histogram of block-level scores. The procedure:

1. Block partitioning: The full key sequence of length $L$ is divided into $N_b$ blocks, where each block contains 64 tokens (the "kernel block" size in Table 2). For an $L = 128\text{K}$ context, this gives $N_b = 2048$ blocks.

2. Per-block low-dimensional scoring: Each CTA (Compute Thread Array, a group of GPU threads that cooperate on a computation) is assigned one block. It computes the low-dimensional projected scores for all 64 tokens in its block against the current query token (Equation 4), then reduces these 64 scores to a block-level pair $(m_b, \ell_b)$, where $m_b$ is the maximum score in the block and $\ell_b$ is the log-sum-exp of all scores in the block. The log-sum-exp serves as a proxy for the block's total attention mass.

3. Histogram construction: Instead of sorting the block pairs $(m_b, \ell_b)$ by $m_b$ (which would cost $O(N_b \log N_b)$), each CTA atomically deposits $\ell_b$ into a 256-bin histogram indexed by a quantized version of $m_b$. Specifically, the range of possible $m_b$ values is divided into 256 bins, and each block's maximum score determines which bin it falls into. The CTA uses atomicAdd to increment that bin's accumulated log-sum-exp. The histogram requires only 1 KB of memory per head (256 bins × 4 bytes per float), regardless of sequence length.

4. Threshold identification: After all blocks have contributed to the histogram, the last CTA to finish (identified by a per-head atomic counter) scans the histogram from the highest bin downward, accumulating the log-sum-exp values, until the cumulative fraction reaches the target $p = 0.9$. The bin at which this threshold is crossed determines the score threshold: all blocks whose maximum score is above the bins up to and partially including the threshold bin are selected.

5. Block mask generation: The last CTA writes a binary block-level mask—a bit per block indicating whether that block's tokens are included in the active set—which is then used to drive the sparse full-dimensional attention computation.

What this computes: The histogram-based selection produces an approximate top-p where the approximation comes from the block-level aggregation. Instead of selecting individual tokens, the method selects entire blocks based on the maximum score within each block. A block is selected if its maximum-scoring token is in the top-p tail. This means some tokens within selected blocks may have low scores (below the true per-token threshold), and some high-scoring tokens in just-barely-excluded blocks may be missed. However, because attention scores tend to be locally smooth (adjacent tokens have correlated relevance), and because the block size of 64 is small relative to the context length, this block-level approximation introduces minimal degradation.

Why a histogram rather than sorting: Sorting $N_b$ elements requires $O(N_b \log N_b)$ comparisons and either complex in-place data movement (for radix sort) or multiple passes (for merge sort). At $N_b = 2048$ (128K context), $\log_2(2048) = 11$, so sorting requires roughly $2048 \times 11 = 22{,}528$ comparison operations per head per query—not prohibitive for a single head, but multiplied by ~30 query heads and many queries per second, it becomes significant. More importantly, binary-search-based selection (finding the threshold via repeated partitioning) requires $O(N_b)$ auxiliary memory per head to store the full list of block scores, which "becomes prohibitive at long context where $N_b$ can exceed 16K." The histogram requires only a fixed 1 KB per head—constant memory regardless of $L$—making it scalable to arbitrary context lengths.

Why 256 bins: The paper does not justify this specific choice, but it reflects a standard engineering trade-off. Too few bins would make the block selection coarse (entire ranges of scores lumped together, causing over-selection), while too many bins would increase the histogram scan cost and memory. At 256 bins, each bin corresponds to roughly 1/256 of the score range, and the histogram scan is 256 sequential steps—trivial compared to the attention computation itself.

Kernel fusion: single launch. A critical engineering detail is that the scoring, histogram construction, and threshold scan are fused into a single kernel launch. Each CTA performs its per-block scoring, atomically contributes to the histogram, and then atomically decrements a counter. The CTA that decrements the counter to zero (the last CTA to finish) then proceeds to scan the histogram and write the block mask. This avoids an additional kernel launch for the selection phase—which would incur launch overhead and require storing intermediate results to global memory—and keeps all state in on-chip memory (registers and shared memory) during the histogram construction and scan.

Bandwidth-optimized sparse decoding kernel (Kernel 2 in Figure 5). Once the block mask is available, a second kernel performs the actual sparse full-dimensional attention over the selected blocks. The key optimization target here is memory bandwidth: even with sparsity, the selected KV blocks can span thousands of tokens, and loading the key and value vectors for these tokens from GPU global memory (HBM) is the bottleneck, not the FLOPs of the attention computation.

The design choices are:

1. Single-warp CTA: Each CTA consists of exactly one warp (32 threads), which is the minimum schedulable unit on NVIDIA GPUs. This CTA handles the attention computation for one query head (one query token, one head). There is no shared memory usage—all intermediate state (running softmax statistics, partial output accumulators) is kept in registers. This is crucial because shared memory is a limited resource (typically 48–164 KB per SM), and using no shared memory means the SM can host the maximum number of concurrent warps, maximizing its ability to hide memory latency through thread-level parallelism.

2. 2-token unrolled inner loop with vectorized loads: The inner loop iterates over the selected KV blocks, processing 2 tokens per iteration. For each pair of tokens, the key and value vectors are loaded from global memory using half2 vectorized instructions (loading 2 half-precision floats at once into a single 32-bit register). The key loads are issued first (to get them in-flight), then the attention score computation against the query begins, with the expectation that the value loads will complete by the time the softmax weights are ready. This overlaps computation with memory loads—the arithmetic of the score computation hides the latency of the value loads.

3. Online softmax: The softmax normalization is computed incrementally using the standard online algorithm (maintaining running max and running sum), avoiding the need to store all intermediate scores and do a two-pass normalization. This is standard practice in efficient attention implementations (FlashAttention, etc.) and is necessary because the number of selected tokens is not known in advance.

4. Cross-split reduction via atomic counter: For very long contexts, $B \times H$ (batch size times number of heads) may be too small to fill the GPU with enough CTAs to hide memory latency. In this case, the KV range for each head is partitioned into multiple "splits," each handled by a separate CTA. After each split completes its partial attention output, the last CTA to finish (identified by another atomic counter, same technique as the histogram) performs a cross-split reduction—combining the partial outputs from all splits into the final attention output—as a sequential step. This is the only sequential bottleneck in the decode pipeline, and it operates over a small number of splits, not over the full sequence.

Why these optimizations matter. Long-context attention is memory-bound because the arithmetic intensity (FLOPs per byte loaded) is low. For each selected token, the kernel loads $2 \cdot d_h$ bytes for K and V (384 bytes for $d_h = 128$ in half-precision) and performs approximately $4 \cdot d_h$ FLOPs (two dot products and a weighted sum). The ratio is roughly 10 FLOPs per byte—well below the ~100+ FLOPs/byte that modern GPUs need to be compute-bound. Therefore, the kernel's performance is determined almost entirely by how efficiently it streams data from HBM. The single-warp, no-shared-memory design maximizes the number of concurrent memory requests in flight, while the vectorized loads and load-compute overlap squeeze maximum utilization out of the available memory bandwidth.

Prefill kernel. The paper provides less detail on the prefill kernel, but it is structurally simpler. During prefill, retrieval heads perform full dense attention (building a complete KV cache), while local heads attend only to sink tokens and a sliding window of 8192. The prefill speedup (2.83× at 32K to 9.36× at 1M, Figure 1) comes from the local heads' drastic reduction in computation: at 1M context, local heads attend to ~0.8% of the sequence, and since they constitute 85% of heads, the total FLOPs for attention are dominated by the 15% retrieval heads that must do full attention. The speedup grows with context length because the local-head savings scale linearly while the full-attention cost scales quadratically—at 1M tokens, the gap is enormous.

Benchmark comparison (Figure 7). The authors benchmark their single-operator top-p decode kernel against two baselines:

• FlashAttention-2 (FA2), the standard optimized full-attention implementation
• A native PyTorch implementation that performs the 4-stage pipeline naively: 16-dimensional GEMV (scoring), sort, cumulative sum, and sparse 128-dimensional GEMV (attention)

At $H = 32$ heads, KV length 512K, the fused kernel achieves 3476 μs versus 6807 μs for FA2 (1.96× speedup) and 40.6 ms for the PyTorch naive implementation (11.7× speedup). The massive gap between the fused kernel and the naive PyTorch implementation highlights the importance of the kernel engineering: the algorithmic sparsity alone is not enough to achieve practical speedups; the selection mechanism must be implemented with careful attention to GPU architecture.

Why FA2 is the right baseline for the kernel benchmark: FlashAttention-2 is the state-of-the-art implementation of exact full attention, using tiling and recomputation to minimize HBM accesses. Comparing RTPurbo's sparse kernel against FA2 isolates the benefit of sparsity from the benefit of kernel optimization—both implementations are highly optimized, so the speedup reflects genuine algorithmic efficiency rather than engineering superiority over a weak baseline. The 1.96–1.99× speedups in Figure 7 (for KV = 128K, 256K, 512K) represent the pure benefit of attending to fewer tokens, holding kernel quality constant.

---

Summary of Design Choices and Their Justifications

• Offline calibration with a synthetic needle over per-input dynamic head selection: head behavior is stable and input-agnostic (Figure 8), making one-time calibration sufficient. Dynamic selection would require recomputing the partition per input, adding overhead without benefit.

• 15% retrieval head ratio over 10% or 30%: empirically, 10% loses accuracy on retrieval-heavy tasks (Appendix B.1), 30% adds no accuracy while reducing sparsity and doubling Stage 1 training parameters. The 15% figure is model-specific but the calibration procedure is general.

• 16-dimensional projection over 4, 32, or full-dimensional: 4 loses fitting quality, forcing over-retention of tokens to recover attention mass; 32 provides no benefit over 16 and slightly increases recalled tokens (Table 12); full-dimensional would defeat the purpose of efficient indexing.

• Pre-RoPE projection over post-RoPE: avoids entanglement of content and position information in the approximate scores, letting the projection focus on pure semantic similarity while position handling is deferred to the exact computation.

• Dynamic top-p ($p=0.9$) over fixed top-k: adapts to query-dependent attention concentration (Figure 3), preserving high attention mass with minimal tokens for concentrated queries while expanding the budget for diffuse queries. Fixed top-k forces a single budget that is mismatched to almost all queries.

• KL-divergence for both training stages over MSE or cross-entropy with one-hot targets: treats attention distributions and output distributions as proper probability distributions, penalizing distributional discrepancy in a principled way.

• Self-distillation with top-10 logit alignment over standard supervised fine-tuning: preserves the original model's full output distribution without dataset-specific bias, keeps training lightweight (only 1.2M label tokens), and aligns with the "minimal surgery" philosophy.

• Sort-free histogram top-p in the decode kernel over sorting or binary search: constant 1 KB memory overhead per head regardless of context length, single kernel launch fused with scoring, and avoids the $O(N_b \log N_b)$ or $O(N_b)$ memory costs of alternatives.

• Single-warp, no-shared-memory CTAs over conventional block-level parallelism: maximizes concurrent warps on each SM, hiding memory latency for the bandwidth-bound sparse attention computation.

• Frozen projections during Stage 2 over joint training: keeps the sparsity pattern fixed during backbone adaptation, preventing the moving-target problem where changing projections would shift the token selection set.

4. Key Insights and Innovations

Innovation 1: The "Intrinsic Sparsity" Reframing — Full-Attention Models Don't Need to Be Sparsified; They Already Are

The paper's most fundamental conceptual move is not a technical innovation but a reframing of the problem itself. Prior work on efficient long-context inference has operated under an implicit assumption: sparse attention is something that must be built into the model, either through native sparse pretraining (Kimi Delta Attention, DeepSeek Sparse Attention) or through post-hoc pattern discovery that imposes sparsity as an external constraint (MInference, FlexPrefill). The dominant narrative has been that full-attention training and sparse inference are opposing design philosophies—you choose one or the other at training time and live with the consequences.

RTPurbo challenges this dichotomy at its root by arguing that full-attention models are already intrinsically sparse. The sparsity is not something the method creates; it is something the model has already learned during standard pretraining, and the method merely surfaces it. This is a fundamentally different relationship between training and inference than what prior work assumes. Sparsity is not imposed from outside (as in MInference's offline pattern discovery) or baked in from the start (as in native sparse pretraining); it is latent in the model weights and can be extracted with minimal disturbance.

The evidence for intrinsic sparsity is distributed throughout the paper but crystallizes in two empirical findings. First, the head calibration procedure (Section 3.1) reveals that only ~15% of heads exhibit meaningful long-range retrieval behavior—and this ratio is stable across inputs and consistent across two different Qwen3 model variants (Appendix A.1, Figure 8). The model has already specialized into retrieval and local heads without any explicit architectural encouragement. Second, even within retrieval heads, the attention distribution is highly concentrated: Table 1 shows that top-2K tokens capture 64.2% of attention mass for a diffuse query, and Appendix A.2 (Table 7) shows that different retrieval heads naturally operate at very different sparsity levels—L43H31 retains only 21 tokens at 64K while L24H25 retains 24,621, a three-order-of-magnitude gap under the same top-p threshold. These are properties of the pretrained model, not artifacts of RTPurbo's intervention.

The significance of this reframing extends beyond the method itself. It implies that the field's heavy investment in native sparse pretraining architectures may be unnecessary for many use cases. Standard full-attention pretraining—with its well-optimized kernels (FlashAttention-2), established training stability, and mature tooling—can produce models that are already suitable for sparse inference, requiring only lightweight post-hoc adaptation. This is not an incremental improvement over existing sparse methods; it is a claim that the entire category of native sparse pretraining may be solving a problem that doesn't exist, at least for models with sufficient capacity to develop head specialization naturally. The paper does not state this aggressively, but the implication is clear from the contrast between RTPurbo's few-hundred-step adaptation and the full-pretraining cost of methods like DeepSeek Sparse Attention.

The reframing also explains why RTPurbo can be so lightweight: because it is not changing the model's behavior but preserving it under a more efficient execution pattern. The two-stage training pipeline (Stage 1: train projections to approximate existing attention; Stage 2: self-distill to match existing outputs) is explicitly designed to minimize deviation from the original model. This is why only 1.2M label tokens suffice for Stage 2—the model is learning to produce the same outputs it already produces, just through a sparser computational path. A method that had to teach the model sparse attention from scratch would need far more data and training.

Innovation 2: RoPE Geometry as a Principled Basis for Retrieval Subspace Compression

The paper's second distinctive contribution is a theoretical insight into Rotary Position Embedding that provides a principled, rather than heuristic, justification for low-dimensional attention approximation. Prior work on efficient attention indexing has used learned indexers (DSA), frequency-aware compression (FASA), or block-level statistics (Quest) without grounding the choice of compression dimension or subspace in the mathematical structure of the positional encoding. RTPurbo's analysis of the RoPE score decomposition (Section 2.2, Equation 2) provides such a grounding.

The insight is that RoPE's frequency structure creates a natural signal-to-noise distinction for long-range retrieval. The query-key score decomposes into a sum over RoPE frequency components, each oscillating with relative distance at a different rate. High-frequency components vary rapidly with distance—a small change in the gap between query and key tokens produces a large change in their contribution to the score. At long range, this rapid variation acts as noise: two token pairs with identical semantic content but slightly different distances can receive very different high-frequency contributions, obscuring the content-based signal that retrieval heads rely on. Low-frequency components, by contrast, vary slowly and preserve the content-dependent coefficients $a_i$ and $b_i$ across wide distance ranges.

This is not merely an empirical observation about which dimensions are "important." It is a structural consequence of RoPE's mathematical form—it follows from the rotation matrix definition (Equation 1) and the relative-position-only dependence (Equation 2) that the frequency ordering directly determines which components are robust to long-range displacement. A retrieval head that needs to match semantically similar tokens regardless of their exact distance must therefore rely primarily on the low-frequency subspace.

What makes this a genuine innovation rather than an incremental observation is that it transforms a hyperparameter choice (compression dimension) into a theoretically motivated design decision. The paper doesn't arbitrarily pick $r = 16$ and then justify it with ablations; it argues from the RoPE structure that a small subspace should suffice, and then validates empirically that 16 dimensions capture the retrieval-relevant signal (over 90% recall). The ablation in Table 12 further supports the theory: dimension 4 is too small (it underfits, forcing over-retention of tokens), while dimension 32 adds no benefit (the additional dimensions capture high-frequency components that don't improve long-range matching). This is exactly what the theory predicts—there exists a "right" dimensionality for the retrieval subspace, and it corresponds roughly to the number of low-frequency RoPE components that remain stable at the relevant distance scales.

This contrasts with prior learned-indexer approaches (DSA) that treat the compression dimension as an arbitrary engineering parameter to be tuned. RTPurbo's approach provides a principled ceiling: the maximum useful compression dimension is bounded by the number of RoPE frequency components that are stable at long range, which in turn depends on the RoPE base frequency and the target context length. This connection between positional encoding design and retrieval efficiency has implications beyond RTPurbo—it suggests that RoPE variants with different frequency spectra (e.g., extended-base RoPE for longer contexts) would have different optimal compression dimensionalities, and that the choice of RoPE configuration during pretraining directly affects how compressible the resulting model's attention will be.

The paper also makes a subtle but important decision in projecting pre-RoPE vectors rather than post-RoPE. This is not an implementation detail—it reflects the theoretical insight that position-dependent rotation couples content and distance information, and that separating them (content in the projections, position in the full attention) is optimal for retrieval. This clean separation is what allows the 16-dimensional indexer to focus purely on semantic similarity without being corrupted by distance-dependent noise.

Innovation 3: Query-Dependent Dynamic Thresholding as a First-Class Sparsity Mechanism

Prior sparse attention methods have almost universally used fixed sparsity budgets: retain exactly $k$ tokens (top-k, as in DSA and RTPurbo's own top-k ablation), retain all tokens within a fixed window (streaming heads in DuoAttention), or retain blocks based on a static pattern (Quest, MoBA). The budget is either a global constant or a per-head constant determined offline. RTPurbo's use of dynamic top-p thresholding is not the first instance of probability-based selection in language models (it inherits from nucleus sampling), but its application to attention token selection—and the empirical demonstration that it resolves a fundamental failure mode of fixed-budget methods—is novel.

The critical diagnostic contribution is Figure 3 and Table 1, which together demonstrate that the token budget needed by a retrieval head is query-dependent to an extreme degree. The same head (L24H25 in the paper's analysis) that needs only 2 tokens to capture 96.6% of attention mass on a needle-in-a-haystack query needs 8,504 tokens to capture 90% on a diffuse retrieval query ("Galápagos" in a long passage). This is not a minor variation around a stable mean; it is a qualitative difference in attention concentration that spans three orders of magnitude in required token count. A fixed top-k budget cannot serve both regimes: if $k$ is set large enough for diffuse queries (e.g., 8K), concentrated queries waste 99.97% of their computation on irrelevant tokens; if $k$ is set small for concentrated queries (e.g., 2K), diffuse queries lose 25%+ of their attention mass (Table 1).

This diagnostic is important because it explains why prior fixed-budget methods have inconsistent accuracy. The paper's evaluation results bear this out: the RTPurbo variant with static top-k ($k=4096$) achieves only 70.53% average on RULER 64K versus 85.49% for top-p (Table 4), and on reasoning tasks it drops from 86.67 to 80.00 on AIME (Table 5). These are not small regressions—they represent catastrophic failures on queries where the fixed budget was severely mismatched. The RULER 64K multi-K task is particularly telling: top-k achieves only 65.53 accuracy (vs. 98.60 for top-p), indicating that the fixed 4K budget simply cannot hold enough tokens for the multi-hop key-value retrieval queries.

The innovation is not the top-p mechanism itself (which is well-known from text generation) but its recontextualization as a sparsity mechanism. In text generation, top-p sampling serves to truncate the low-probability tail of the vocabulary, with the goal of improving output quality by avoiding low-probability tokens. In attention selection, top-p serves a different purpose: it provides a probabilistic guarantee that the selected tokens preserve a specified fraction of the attention mass, regardless of how concentrated or diffuse the distribution is. This guarantee is what fixed top-k lacks, and it is what enables RTPurbo to maintain near-lossless accuracy across dramatically different query types.

The paper further extends this insight beyond single-head, single-query analysis. Appendix A.2 (Table 7) shows that different retrieval heads have intrinsically different sparsity levels even under the same top-p threshold and the same input—L43H31 consistently retains 1–2 orders of magnitude fewer tokens than L24H25 across all context lengths. This means the sparsity budget must be not only query-dependent but also head-dependent. RTPurbo's per-head independent top-p naturally accommodates this: each head gets its own threshold applied independently, so concentrated heads stay sparse while diffuse heads expand their budgets. This is a level of adaptivity that no prior method achieves, and it is what enables the high sparsity levels reported in Figure 6 (over 97% at 512K) without sacrificing accuracy on the complex multi-hop tasks.

Innovation 4: Self-Distillation as a "Minimal Surgery" Training Paradigm for Model Adaptation

The paper's approach to training is philosophically distinctive: rather than optimizing the sparse model to perform well on a task (standard supervised fine-tuning), or to match ground-truth labels, RTPurbo's Stage 2 optimizes the sparse model to match its own dense counterpart's predictions. This is self-distillation in the literal sense—the teacher and student share the same architecture and pretraining, differing only in the attention sparsity pattern. The innovation is not in the distillation technique itself (which is standard), but in its deployment as a model-preserving adaptation strategy that treats the original model's outputs as the ground truth to be preserved, not the task labels.

This framing has important implications. Standard fine-tuning on long-context data risks distributional shift: the model adapts to the specific data mixture and may lose capabilities on tasks not represented in the fine-tuning set. The paper's claim that self-distillation "bypasses the negative impact of specific dataset distributions, thereby eliminating the tedious need to ablate and tune data mixtures" (Section 3.3) is an argument about robustness to data curation. Because the targets are the original model's own predictions—which already encode its full capability distribution—the distillation loss does not pull the model toward any particular task distribution. It only penalizes deviation from the original model. This means the choice of distillation corpus matters less for preserving capabilities than it would for standard fine-tuning; the corpus primarily needs to provide diverse long-range attention patterns for the model to practice its sparse computation, not task-specific supervision.

The practical consequence—that only ~1.2M label tokens and 600 training steps suffice to recover near-lossless accuracy—is a direct result of this philosophy. If the training objective were to learn long-context reasoning from scratch on the distillation corpus, far more data would be needed. But because the objective is to preserve existing capabilities under a new computational pattern, the model already "knows" the correct outputs; it just needs to learn to produce them through the sparser attention path. The small learning rate ($3 \times 10^{-6}$, roughly 300× smaller than Stage 1's peak) further enforces the "minimal surgery" constraint—the model's weights move only slightly from their pretrained values.

The dual-stage design (projection training followed by self-distillation) with frozen projections in Stage 2 is another aspect of this philosophy. By fixing the sparsity pattern before the model adapts to it, RTPurbo ensures that the adaptation target is stationary. The model learns to produce accurate outputs given a fixed sparsity regime, rather than jointly optimizing what to attend to and what to output—a coupled optimization that could lead to degenerate solutions (the model learning to exploit the sparsity mechanism rather than preserving its original behavior). This separation of concerns—"first decide what to attend to, then learn to think with what you attend to"—is a clean engineering principle that makes the training more stable and interpretable, and it is notably absent from prior work that either jointly trains sparse attention (native sparse pretraining) or doesn't train at all (training-free eviction methods).

The negative result with ReST$^{EM}$ in Appendix K (Figure 16)—where attempted RL-style optimization of the revision model caused performance degradation—serves as a contrastive case that highlights the value of the self-distillation approach. Aggressive optimization (ReST$^{EM}$) on on-policy data amplified spurious correlations and broke the model's behavior, while gentle self-distillation on offline teacher predictions preserved it. The paper doesn't make this connection explicitly, but the implication is clear: for post-hoc sparsification, preservation is a more robust objective than improvement.

Innovation 5: The Synthesis as an Intellectual Contribution — Unifying Head Specialization, RoPE Geometry, and Dynamic Sparsity into a Coherent Framework

The final innovation is architectural rather than conceptual: RTPurbo is the first method to simultaneously address head-level specialization, token-level indexing, and query-level budget adaptivity within a single integrated framework. None of the individual components is entirely unprecedented:

• Head partitioning into retrieval and local groups appeared in RazorAttention and DuoAttention, but those methods treated retrieval heads as monolithic—retaining full KV caches for them without any internal sparsity.
• Low-dimensional attention approximation for efficient indexing appeared in DSA and FASA, but those methods used fixed top-k budgets and didn't exploit RoPE's frequency structure as a principled design criterion.
• Dynamic thresholding (top-p) is standard in text generation, but had not been applied to attention token selection as a per-head, per-query sparsity mechanism.

What is novel is the combination and the way the components reinforce each other. The head partition tells us which heads need the expensive treatment (low-dimensional indexer + dynamic selection); the RoPE geometry tells us how to compress the indexing computation for those heads (pre-RoPE projection to a low-frequency subspace); the dynamic top-p tells us how many tokens to retain given the compressed scores (query-adaptive threshold). Each component addresses a limitation of prior work that used similar ideas in isolation:

• RazorAttention's retrieval heads waste computation because they do full attention on all tokens; RTPurbo's low-dimensional indexer and top-p selection sparsify them.
• DSA's learned indexer lacks a theoretical basis for its compression dimension, and its fixed top-k fails on query-dependent sparsity; RTPurbo's RoPE-grounded dimensionality and dynamic top-p fix both.
• Quest's block-level sparsity loses precision at the token level; RTPurbo's token-level selection (via histogram-based approximate top-p at the block level, but with per-block inclusion based on token scores) recovers that precision.

The synthesis is what enables the paper's headline result: near-lossless accuracy at high sparsity with only a few hundred training steps. No single component could achieve this alone. Head partitioning alone (as in RazorAttention) preserves accuracy but leaves substantial efficiency on the table because retrieval heads are still dense. Low-dimensional indexing alone (as in DSA) loses accuracy because of fixed budgets. Dynamic thresholding alone (applied uniformly without head partitioning) would waste computation on local heads that don't need it and would lack the efficient scoring mechanism to make selection cheap. Only the combination delivers both accuracy preservation and substantial speedups.

This synthesis is what distinguishes RTPurbo from being "just another sparse attention method" and elevates it to a framework-level contribution. It provides a template for how to approach post-hoc model sparsification: (1) identify which components of the model genuinely need full computation (head calibration), (2) find a principled low-dimensional surrogate for the expensive operation (RoPE subspace analysis), (3) use a dynamic, quality-guaranteeing selection rule (top-p), and (4) preserve the original model's behavior through lightweight self-distillation. This template is not specific to attention—it could apply to other model components (e.g., MLP sparsification, mixture-of-experts routing) where similar intrinsic sparsity might exist. The paper doesn't explore these extensions, but the framework's generality is implicit in its design.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on two categories of benchmarks. The first category is long-context benchmarks: LongBench (Bai et al., 2024), a bilingual multitask benchmark for long-context understanding, and RULER (Hsieh et al., 2024), which tests the effective context size of LLMs across synthetic retrieval tasks (needle-in-a-haystack variants, multi-hop key-value retrieval, etc.). The second category is reasoning benchmarks: AIME24 and AIME25 (competition math problems from the American Invitational Mathematics Examination), and MMLU-PRO (Wang et al., 2024), a more challenging multi-task language understanding benchmark with subcategories including Biology, Business, Chemistry, Computer Science, Math, Philosophy, and Physics. The reasoning benchmarks are used specifically to assess both long-decode performance (AIME generates traces up to 32K tokens) and general reasoning ability after sparsification.

• Base model(s). All experiments use the Qwen3 model family (Yang et al., 2025). For long-context benchmarks (LongBench, RULER), the paper uses Qwen3-Coder-30B-A3B, a 30-billion-parameter model with 3 billion activated parameters per token (Mixture-of-Experts architecture). For reasoning benchmarks (AIME, MMLU-PRO), the paper uses Qwen3-30B-A3B-Think, a reasoning-specialized variant. Both models employ Grouped Query Attention (GQA). The paper argues that 15% retrieval head ratio is appropriate for the Qwen3-30B-A3B architecture (Appendix B.1), noting that reducing the ratio to 10% causes substantial accuracy drops while increasing to 30% yields almost no improvement. The specific choice of Qwen3 models reflects their representative position in the open-weight model ecosystem and the practical relevance of sparsifying MoE architectures for long-context inference.

• Metrics. The primary metric is accuracy, defined as the fraction of test instances for which the model produces the correct answer. For LongBench, this is the standard task-specific accuracy metric reported per sub-task and averaged across all 16 sub-tasks. For RULER, accuracy is evaluated separately on each synthetic task variant (CWE, FWE, VT, HotPot, Squad, multi-Q, multi-V, multi-K, niah-S) and averaged. For AIME, accuracy is binary per problem (correct final answer or not). For MMLU-PRO, accuracy is the fraction of correctly answered multiple-choice questions, reported per subject. All evaluations use the lm-eval framework (Gao et al., 2024) for consistency.

• Baselines. The paper compares against five representative sparse-attention methods. RazorAttn (Tang et al., 2025) partitions heads into retrieval and streaming groups, using the same 15% retrieval-head ratio as RTPurbo. MInference (Jiang et al., 2024) assigns each head an offline-discovered sparse pattern (e.g., vertical-slash, block-sparse). FlexPrefill (Lai et al., 2025) makes pattern selection context-aware, with cumulative-attention threshold γ set to 0.9 to match RTPurbo's top-p threshold. Quest (Tang et al., 2024) uses query-aware page ranking with min-max key statistics, and does not apply sparse attention to the first two layers per its official implementation. SnapKV (Li et al., 2024) compresses the KV cache using relevance to recent local queries. Additionally, the paper implements a custom top-k variant of RTPurbo with k empirically set to 4096, to isolate the benefit of dynamic thresholding. Full attention serves as the upper-bound oracle, excluded from the ranking of sparse methods.

• Generation budget / compute accounting. Compute is measured in two ways. For accuracy evaluation, the metric is task accuracy under the sparse attention configuration, with no explicit generation budget constraint—the focus is on whether accuracy is preserved at a given sparsity level, not on controlling the number of tokens generated. For efficiency evaluation (Section 4.2), compute is measured in terms of (1) sparsity: the fraction of tokens not attended to, reported as both compute sparsity (average over query heads) and memory sparsity (union over KV heads); (2) active tokens: the dynamically retained token count per retrieval head; (3) attention mass: the preserved cumulative probability; and (4) speedup: wall-clock time ratio relative to FlashAttention-2, measured for a single attention layer at varying context lengths (32K to 1M for prefill; 128K to 512K for the kernel micro-benchmark). The prefill speedup comparison accounts for the fact that RTPurbo's retrieval heads still perform full dense attention during prefill.

• Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. The paper uses deterministic benchmark evaluation with fixed model weights and fixed hyperparameters (Table 2). Difficulty-based binning or stratified evaluation is not applied, since the benchmarks are evaluated holistically. The offline head calibration uses a single long document, with the claim that head behavior is "input-agnostic" (Appendix A.1, Figure 8). For training, the data is drawn from FineWeb (Stage 1) and Dolma 3 Longmimo Mix (Stage 2) and used directly without held-out validation tuning of hyperparameters beyond the ablations reported in Appendix B.

Main Quantitative Results

Long-Context Benchmarks: LongBench

Headline result. RTPurbo with dynamic top-p achieves the highest average accuracy among all sparse attention methods on LongBench, with 54.24% average over 16 sub-tasks, versus 53.80% for full attention—a difference of only +0.44 percentage points (Table 3). This places RTPurbo with top-p as the only sparse method that matches or exceeds the full-attention baseline in aggregate, confirming the paper's "near-lossless" claim. Among sparse methods, RTPurbo top-p ranks first, RTPurbo top-k ranks second (53.30%), and RazorAttn ranks third (52.98%).

Comparison to baselines. The results reveal a clear hierarchy among sparse attention approaches, with pattern-based methods that account for head specialization (RTPurbo, RazorAttn) generally outperforming block-sparse and token-wise methods:

• RazorAttn (52.98%): The closest competitor, benefiting from the same head partition but lacking low-dimensional indexing and dynamic thresholding within retrieval heads. It underperforms RTPurbo top-p on retrieval-heavy tasks: HotpotQA (50.92 vs. 54.74), Musique (29.15 vs. 35.29), and 2WikiMultihopQA (40.31 vs. 44.49). This gap isolates the benefit of RTPurbo's internal retrieval-head sparsification—RazorAttn retains full dense attention for all retrieval heads, which means it does not suffer from token selection errors, but it also gains no additional sparsity and cannot adapt to query-dependent token budgets.

• MInference (48.39%): Degrades significantly on multi-hop tasks (multi-en: 52.09 vs. RTPurbo's 53.34; multi-zh: 64.92 vs. 65.56) and collapses on TriviaQA (51.95 vs. 90.85) and TREC (82.50 vs. 84.50). The multi-hop degradation stems from MInference's reliance on recent queries for global attention estimation—when evidence is dispersed across a long context, the recent-query proxy fails to capture distant relevant tokens. The TriviaQA and TREC drops suggest that MInference's offline-discovered patterns are poorly matched to factual retrieval tasks requiring flexible long-range attention.

• FlexPrefill (49.42%): Suffers on tasks requiring synthesis from dispersed evidence. It achieves only 36.48 on 2WikiMultihopQA (vs. 44.49 for RTPurbo top-p), 93.50 on PR-zh (vs. 99.75), and 69.50 on PR-en (vs. 100). The PR-zh and PR-en drops are particularly diagnostic: these are passage retrieval tasks where the relevant information is scattered throughout the context, and FlexPrefill's reliance on adjacent blocks causes it to miss evidence chunks that fall outside the selected block neighborhood. The multi-V results on RULER (Table 4) corroborate this pattern: FlexPrefill drops to 60.30 at 32K and 55.50 at 64K, while RTPurbo top-p maintains 99.15 and 97.50 respectively.

• Quest (50.69%): Shows a general accuracy loss across most tasks, characteristic of coarse block-level sparsity. It underperforms RTPurbo top-p on HotpotQA (53.27 vs. 54.74), TriviaQA (79.88 vs. 90.85), and repo-p (24.31 vs. 34.08). The block granularity means Quest either over-selects (retaining entire blocks for a single relevant token, wasting computation) or under-selects (discarding blocks that contain relevant tokens because the block-level statistic doesn't capture token-level relevance). The repo-p result is notable—code completion requires precise token-level attention, and block-level approximation degrades code understanding.

• SnapKV (50.74%): Performs competitively on some tasks but catastrophically on others. It achieves 42.56 on 2WikiMultihopQA (close to RTPurbo's 44.49) but drops to 16.11 on g-report and 11.03 on vcsum—both tasks requiring synthesis of dispersed information. This bimodal performance is a signature of SnapKV's reliance on recent local queries: when the answer can be inferred from near-end context, SnapKV works well; when evidence is scattered across the entire context, it fails because the compressed KV cache discards essential distant information.

Static top-k vs. dynamic top-p within RTPurbo. The head-to-head comparison between RTPurbo's top-k and top-p variants isolates the contribution of dynamic thresholding. With top-k (k=4096), RTPurbo achieves 53.30% average—competitive with RazorAttn (52.98%) but 0.94 percentage points below top-p. The gap is concentrated on retrieval-intensive tasks: HotpotQA (53.10 vs. 54.74), TriviaQA (89.10 vs. 90.85), lsht (55.50 vs. 60.00), and repo-p (33.47 vs. 34.08). On tasks with less retrieval variation, the two are nearly identical (e.g., vcsum: 13.88 vs. 14.00). This pattern confirms the diagnostic from Figure 3 and Table 1: fixed top-k under-recalls on diffuse queries and wastes computation on concentrated ones, while top-p adapts the budget per query. The 0.94-point average gap understates the per-task variation—the gap ranges from 0.12 (vcsum) to 4.5 (lsht)—because averaging across tasks with different retrieval characteristics dilutes the effect.

Performance relative to full attention. RTPurbo top-p achieves accuracy within 1 percentage point of full attention on 12 of 16 LongBench sub-tasks. The four tasks with larger gaps are: 2WikiMultihopQA (+2.41 points over full attention), Musique (−3.01), qasper (+1.03), and lcc (−0.52). The fact that RTPurbo exceeds full attention on some tasks (2Wiki, HotpotQA, PR-en) suggests that the sparse attention pattern may have a regularizing effect—eliminating attention to irrelevant tokens could sharpen the model's focus on key evidence, though the paper does not investigate this hypothesis.

Long-Context Benchmarks: RULER

Headline result. On RULER, RTPurbo with dynamic top-p achieves 90.06% average accuracy at 32K context length and 85.49% at 64K, versus 89.65% and 86.23% for full attention (Table 4). At 64K, RTPurbo top-p is the only sparse method to maintain accuracy within 0.74 points of the full-attention baseline, while the next-best sparse method (RazorAttn) drops to 85.11% (1.12 points behind full attention). At 32K, RTPurbo top-p actually outperforms full attention by 0.41 points.

Comparison to baselines at 32K. Despite all methods operating at relatively modest context lengths where sparsity advantages are smaller, clear performance differences emerge:

• RazorAttn (88.69%): Strong overall but degrades on the multi-V task (97.65 vs. RTPurbo's 99.15). Multi-V requires retrieving information from multiple distinct values scattered across the context, placing high demand on retrieval head accuracy. RazorAttn's dense retrieval heads have the capacity but lack the fine-grained selection to distinguish relevant from irrelevant value occurrences.

• MInference (83.58%): Shows the first signs of multi-hop degradation: multi-K drops to 70.73 (vs. RTPurbo's 99.60) and multi-Q drops to 97.45 (vs. 99.95). The multi-K collapse is striking—losing nearly 29 points relative to RTPurbo—and indicates that MInference's recent-query-based global attention estimation fundamentally cannot track multiple key-value pairs when the keys are dispersed throughout the context.

• FlexPrefill (83.40%): The multi-V collapse is already evident at 32K: 60.30 vs. RTPurbo's 99.15. This is the same adjacent-block limitation visible in LongBench's PR tasks—when values are dispersed, block-neighborhood-based selection misses them.

• Quest (78.97%): The lowest average among all methods at 32K. It loses accuracy across nearly every task, not just multi-hop variants, reflecting the fundamental granularity limitation of block-level sparsity even at moderate context lengths. The CWE (common word extraction) task is particularly affected: 57.26 vs. RTPurbo's 85.44, suggesting that block-level selection systematically misses short, frequent-word patterns.

• SnapKV (83.43%): Similar profile to FlexPrefill, with multi-V at 71.10 and multi-Q at 94.00, reflecting its local-query compression bias.

• RTPurbo top-k (84.36%): Already shows a substantial gap versus top-p (5.7 points), driven primarily by multi-K (65.53 vs. 99.60) and niah-S (94.06 vs. 100). The fixed 4K budget is insufficient for the multi-key retrieval task, where the model must track many key-value pairs simultaneously.

Comparison to baselines at 64K. Scaling to 64K context length amplifies the differences:

• Full attention maintains 86.23%, dropping only 3.42 points from 32K. This is the inherent difficulty increase of longer contexts.

• RTPurbo top-p drops to 85.49%, losing 4.57 points—slightly more than full attention, but only 0.74 points behind at 64K. The multi-V task remains strong at 97.50 (vs. 97.60 for full attention), and niah-S is nearly perfect at 99.93. Multi-K drops to 98.60, losing about 1 point relative to the 32K result but still dramatically better than any other sparse method.

• RazorAttn drops to 85.11%, remaining competitive but showing larger gaps on multi-V (95.20 vs. 97.50) and multi-K (98.73 vs. 98.60—actually slightly better than RTPurbo top-p on this task).

• MInference collapses to 65.61%, losing 17.97 points from its already-subpar 32K performance. The multi-Q (82.25 → 55.50), multi-K (70.73 → 40.07), and HotPot (62.00 → 42.80) drops indicate that MInference's pattern-based approach degrades rapidly with context length, likely because the offline-discovered patterns become increasingly mismatched to the broader distribution of attention patterns at longer contexts.

• FlexPrefill (77.77%) and SnapKV (75.81%) both degrade significantly, with multi-V at 55.50 and 60.50 respectively—losing over 40 points relative to RTPurbo top-p. This confirms that local-context-biased methods are fundamentally incompatible with the dispersed-evidence requirements of multi-V at scale.

• Quest (70.60%) shows broad degradation, with CWE at 36.56 (down from 57.26 at 32K) and FWE at 63.80, indicating that block-level selection loses proportionally more precision as the context grows.

• RTPurbo top-k drops to 70.53%, losing 13.83 points—nearly all of which can be attributed to the fixed budget becoming catastrophically insufficient at 64K. Multi-K drops to 50.66 (vs. 98.60 for top-p), niah-S drops to 76.53 (vs. 99.93), and CWE drops to 59.92 (vs. 65.10). The 4K budget that was merely suboptimal at 32K becomes actively harmful at 64K, where the same fraction of context represents twice as many tokens.

Ultra-long context scaling (128K–512K). Figure 6 extends the evaluation to contexts far beyond the training data distribution (Stage 2 training uses sequences of 32K–80K). On the multi-hop tasks multi-K, multi-Q, and multi-V:

• RTPurbo maintains robust accuracy at 512K: 89.4% on multi-K, 82.5% on multi-Q, and 76.2% on multi-V. This is despite the attention sparsity reaching 97.1–97.4% at 512K. The gap between RTPurbo and the baselines widens dramatically: at 512K, MInference achieves only 4.2% on multi-K (vs. 89.4%), FlexPrefill achieves 4.3% on multi-V (vs. 76.2%), and Quest achieves numbers in the single digits across all tasks.

• Accuracy degradation with length is gradual for RTPurbo (e.g., multi-K: 99.0 at 128K → 97.0 at 256K → 89.4 at 512K) but catastrophic for baselines, which often drop below 15% by 360K. This suggests RTPurbo's dynamic thresholding successfully preserves the retrieval signal even as the noise floor (number of irrelevant tokens) grows by orders of magnitude.

• Sparsity increases with length, from ~97.1% at 128K to ~97.4% at 512K, showing that the top-p mechanism maintains roughly constant absolute token budgets (the number of tokens needed to reach p=0.9 grows sublinearly with context length) rather than constant fractional budgets.

Reasoning Benchmarks: AIME and MMLU-PRO

Headline result. On AIME24 and AIME25, RTPurbo with dynamic top-p achieves 86.67 on both, perfectly matching the full-attention baseline (Table 5). On MMLU-PRO subcategories, RTPurbo top-p achieves accuracy within 1 point of full attention on 6 of 7 subjects, with the largest gap on Physics (91.00 vs. 90.80, a 0.20-point advantage for RTPurbo). This is the strongest evidence for the "near-lossless" claim—reasoning tasks are the most sensitive to attention degradation because they require precise multi-step logical chains.

Why reasoning benchmarks matter. The reasoning tasks differ fundamentally from the long-context benchmarks in their compute profile. AIME prompts are extremely short (<300 tokens) but generate reasoning traces up to 32K tokens. MMLU-PRO questions average ~10K tokens of generated reasoning. The bottleneck is entirely in the decode phase—there is essentially no prefill cost. This means the decode-phase sparsity (dynamic top-p on retrieval heads, static sink+window on local heads) is the sole determinant of efficiency, and any accuracy loss from sparse decoding would be directly visible.

Comparison to baselines. The gap between RTPurbo and competing methods is larger on reasoning than on long-context tasks:

• Quest achieves only 46.67 on AIME24 and AIME25—a catastrophic 40-point drop from full attention. This indicates that block-level sparsity fundamentally breaks multi-step mathematical reasoning, likely because precise token-level attention is needed to track variable bindings, intermediate results, and algebraic manipulations across long reasoning traces.

• SnapKV achieves 43.33 on AIME24 and 46.67 on AIME25—similar collapse. SnapKV's local-query bias is particularly damaging for reasoning because the evidence needed for each reasoning step may be distributed across the entire trace, not concentrated near the end.

• RTPurbo top-k (k=4096) achieves 80.00 on AIME24 and 80.00 on AIME25—a substantial 6.67-point drop from full attention. This is the clearest demonstration that fixed-budget sparsity cannot handle the query-dependent variation in reasoning attention. On MMLU-PRO, top-k shows erratic performance: it matches full attention on Biology (88.80 vs. 89.20) and Physics (90.60 vs. 90.80), but collapses on Chemistry (51.40 vs. 88.00) and Computer Science (50.49 vs. 86.10). The catastrophic failures on specific subjects suggest that top-k under-recalls essential tokens for certain reasoning patterns while being adequate for others.

• RTPurbo top-p shows no such variance: its accuracy is consistently close to full attention across all MMLU-PRO subjects, with the largest deviation being +0.60 on Biology. This stability across subjects with different reasoning demands is strong evidence for the robustness of the dynamic thresholding approach.

MMLU-PRO subject-level analysis. The seven MMLU-PRO subjects span different reasoning types:

• Math (93.60 vs. 93.60): Perfect match. Mathematical reasoning under sparse attention is fully preserved.
• Physics (90.80 vs. 91.00): RTPurbo slightly exceeds full attention, suggesting a possible regularization benefit.
• Biology (89.20 vs. 89.80): +0.60, the largest positive deviation for RTPurbo.
• Business (87.40 vs. 87.20): −0.20, negligible.
• Chemistry (88.00 vs. 87.80): −0.20, negligible.
• Computer Science (86.10 vs. 85.12): −0.98, the largest negative deviation but still within 1 point.
• Philosophy (69.94 vs. 69.54): −0.40, negligible.

The consistent performance across subjects with markedly different reasoning demands (symbolic in Math, factual in Biology, procedural in CS) indicates that RTPurbo's sparsification does not disproportionately affect any particular reasoning modality.

Efficiency and Sparsity Analysis

Prefill speedup (Figure 1, left panel). RTPurbo achieves the following speedups over FlashAttention-2 for a single attention layer during prefill:

The speedup increases with context length because the local heads (85% of all heads) attend to a constant 8192-token window regardless of total length, while retrieval heads (15%) perform full attention. At 1M tokens, local heads process ~0.8% of the context, and their FLOPs are negligible compared to retrieval heads. The speedup is bounded above by roughly $1/0.15 \approx 6.67\times$ (the ratio if retrieval heads were purely sequential overhead), but the actual speedup exceeds this at 1M (9.36×) because the custom kernel implementation is more efficient than FlashAttention-2 even for the dense retrieval-head computation, and because the kernel launch overhead and memory access patterns favor the sparse execution.

RTPurbo also outperforms other sparse baselines (MInference, FlexPrefill) at all tested context lengths, with the gap widening at longer contexts. The paper does not provide exact numbers for the competing methods' prefill speedups, but Figure 1 (left panel) shows RTPurbo's line consistently above the other sparse methods.

Decode speedup (Figure 1, right panel). RTPurbo achieves 1.47× speedup at 32K, increasing to 2.01× at 1M, over FlashAttention-2 for decode. The decode speedup is lower than prefill because (1) decode is inherently memory-bound rather than compute-bound (each query attends to the full KV cache, but only one query is processed at a time), and (2) the dynamic top-p selection adds overhead (16-dimensional scoring of all cached keys) that partially offsets the savings from attending to fewer tokens. The speedup grows with context length because the scoring cost scales as $O(L \cdot r)$ with $r = 16$ while the full attention cost scales as $O(L \cdot d_h)$ with $d_h = 128$, so the relative benefit of sparsity increases with $L$.

Decode kernel micro-benchmark (Figure 7). Comparing RTPurbo's fused top-p decode kernel against FlashAttention-2 and a naive 4-stage PyTorch implementation at $H = 32$ heads:

The speedup versus FA2 is stable at ~1.97× across all tested lengths, indicating that the sparse attention benefit is roughly constant in proportion. The massive gap versus the naive PyTorch implementation (12.3× at 128K, 11.7× at 512K) highlights that the algorithmic sparsity alone (top-p selection) does not guarantee practical speedups—the kernel implementation (sort-free histogram, single-warp CTAs, load-compute overlap) is essential to realizing the efficiency gains.

Decode-stage sparsity profiling (Table 6). For Layer 25 of Qwen3-Coder-30B-A3B (a representative middle layer), the dynamic token budget varies dramatically with task and context length:

Several patterns are notable:

• Task-driven variation in token budget: At 32K, niah-S (a simple needle-in-a-haystack retrieval) requires only 468.8 tokens per retrieval head, while multi-K (tracking multiple dispersed key-value pairs) requires 2462.1—a 5.3× difference. This validates the central claim that token budgets must be query-dependent and that a fixed top-k would be either wasteful or insufficient.

• Sparsity increases with context length: At 64K, compute sparsity reaches 89.2% for niah-S and 88.7% for multi-K, compared to ~78% at 32K. The active token counts increase sublinearly with context length (niah-S: 468.8 → 1126.8, a 2.4× increase for 2× context; multi-K: 2462.1 → 3316.1, a 1.35× increase), meaning the fraction of tokens attended to decreases. This is exactly what dynamic top-p should produce: as the context grows, the number of truly relevant tokens stays roughly constant while the number of irrelevant tokens grows linearly.

• Attention mass preservation: Despite high sparsity (up to 89.2%), the preserved attention mass remains above 0.93, confirming that the 16-dimensional indexer successfully identifies the tokens carrying the vast majority of the probability mass.

• Memory sparsity trails compute sparsity: The 2–3 percentage point gap between compute and memory sparsity (e.g., 78.7% vs. 76.2% at 32K niah-S) reflects the union operation over KV heads in GQA—different query heads sharing the same KV head select slightly different token sets, and the union is necessarily larger than any individual set.

Ultra-long context sparsity (Figure 6). At contexts from 128K to 512K on multi-hop tasks:

Sparsity plateaus around 97.1–97.4% while accuracy degrades gradually from 99.0 at 128K to 89.4 at 512K. The plateauing sparsity suggests that the top-p mechanism reaches a natural floor: even at extreme lengths, the number of tokens carrying significant attention mass remains roughly constant, so the sparsity fraction approaches 100% asymptotically. The gradual accuracy decline at extreme lengths is likely due to the 16-dimensional indexer's recall degrading as the absolute number of tokens to search through grows by orders of magnitude, causing some relevant tokens to fall below the top-p threshold.

Ablation Studies and Robustness Checks

Retrieval head ratio: 15% vs. 30% vs. 10% (Appendix B.1, Tables 8–10). Increasing the proportion of heads treated as retrieval heads from 15% to 30% brings almost no accuracy improvement on MMLU-PRO (Biology: 86.0 → 85.8; CS: 76.8 → 76.8; Math: 88.2 → 88.2) or on RULER 64K (FWE: 81.7 → 81.4; HotPot: 62.8 → 62.6; multi-Q: 99.7 → 99.7; multi-K: 98.8 → 98.6; niah-S: 99.9 → 99.9), while roughly doubling the number of trainable Stage-1 projection parameters (420 heads vs. 210) and reducing overall sparsity. Reducing to 10% causes substantial accuracy drops: MMLU-PRO Math drops from 88.2 to 79.3, MMLU-PRO CS from 76.8 to 70.2, and RULER multi-K from 98.8 to 97.4. This indicates that 15% is near the minimum viable retrieval-head ratio for Qwen3-30B-A3B—fewer heads cause retrieval failures, while more heads provide no benefit because they don't perform meaningful long-range retrieval (consistent with the calibration showing retrieval scores concentrated in a minority of heads).

Low-dimensional projection size: 4, 16, 32 (Appendix B.2, Tables 11–12). End-to-end accuracy is surprisingly robust to projection dimension: dim=4, 16, and 32 all achieve comparable accuracy on MMLU-PRO Math (89.1, 88.2, 88.2), MMLU-PRO CS (76.9, 76.8, 76.8), RULER niah-S (100, 99.9, 99.9), and RULER HotPot (63.0, 62.8, 62.7). Dim=4 actually achieves the highest accuracy in several cases, but the paper correctly identifies this as artificial: Table 12 shows that dim=4 has substantially weaker fitting ability, forcing the top-p selector to retain many more tokens to recover the same attention mass (at 64K, dim=4 retains 45,280 tokens for L24H25 vs. 25,725 for dim=16). The higher accuracy of dim=4 is therefore a consequence of lower sparsity, not better indexing. Dim=16 achieves the best balance: it uses the smallest recalled-token budget (25,725 at 64K vs. 45,280 for dim=4 and 28,464 for dim=32), indicating it best captures the retrieval structure without overfitting. Dim=32 consistently retains more tokens than dim=16 (Table 12: 15,526 vs. 13,771 at 32K; 28,464 vs. 25,725 at 64K; 54,727 vs. 49,133 at 128K), suggesting that the additional dimensions introduce noise rather than signal—they capture high-frequency RoPE components that degrade long-range matching precision.

Static top-k vs. dynamic top-p (Tables 3, 4, 5, throughout). This is the most extensively tested ablation, appearing across all benchmark suites. Dynamic top-p consistently and substantially outperforms static top-k with $k=4096$:

• LongBench: 54.24 vs. 53.30 (+0.94 average)
• RULER 32K: 90.06 vs. 84.36 (+5.70)
• RULER 64K: 85.49 vs. 70.53 (+14.96)
• AIME24: 86.67 vs. 80.00 (+6.67)
• AIME25: 86.67 vs. 80.00 (+6.67)

The gap widens dramatically with context length—from +5.7 at 32K to +15.0 at 64K on RULER—because the fixed 4K budget becomes proportionally smaller (12.5% of 32K but only 6.25% of 64K), under-recalling on increasingly diffuse retrieval distributions. On MMLU-PRO, top-k shows extreme variance across subjects (Chemistry: 51.40; CS: 50.49; Math: 71.60) while top-p is stable (all within 1 point of full attention), indicating that fixed-budget failures are not uniform but catastrophic on specific reasoning patterns.

Head-wise sparsity diversity (Appendix A.2, Table 7). Even under the same top-p threshold ($p=0.9$) and on the same input, different retrieval heads exhibit dramatically different sparsity levels. At 64K context, L43H31 retains only 21 tokens while L24H25 retains 24,621—a factor of 1,172×. This diversity is consistent across context lengths: L43H31 retains 55/21/42 tokens at 32K/64K/128K, while L24H25 retains 13,836/24,621/39,614. This heterogeneity within the retrieval head set means that any single fixed budget (even per-head fixed budgets) would be simultaneously too large for concentrated heads and too small for diffuse ones. The dynamic per-head top-p naturally handles this diversity because the threshold is on cumulative probability, not token count.

Input-agnostic head calibration (Appendix A.1, Figure 8). The retrieval scores of individual heads are highly consistent across different input documents. Figure 8 shows the per-head retrieval score heatmap for all 1536 query heads, demonstrating that retrieval heads are not randomly distributed but cluster in the latter half of the model, and that head behavior is stable enough that calibration on a single long sequence suffices. The paper does not quantify this stability with correlation coefficients across different calibration documents, but the visual evidence and the practical success of the single-document calibration support the claim.

Negative result: ReST$^{EM}$ training degrades revision model performance (Appendix K, Figure 16). Although not directly an ablation of RTPurbo, this negative result on a related method (revision models) indirectly validates RTPurbo's self-distillation approach. Attempting to optimize a revision model with ReST$^{EM}$ (reinforcement learning on on-policy data) caused sequential revision performance to "substantially hurt," with the hypothesis that on-policy data collection exacerbates spurious correlations. This contrasts with RTPurbo's self-distillation, which uses frozen teacher predictions (off-policy, fixed targets) and succeeds with far less data. The negative result highlights the fragility of aggressive training approaches and supports the paper's "minimal surgery" design philosophy.

Critical Assessment

Claim 1: RTPurbo achieves near-lossless accuracy while delivering substantial speedups. This is the paper's central claim, and the evidence is generally strong but has important boundary conditions.

What is demonstrated: On LongBench, RTPurbo top-p achieves 54.24% average vs. 53.80% for full attention (Table 3)—actually exceeding the baseline. On RULER 64K, it achieves 85.49% vs. 86.23% (Table 4). On AIME24/25, it perfectly matches full attention at 86.67 (Table 5). On MMLU-PRO, all subjects are within 1 point of full attention. The prefill speedup reaches 9.36× at 1M context, and decode reaches 2.01× (Figure 1).

Boundary conditions that qualify the claim:

1. The "near-lossless" claim holds at 64K but degrades at extreme lengths. Figure 6 shows that by 512K, RTPurbo achieves 89.4 on multi-K (down from 99.0 at 128K), 82.5 on multi-Q (down from 99.3), and 76.2 on multi-V (down from 97.4). These are substantial degradations of 9–23 points from the 128K baseline, even though they remain far above competing methods. At 1M context, accuracy is not reported for these tasks. The claim "near-lossless" is supported up to 64K (matching the Stage 2 training data length), but at 512K the model has lost 10+ points on multi-hop tasks—a non-trivial degradation that the paper does not adequately acknowledge in its headline claims.

2. The accuracy comparison is against a single full-attention pass. RTPurbo's dynamic top-p uses a threshold of $p=0.9$, meaning it discards approximately 10% of the attention mass. The fact that accuracy sometimes increases (e.g., on 2WikiMultihopQA, HotpotQA) suggests that the discarded tokens may have included distracting or irrelevant information, providing an inadvertent attention-regularization benefit. This is serendipitous, not engineered—on a different distribution of tasks, discarding 10% of attention mass could hurt rather than help. The paper provides no analysis of which tokens are being discarded or whether the improvement is robust.

3. The speedup figures are for a single attention layer. The prefill speedup of 9.36× at 1M (Figure 1) is measured at the attention operator level. End-to-end speedup would be lower because non-attention components (MLP layers, layer norm, embedding, etc.) are not accelerated. The paper does not report end-to-end throughput or latency. At 1M context, attention dominates but does not constitute 100% of the computation, so the end-to-end speedup is necessarily less than 9.36×. The decode speedup of 2.01× at 1M is more meaningful because decode is attention-dominated, but still does not account for the full pipeline.

4. The paper evaluates primarily on English benchmarks with clear correctness signals. LongBench includes Chinese sub-tasks (multi-zh, PR-zh), but no non-English or multilingual reasoning evaluation is provided. The benchmarks all have unambiguous correct answers (multiple choice, passage retrieval, exact math answers). Open-ended generation, summarization quality, or dialogue coherence are not evaluated. The claim of "near-lossless" is restricted to tasks with verifiable correctness.

Claim 2: Only a few hundred training steps are needed. Strongly supported for the specific models and datasets tested.

Evidence: Stage 1 converges within ~600 steps on ~30M tokens (Figure 9a, Appendix C). Stage 2 converges within ~600 steps on ~1.2M label tokens (Figure 9b, Appendix C). The total training cost is minuscule by modern standards.

Caveats:

1. The difficulty estimation cost is not included. The offline head calibration requires running the full model on at least one long sequence and computing retrieval scores for all heads. This is a one-time cost but is not amortized in any reported training budget. For a model with 1536 heads evaluated on a long document, this calibration pass is non-trivial—it requires storing attention matrices for all layers and heads, which at 64K context length is a substantial memory and I/O cost.

2. The claim depends on the model already having strong head specialization. Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think both exhibit the necessary retrieval/local head structure. The paper provides no evidence that models from other families (Llama, Gemma, Mistral) exhibit similar specialization, or that RTPurbo would work with similarly few steps on them. If head specialization is weaker or differently structured, more training might be needed, or the approach might fail entirely.

3. The 600-step figure is specific to the chosen hyperparameters. The paper does not ablate over training duration to determine the minimum steps needed. It reports that loss converges within 600 steps for both stages, but does not test whether 300 or 1200 steps would yield different accuracy. The "few hundred" claim is empirically supported but the precise minimum is unknown.

Claim 3: The three design components (head partition, low-dimensional indexer, dynamic top-p) are individually important. Supported by ablation, but with varying strength.

Head partition (15% vs. 30% vs. 10%): Well-supported. The 30% ratio shows no accuracy benefit (Tables 8–10), confirming that adding more retrieval heads is unnecessary. The 10% ratio shows clear degradation, confirming that the retrieval head set is essential. However, the ablation only varies the ratio, not the partition method itself. The paper does not compare against random head selection or against selection based on other metrics (e.g., average attention entropy, gradient-based importance). The claim that the calibration-based partition is optimal is not tested; only that the chosen partition works and that changing the ratio changes performance in predictable ways.

Low-dimensional indexer (dim=4, 16, 32): Well-supported with an important nuance. The accuracy results (Table 11) are misleading if read naively—dim=4 appears best but is actually less sparse (Table 12). The paper correctly interprets this, but the ablation reveals that accuracy and sparsity are coupled in non-obvious ways. A method that achieved higher accuracy by being less sparse would be a valid alternative, even if less efficient. The paper's framing—that dim=16 is optimal because it achieves the best sparse accuracy at a given sparsity level—is reasonable, but a formal accuracy-vs-sparsity Pareto analysis across dimensions would have been more convincing.

Dynamic top-p vs. top-k: The strongest and most consequential ablation. The gap between top-p and top-k widens with context length (RULER: +5.7 at 32K, +15.0 at 64K) and is dramatic on reasoning tasks (AIME: +6.67). This is direct evidence for the central claim that sparsity budgets must be query-dependent. The ablation only tests a single k value (4096), and it's possible that a different fixed k would have performed better on average, but the core point—that fixed k cannot simultaneously serve concentrated and diffuse queries as well as top-p—is structurally true and well-demonstrated. An ablation sweeping k values (1024, 2048, 4096, 8192) would have strengthened this point further.

Claim 4: Full-attention models can be sparsified post-hoc with minimal adaptation, challenging the native sparse pretraining narrative. The most significant but also the most conditionally supported claim.

What is demonstrated: RTPurbo achieves competitive or superior accuracy to full attention on the tested benchmarks using a standard full-attention pretrained model (Qwen3), with only lightweight post-hoc training. This is a genuine existence proof that post-hoc sparsification is viable for some models and tasks.

What is not demonstrated:

1. The comparison to native sparse pretraining is implicit, not head-to-head. The paper does not compare RTPurbo against a natively sparse-pretrained model of comparable scale. The baselines (MInference, FlexPrefill, Quest, SnapKV, RazorAttn) are all post-hoc methods themselves. Methods like DeepSeek Sparse Attention or Kimi Delta Attention are mentioned in the introduction and related work but never evaluated. Without this comparison, the claim that "full-attention training remains a highly competitive choice" is supported only by RTPurbo's strong performance relative to full attention, not by direct evidence that it matches or exceeds native sparse training.

2. The model was trained with full attention by someone else. RTPurbo starts from a pretrained Qwen3 model. The claim that full-attention training is competitive implies that training from scratch with full attention and then sparsifying is preferable to training from scratch with sparse attention. But the paper doesn't run this experiment—it doesn't train a model from scratch with sparse attention and compare the total FLOPs (pretraining + RTPurbo adaptation) to the FLOPs of training a full-attention model and then sparsifying. The adaptation cost of RTPurbo (~30M + ~1.2M label tokens) is negligible relative to pretraining, so this is a minor point, but the full lifecycle comparison is missing.

3. The "minimal surgery" frame depends on the model having already developed the right structure. If a model were pretrained with a different architecture, different RoPE configuration, or on different data, the intrinsic sparsity might manifest differently. The paper's evidence comes from two variants of the same model family (Qwen3). Generalization to Llama, Gemma, Mistral, or proprietary models is assumed but untested.

Missing experiments that would strengthen the paper:

1. Evaluation on other model families. Testing RTPurbo on Llama-3, Mistral, or Gemma models would establish whether the head specialization and RoPE compressibility properties are universal or Qwen3-specific. The paper's claims about intrinsic sparsity in "full-attention LLMs" (plural, generic) would be much stronger with multi-family evidence.

2. End-to-end throughput/latency benchmarks. The paper reports single-layer attention speedups. End-to-end inference benchmarks—measuring tokens per second on complete model forward passes—would translate the algorithmic efficiency into deployment-relevant metrics. The attention speedups of 9.36× (prefill) and 2.01× (decode) at 1M context likely translate to lower end-to-end speedups because MLP layers and other components are not accelerated.

3. Ablation over top-p threshold p. The paper uses $p=0.9$ throughout without testing other values (e.g., 0.8, 0.85, 0.95). This is a single-parameter trade-off between sparsity and accuracy, and a sweep across $p$ would reveal whether 0.9 is a sweet spot or whether substantially higher sparsity (p=0.8) could be achieved with minimal accuracy loss.

4. Ablation over sliding window size. Local heads use a fixed window of 8192 tokens. A sweep over window sizes (2048, 4096, 8192, 16384) would reveal whether this is conservative and whether further prefill speedup could be achieved with smaller windows on local heads.

5. Analysis of discarded tokens. When RTPurbo discards 10% of attention mass (p=0.9), what kinds of tokens are lost? Are they noise, redundant information, or occasionally critical evidence? An analysis of failure cases—queries where top-p selection misses essential tokens—would characterize the failure mode and guide future improvements.

6. Training data ablation for Stage 2. The paper uses 8,000 long reasoning sequences from Dolma 3 Longmimo Mix. How does performance vary with corpus size (e.g., 1K, 2K, 4K, 8K, 16K sequences) and with corpus domain (web text vs. reasoning vs. code)? This would reveal whether self-distillation is robust to data quantity and domain, or whether careful data curation is needed.

7. Memory footprint analysis. RTPurbo's memory sparsity (Table 6) is slightly lower than compute sparsity due to GQA union overhead. What is the actual KV cache memory reduction at various context lengths? At 1M context with full attention, the KV cache per layer is substantial; RTPurbo's memory savings are a key practical benefit that is not quantified beyond the sparsity percentages.

8. Comparison to full attention with best-of-N or majority voting. RTPurbo is compared to a single full-attention forward pass. If full attention were augmented with test-time compute strategies (e.g., majority voting over multiple samples), would the accuracy gap close or reverse? This would contextualize the efficiency-accuracy trade-off.

Overall strength of experimental support. The paper's experiments are thorough within their chosen scope and convincingly demonstrate that RTPurbo works well on Qwen3 models for long-context and reasoning benchmarks. The ablation studies (head ratio, projection dimension, top-p vs. top-k) are well-designed and clearly support the paper's design choices. The main limitations are (1) single model family, (2) operator-level rather than end-to-end efficiency reporting, and (3) absence of direct comparison to native sparse pretraining methods. These limitations are standard for a systems paper introducing a new method and do not undermine the demonstrated results, but they do bound the generality of the claims. The paper would benefit from even a single experiment on a non-Qwen3 model to begin establishing cross-architecture robustness.

6. Limitations and Trade-offs

Limitation 1: The Claim of "Minimal Surgery" Depends on Stable Head Specialization That May Not Exist in All Models

The assumption or constraint. RTPurbo's entire design rests on the empirical observation that attention heads can be partitioned into retrieval and local groups through offline calibration, and that this partition is "highly stable and largely input-agnostic" (Section 3.1). The paper states this directly: "Our method relies on the empirical observation that attention heads can be partitioned into retrieval and local groups through offline calibration. While this behavior is stable in the models we study, the quality of this partition may degrade for models with weaker head specialization or under substantial domain shift" (Appendix D, Limitations). The calibration uses a single synthetic needle-in-a-haystack document to compute retrieval scores for all heads, asserting that "running this calibration on just one single long text sequence is sufficient to robustly score and partition all query heads" (Section 3.1).

The consequence. If head specialization is weaker, differently structured, or unstable across domains in other model families, the entire RTPurbo pipeline could fail. A model where retrieval behavior is distributed uniformly across many heads (rather than concentrated in 15%) would either require retaining a much larger fraction of heads as "retrieval" (destroying sparsity) or would lose critical long-range retrieval capability when local-head sparsification is applied. Similarly, if head specialization shifts significantly between pretraining corpora (e.g., code vs. natural language) or between tasks (factual retrieval vs. mathematical reasoning), the one-time offline calibration would produce a partition mismatched to some deployment scenarios, causing unpredictable accuracy degradation on those inputs.

What evidence exists in the paper. The evidence for stable specialization is entirely from two variants of the Qwen3 family (Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think). Figure 8 (Appendix A.1) shows the per-head retrieval score heatmap for Qwen3-Coder-30B-A3B, confirming retrieval heads cluster in later layers. Appendix A.1 states that Qwen3-30B-A3B-Think "exhibits a head distribution pattern largely consistent with Qwen3-Coder-30B-A3B." No evidence is provided for Llama, Gemma, Mistral, DeepSeek, or any non-Qwen3 architecture. No quantification of cross-domain stability is reported (e.g., calibration on web text vs. code vs. dialogue, with correlation coefficients between the resulting partitions). No evidence is provided for models at different scales (e.g., 7B, 70B, 405B parameters) where head specialization might differ due to capacity effects. The stability claim is supported only by visual inspection of a single heatmap and the practical success on Qwen3 benchmarks.

Mitigation status. The paper explicitly acknowledges this as a limitation (Appendix D) and suggests future work but provides no mitigation strategy. The calibration procedure itself is cheap (one forward pass on one document), so recalibration per domain or per model is feasible in principle, but there is no mechanism for detecting when the calibration has produced a poor partition—the system would simply degrade silently. A practitioner deploying RTPurbo on a new model family would need to independently validate head specialization stability before trusting the method.

---

Limitation 2: Retrieval Heads Still Perform Full Dense Attention During Prefill, Capping the Prefill Speedup

The assumption or constraint. RTPurbo applies different attention patterns in prefill and decode phases. During decode, retrieval heads use the full sparse mechanism—low-dimensional indexing followed by dynamic top-p selection and exact full-dimensional attention on selected tokens. During prefill, however, retrieval heads perform "full dense attention to build the complete KV cache" (Section 3.2), while local heads use the static sink+window pattern. The paper acknowledges this directly: "In the current design, retrieval heads still use full dense attention during prefill" (Appendix D, Limitations).

The consequence. This creates a fundamental asymmetry in the efficiency profile. The prefill speedup comes entirely from local heads (85% of heads attending to only ~0.8% of tokens at 1M context). Retrieval heads (15%) incur the full quadratic cost of building their KV caches. This places a hard ceiling on prefill speedup of approximately 1/0.15 ≈ 6.67× at very long contexts (when local-head cost becomes negligible relative to retrieval-head cost). The paper reports 9.36× at 1M, which exceeds this ceiling because the custom kernel implementation is more efficient than FlashAttention-2 for the retrieval-head computation as well—but the retrieval heads are still computing full quadratic attention, and this cost will eventually dominate as context length continues to grow. For extremely long contexts (beyond 1M), the prefill speedup will asymptotically approach a constant factor determined by the retrieval head ratio, not continue to improve.

Additionally, the full KV cache built during prefill must be stored for all retrieval heads. This means RTPurbo does not reduce the memory footprint of the KV cache for retrieval heads—only for local heads. At 1M context, the KV cache for retrieval heads (15% of heads) still stores 150K tokens worth of key-value pairs per head, which is a substantial memory overhead that limits batch size and maximum context length on memory-constrained hardware. The paper's memory sparsity figures (Table 6: 76.2–87.7% at 32K–64K) apply only to the decode phase and only reflect the union of selected tokens across query heads sharing KV heads during decoding—they do not measure KV cache storage reduction during prefill or the persistent memory cost of the full retrieval-head caches.

What evidence exists in the paper. Figure 1 (left panel) reports prefill speedups from 2.83× at 32K to 9.36× at 1M. The speedup grows sublinearly with context length—the incremental benefit from 512K to 1M (8.62× → 9.36×) is only 0.74×, compared to 1.67× from 32K to 64K (2.83× → 4.25×)—consistent with retrieval-head full attention beginning to dominate. No prefill speedup numbers are reported beyond 1M, so the asymptotic behavior is not characterized. No KV cache memory footprint measurements are provided. Appendix D acknowledges the limitation without quantifying it.

Mitigation status. The paper suggests "stronger prefill sparsification" for retrieval heads as future work (Appendix D) but implements no mechanism for it. A natural extension—applying the low-dimensional indexer and top-p selection during prefill as well, building a sparse KV cache from the start—is not explored. This would require addressing the circular dependency (the indexer needs the KV cache to score tokens, but the KV cache is being built during prefill), possibly through a two-pass approach or online index construction. The paper does not discuss these challenges or propose solutions.

---

Limitation 3: Generalization to Other Model Families, Architectures, and Domains Is Entirely Untested

The assumption or constraint. All experiments in the paper use models from the Qwen3 family (Qwen3-Coder-30B-A3B for long-context benchmarks, Qwen3-30B-A3B-Think for reasoning benchmarks). The paper's claims about intrinsic sparsity in "full-attention LLMs" (plural, generic) and the statement that "full-attention training remains a highly competitive and practical choice" (Section 5) are stated as general findings, not Qwen3-specific results. The paper acknowledges in Appendix D that "broader validation on other architectures and domains is still needed."

The consequence. Several aspects of RTPurbo could be model-specific in ways that would cause failure or substantially degraded performance on other architectures:

• Head specialization patterns: Qwen3 models use Grouped Query Attention (GQA). Models using Multi-Head Attention (MHA) or Multi-Query Attention (MQA) may exhibit different retrieval head distributions. MQA models, with only one KV head per layer, concentrate all retrieval pressure onto a single head, which might change the optimal retrieval ratio or make the retrieval/local binary partition too coarse.

• RoPE configuration: The 16-dimensional projection's effectiveness depends on the RoPE frequency spectrum (determined by the base frequency and dimension scaling). Models with different RoPE bases (e.g., 10,000 vs. 500,000 vs. 1,000,000) or different frequency allocation schemes would have different optimal compression dimensionalities. The paper's ablation shows that dim=16 is optimal for Qwen3; a different model might require dim=8 or dim=32 or might not exhibit clean frequency-based compressibility at all.

• Scale dependence: The paper evaluates on ~30B-parameter MoE models. Smaller dense models (e.g., 7B parameters) might not have developed the same degree of head specialization, since specialization often emerges with scale. Larger models (70B, 405B) might have different retrieval head ratios or different layer-wise distributions. No scaling trend is established.

• Domain transfer: The benchmarks cover English (and some Chinese) long-context QA and reasoning. Models trained primarily on code, multilingual text, or domain-specific corpora might exhibit different attention sparsity patterns. A model fine-tuned for a specific domain (medicine, law, finance) might have shifted its attention patterns relative to the pretrained base, potentially invalidating the calibration partition.

What evidence exists in the paper. Zero experiments on non-Qwen3 models. The only cross-model comparison is between Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think—two variants of the same architecture, same scale, same training paradigm. The paper notes that the two show "largely consistent" head distribution patterns (Appendix A.1), but this is weak evidence for cross-architecture generalization since they share the same base pretraining.

Mitigation status. The paper acknowledges this limitation (Appendix D) as a direction for future work. There is no analysis of which architectural properties (attention type, RoPE base, model scale, training data) are necessary or sufficient for RTPurbo to work. A practitioner with a non-Qwen3 model has no guidance on whether to expect the method to succeed, and no diagnostic for detecting failure before deployment.

---

Limitation 4: Efficiency Gains Are Reported at the Operator Level, Not End-to-End, and Latency vs. Throughput Trade-offs Are Unexplored

The assumption or constraint. All speedup numbers in the paper are measured at the granularity of a single attention layer or a single attention operator. Figure 1 reports "speedup of a single attention layer under our sparse execution scheme." Figure 7 benchmarks the "single-operator top-p decode kernel." The reported 9.36× prefill speedup and 2.01× decode speedup at 1M context are operator-level, not end-to-end model throughput or latency.

The consequence. Operator-level speedups systematically overstate end-to-end benefits. A full transformer forward pass includes non-attention components—MLP layers (typically 2× the FLOPs of attention), layer normalization, embedding lookup, and output projection—that are not accelerated by RTPurbo. At 1M context, attention dominates the prefill phase, but the exact fraction depends on model architecture (MLP expansion ratio, number of layers, hidden dimension relative to attention dimension). An operator-level speedup of 9.36× on attention might translate to, optimistically, a 4–6× end-to-end speedup at 1M context, depending on the attention-to-MLP FLOP ratio. The paper provides no end-to-end numbers, making it impossible for a practitioner to estimate actual deployment benefits.

Additionally, the dynamic top-p selection in decode introduces a per-query computational overhead that is latency-sensitive. For each new token, every retrieval head must: (1) project the query through the 16-dimensional indexer, (2) compute approximate scores against all cached keys, (3) execute the sort-free histogram top-p to select blocks, and (4) compute full-dimensional attention on selected tokens. Step (2) scales as O(L · r) and is not parallelizable across queries during autoregressive generation (only one query exists at a time). While the per-token scoring cost is 8× cheaper than full-dimensional scoring (r=16 vs. d=128), it still grows linearly with context length and is a mandatory cost for every generated token, even for queries where full attention would have been adequate. The paper's decode speedup of 2.01× at 1M is operator-level; the end-to-end decode latency improvement would be lower.

The paper also does not distinguish between latency (time to generate one token) and throughput (tokens per second across a batch). During decode, the single-warp CTA design (Section 3.4) maximizes concurrent warps on each SM to hide memory latency—this is a throughput optimization that assumes many queries (across batch or across heads) can be processed simultaneously. At batch size 1 (common for interactive applications), the GPU may be underutilized, and the speedups relative to FlashAttention-2 could be smaller or even negative if the top-p selection overhead dominates. No latency numbers at batch size 1 are reported.

What evidence exists in the paper. Figure 1 provides operator-level speedup curves for prefill and decode. Figure 7 provides a kernel micro-benchmark comparing the fused top-p decode kernel against FlashAttention-2 and a naive PyTorch implementation, at H=32 heads and KV lengths of 128K–512K, showing ~1.96× speedup over FA2. These are all single-operator measurements. End-to-end throughput (tokens/second for complete model inference), end-to-end latency (milliseconds per token for batch size 1), and KV cache memory footprint are not reported. The training cost (Stage 1: ~30M tokens, Stage 2: ~1.2M label tokens) is quantified, but the inference cost in absolute terms (GPU-hours per 1M tokens generated, or similar) is not.

Mitigation status. The paper does not address end-to-end efficiency or the latency/throughput distinction. A practitioner would need to implement RTPurbo in their serving stack and benchmark it on their specific hardware and workload to determine actual deployment benefits. The operator-level speedups establish that the attention computation is genuinely faster, but the translation to end-to-end metrics requires additional engineering and measurement that is not provided.

---

Limitation 5: The Top-p Threshold and Sliding Window Size Are Fixed Global Hyperparameters with No Adaptation to Task or Context Length

The assumption or constraint. RTPurbo uses a fixed top-p threshold of p = 0.9 (Table 2) for all retrieval heads, all queries, and all context lengths. The sliding window for local heads is fixed at 8192 tokens regardless of task or model. These hyperparameters are set once and never adapted. The paper provides no ablation or sensitivity analysis over p values, and only a single window size is evaluated.

The consequence. A fixed p = 0.9 discards approximately 10% of the attention mass for every query. As Table 6 shows, the actual preserved attention mass ranges from >0.93 to >0.96—the top-p mechanism is approximate (using block-level quantized scores through the histogram) and tends to be slightly conservative, retaining slightly more than 90%. However, across all queries and all heads, a constant fraction of information is discarded. This is a fixed accuracy-efficiency trade-off that cannot adapt:

• For easy queries (e.g., simple factual retrieval where the answer is obvious from a few tokens), p = 0.9 is unnecessarily conservative—p = 0.7 or 0.8 might preserve full accuracy while enabling higher sparsity. The model pays a computation cost for retaining tokens that contribute little marginal information.

• For hard queries (e.g., multi-hop reasoning requiring synthesis from many weakly-attended tokens), p = 0.9 might be too aggressive—the discarded 10% of attention mass could contain essential evidence that, while individually low-probability, is collectively important. The paper's ultra-long context results (Figure 6) show accuracy degrading from 99.0 at 128K to 89.4 at 512K on multi-K despite sparsity remaining nearly constant (~97.1–97.4%), suggesting that at extreme lengths, the fixed p begins to lose critical information.

• Across context lengths, the optimal p might change. At short contexts (32K), attention is naturally more concentrated, and a lower p might suffice. At extreme contexts (512K+), the absolute number of discarded tokens grows even though the fraction remains constant, and the probability that essential information falls in the discarded tail increases (the "dilution" effect—as the context grows, relevant tokens become a smaller fraction, and a fixed-percentage tail discards more absolute information).

The fixed window size of 8192 for local heads is similarly rigid. A model processing primarily short-context tasks might benefit from a smaller window (higher sparsity, faster prefill), while a model deployed exclusively for extreme long-context tasks might need a larger window for local heads that have learned to attend beyond 8K tokens. The paper provides no evidence that 8192 is optimal or that performance is insensitive to this choice.

What evidence exists in the paper. Table 2 lists the configuration with p = 0.9 and window size 8192. Table 6 shows the resulting sparsity and attention mass for two tasks at two context lengths. There is no ablation over p values. There is no ablation over window sizes. The ultra-long context results (Figure 6) show accuracy degradation at 512K despite constant sparsity, but the paper does not investigate whether adjusting p or the window size could mitigate this degradation.

Mitigation status. Not addressed. The thresholds are treated as fixed design parameters. A natural extension—adaptive thresholding where p depends on estimated query difficulty, context length, or the concentration of the projected scores—is not explored. The paper's framework would naturally support such adaptivity (difficulty estimation was a key component of the compute-optimal inference methods discussed in prior sections), but it is not implemented.

---

Limitation 6: Comparison to Native Sparse Pretraining Is Implicit and Unquantified

The assumption or constraint. The paper positions itself as an alternative to native sparse pretraining, stating that "full-attention training remains a highly competitive and practical choice" and that "native sparse pretraining is not the only path to efficient long-context inference" (Section 5). However, the paper never evaluates a natively sparse-pretrained model as a baseline. The five baselines (RazorAttn, MInference, FlexPrefill, Quest, SnapKV) are all post-hoc sparsification or training-free methods applied to full-attention models. Methods like DeepSeek Sparse Attention (DSA) or Kimi Delta Attention, which are explicitly discussed in the introduction, are not evaluated.

The consequence. The claim that full-attention + RTPurbo is "competitive" with native sparse pretraining is unsubstantiated. To substantiate it, the paper would need to compare:

• Total training cost: FLOPs for full-attention pretraining + RTPurbo adaptation (~30M tokens for Stage 1, ~1.2M label tokens for Stage 2) versus FLOPs for native sparse pretraining from scratch. RTPurbo's adaptation cost is negligible relative to pretraining, so the comparison would largely depend on whether native sparse pretraining is more or less expensive per effective FLOP than full-attention pretraining—a question the paper does not address.

• Inference efficiency at equal accuracy: At a given accuracy target (e.g., within 1 point of full-attention accuracy on RULER 64K), does RTPurbo achieve higher or lower sparsity/speedup than a natively sparse model designed for that efficiency target? Native sparse models can be architected to never compute full attention, potentially achieving higher sparsity than RTPurbo's 85% local-head + dynamic-top-p retrieval-head combination.

• Accuracy at equal efficiency: At a fixed sparsity budget (e.g., 90% compute sparsity), how do the methods compare? Native sparse models might maintain higher accuracy than RTPurbo at the same sparsity because their training objective directly optimizes for sparse attention patterns, while RTPurbo must approximate the full-attention distribution—an approximation that may lose information.

What evidence exists in the paper. Zero direct comparison to native sparse pretrained models. The paper's evidence for the "highly competitive" claim is entirely relative to the full-attention baseline and to other post-hoc sparsification methods. Table 3 shows RTPurbo achieving 54.24% on LongBench vs. 53.80% full attention and 52.98% for the next-best sparse method (RazorAttn). Table 4 shows 85.49% on RULER 64K vs. 86.23% full attention and 85.11% for RazorAttn. These results demonstrate that RTPurbo is the best post-hoc method, but they do not compare against the alternative paradigm (native sparse training) that the paper's framing explicitly challenges.

Mitigation status. Not addressed. The limitation is not acknowledged in Appendix D. A direct comparison would require access to a natively sparse-pretrained model of comparable scale and training data to Qwen3-30B-A3B, which may not be publicly available. However, the paper could have (1) acknowledged this gap explicitly, (2) compared RTPurbo against a smaller-scale natively sparse model as a proof of concept, or (3) restricted its claims to "among post-hoc sparsification methods" rather than "challenging the native sparse pretraining narrative." As written, the claims about native sparse pretraining are rhetorical rather than empirically grounded.

7. Implications and Future Directions

How This Work Changes the Landscape

RTPurbo shifts the conversation around efficient long-context inference from "how do we train sparse models from scratch?" to "how do we extract the sparsity already present in dense models?" This is not a paradigm shift in the sense of overturning established theory—head specialization and attention sparsity were already known—but it is a reframing with substantial practical consequences. Prior to this work, the dominant narrative held that achieving strong sparse inference required native sparse pretraining (Kimi Delta Attention, DeepSeek Sparse Attention), an expensive commitment that locks in architectural choices at the start of training and forecloses the use of standard full-attention pretraining infrastructure. RTPurbo demonstrates that a full-attention model can be sparsified post-hoc with only ~600 training steps and ~1.2M label tokens, achieving near-lossless accuracy on long-context and reasoning benchmarks while delivering up to 9.36× prefill speedup and 2.01× decode speedup at 1M context (Figure 1).

The significance of this reframing lies not in any single technical innovation—head partitioning appeared in RazorAttention, low-dimensional indexing in DSA and FASA, top-p thresholding in text generation—but in the synthesis that makes post-hoc sparsification viable as a general strategy. RTPurbo shows that if you (1) identify which heads genuinely need full-context access (offline calibration), (2) compress the retrieval relevance computation into a principled low-dimensional subspace (RoPE frequency analysis), (3) adapt the token budget per query rather than using a fixed allocation (dynamic top-p), and (4) gently realign the model to preserve its original behavior (self-distillation on top-10 logits), the combination works with minimal data and training. No prior method put all four pieces together, and none achieved this efficiency–accuracy trade-off curve.

This reframing has several downstream effects on research priorities:

• Native sparse pretraining becomes a harder sell for organizations that already have full-attention models in production. If post-hoc sparsification can recover comparable efficiency with negligible training cost, the value proposition of retraining from scratch with a sparse architecture weakens considerably. The paper does not directly compare against native sparse models, but the implication is clear: the bar for native sparse pretraining to justify its training cost is now higher, because the alternative is not "dense and slow" but "dense, then sparsify with RTPurbo."

• Attention interpretability research gains practical relevance. RTPurbo's head calibration (Section 3.1) and RoPE geometry analysis (Section 2.2) are fundamentally interpretability techniques deployed for engineering purposes. This paper demonstrates that understanding why certain heads behave as they do and what mathematical structure governs their attention patterns directly translates to better system design. Research on induction heads (Olsson et al., 2022), head specialization (Xiao et al., 2025; Tang et al., 2025), and positional encoding geometry becomes not merely descriptive but prescriptive—it tells you which dimensions to keep, which heads to sparsify, and how to design the compression mechanism.

• The RoPE frequency spectrum becomes a design parameter for inference efficiency. The paper's analysis of RoPE-induced compressibility (Section 2.2) establishes a direct link between positional encoding configuration and the achievable sparsity–accuracy trade-off. Models with different RoPE bases, different frequency allocation schemes, or different maximum context lengths will have different optimal compression dimensionalities and different sparsity ceilings. This means RoPE configuration during pretraining is not just a matter of context length extension; it is also an inference efficiency lever that can be tuned with post-hoc sparsification in mind.

• Self-distillation as a model-preserving adaptation paradigm is validated at a new scale and for a new purpose. The paper's Stage 2 training (Section 3.3) shows that aligning the sparse model to the dense model's top-10 logits, with a tiny learning rate ($3 \times 10^{-6}$) and only ~1.2M label tokens, successfully recovers accuracy without distributional drift. This paradigm—treat the original model as the ground truth, not the training labels—is not new, but its application to attention sparsification demonstrates that it works for structural modifications to the model's computation graph, not just for distillation to smaller architectures. This opens the door to using self-distillation for other types of post-hoc model surgery (MLP sparsification, quantization, architectural modifications) where preservation of the original model's behavior is the primary objective.

Reconciling prior contradictions. The paper indirectly resolves a tension in the sparse attention literature between pattern-based methods (which work well on some tasks but fail on others) and token-wise methods (which have the opposite failure profile). MInference degrades on retrieval-heavy multi-hop tasks (Table 4: multi-K drops to 40.07 at 64K) because its offline-discovered patterns cannot adapt to query-dependent retrieval demands. SnapKV degrades on tasks requiring dispersed evidence (Table 3: g-report drops to 16.11) because its local-query bias discards essential distant tokens. RTPurbo's dynamic top-p mechanism handles both: for concentrated queries, it retains few tokens (Table 6: 468.8 for niah-S at 32K); for diffuse queries, it expands the budget (2462.1 for multi-K at 32K). The prior contradictions—"pattern-based methods are precise but brittle" vs. "token-wise methods are flexible but imprecise"—are resolved by a method that is both head-aware (retrieval vs. local partition) and token-flexible (dynamic per-query budget within retrieval heads). The reconciliation is not that one approach was wrong, but that the two dimensions of adaptivity (what to attend to vs. how much to attend) must be addressed together.

Follow-Up Research This Work Enables

Testing RTPurbo on non-Qwen3 architectures to establish the generality of head specialization and RoPE compressibility. The paper's experiments are restricted to two variants of the Qwen3 family (Qwen3-Coder-30B-A3B and Qwen3-30B-A3B-Think). The claims about intrinsic sparsity in "full-attention LLMs" (plural, generic) require validation on Llama-3, Mistral, Gemma, DeepSeek-V2, and other widely-used architectures. A strong follow-up would replicate the full RTPurbo pipeline—offline calibration, Stage 1 projection training, Stage 2 self-distillation—on at least three architecturally distinct model families spanning different attention mechanisms (MHA, GQA, MQA), different scales (7B, 30B, 70B), and different RoPE configurations. The key measurements would be: (1) the retrieval head score distribution (does the 15% concentration hold across families?), (2) the optimal projection dimension (does the RoPE frequency spectrum predict it?), (3) the accuracy–sparsity trade-off curve (do all families achieve near-lossless accuracy at comparable sparsity?). Negative results—e.g., a model family where retrieval behavior is uniformly distributed across heads, or where 16-dimensional projections fail to achieve >90% recall—would be as valuable as positive ones, because they would establish the boundary conditions for RTPurbo's applicability and identify which architectural properties are necessary for intrinsic sparsity to manifest.

Prefill sparsification for retrieval heads to remove the dense-attention bottleneck. RTPurbo's retrieval heads perform full dense attention during prefill (Section 3.2), which caps the prefill speedup and prevents KV cache memory reduction for those heads. The paper acknowledges this limitation (Appendix D) but implements no solution. A natural extension would apply the low-dimensional indexer and top-p selection during prefill as well, building a sparse KV cache from the start. The challenge is a circular dependency: the indexer needs the KV cache to score tokens, but the KV cache is being built during prefill. A two-pass approach could solve this: Pass 1 computes low-dimensional projections for all token pairs and builds a preliminary sparse attention pattern; Pass 2 computes full-dimensional attention only on the selected pairs and writes the sparse KV cache. The cost would be approximately $O(L^2 \cdot r)$ for Pass 1 (with $r = 16$, an 8× savings over full attention) plus $O(L \cdot |\mathcal{S}| \cdot d_h)$ for Pass 2. For a 1M context at 90% sparsity, this would reduce prefill FLOPs for retrieval heads by ~85–90%, potentially doubling the overall prefill speedup beyond the current 9.36×. A strong evaluation would measure end-to-end prefill latency at 128K–1M contexts, KV cache memory footprint reduction, and accuracy on RULER multi-hop tasks to verify that sparse prefill does not degrade retrieval quality.

Adaptive difficulty-aware thresholding for top-p and sliding window size. The paper uses fixed global hyperparameters—$p = 0.9$ for top-p, 8192 for the sliding window—with no adaptation to query difficulty, task type, or context length. The ultra-long context results (Figure 6) show accuracy on multi-K degrading from 99.0 at 128K to 89.4 at 512K while sparsity remains nearly constant at ~97.1–97.4%, suggesting that at extreme lengths, the fixed threshold begins to lose critical information as the absolute number of discarded tokens grows. A natural extension—building on the compute-optimal inference framework from the companion paper analyzed in prior sections—would make $p$ and the window size functions of estimated query difficulty. Easy queries (concentrated attention, high projected-score entropy) could use more aggressive sparsity ($p = 0.8$); hard queries (diffuse attention, low entropy) could use more conservative thresholds ($p = 0.95$). The difficulty estimate could come from the projected score distribution itself (entropy, concentration, or the number of tokens needed to reach $p = 0.9$), making it a zero-overhead adaptation. A strong experiment would: (1) sweep $p$ values from 0.7 to 0.98 on RULER at multiple context lengths to establish the accuracy–sparsity Pareto frontier, (2) train a lightweight difficulty predictor on the projected-score statistics, (3) demonstrate that adaptive thresholding recovers the accuracy lost at 512K in Figure 6 while maintaining comparable sparsity.

Combining RTPurbo with KV cache quantization for compound memory savings. RTPurbo reduces compute by attending to fewer tokens, but the KV cache memory footprint for retrieval heads remains full-precision (the full KV cache is built during prefill). KV cache quantization (e.g., 4-bit or 8-bit key-value storage) is orthogonal to sparsification and could compound the memory savings. A natural combination would retain full-precision KV cache only for the tokens selected by top-p during decode, while storing the remaining (unselected) retrieval-head KV cache at 4-bit precision as a fallback for queries where the top-p selection proves insufficient. This would provide a "safety net": most queries use the sparse, full-precision path; queries where the low-dimensional indexer fails (perhaps detected by low projected-score confidence) can fall back to the quantized full cache. A strong evaluation would measure the compound memory reduction (sparsification × quantization) and the accuracy impact of the fallback mechanism on RULER multi-hop and NIAH tasks at 128K–512K contexts, comparing against pure sparsification and pure quantization baselines.

Extending the intrinsic sparsity framework to other model components (MLP sparsification, MoE routing). RTPurbo's design template—(1) identify which components genuinely need full computation via offline calibration, (2) find a principled low-dimensional surrogate for the expensive operation, (3) use dynamic thresholding, (4) self-distill to preserve behavior—is not specific to attention. MLP layers in transformers exhibit activation sparsity (many neurons output near-zero values for a given input), and Mixture-of-Experts models exhibit routing sparsity (only a subset of experts is activated per token). A follow-up could investigate whether these sparsity patterns are similarly "intrinsic" and extractable with minimal post-hoc adaptation. For MLP sparsification, the analogue would be: calibrate which MLP neurons are consistently active across a range of inputs, train a lightweight gating mechanism to predict neuron activation, and apply dynamic thresholding to skip inactive neurons during inference. For MoE routing, the analogue would be: calibrate which experts are genuinely needed per token type, train a compressed router, and sparsify the expert selection. A strong evaluation would apply this template to a Qwen3 MoE model (which already has expert sparsity) and measure whether additional MLP sparsification provides compound efficiency gains beyond attention sparsification alone, on both prefill and decode.

Stress-testing RTPurbo under distribution shift to identify failure modes and robustness boundaries. The paper evaluates on in-distribution benchmarks (LongBench, RULER, AIME, MMLU-PRO) using models pretrained on web-scale corpora that likely overlap with the benchmark distributions. A critical open question is whether the offline calibration and self-distillation generalize under substantial domain shift. If a model calibrated on web text is deployed on code, multilingual text, or domain-specific corpora (medicine, law, finance), do the retrieval head assignments remain valid? Does the low-dimensional projection trained on web-text attention distributions transfer to code attention patterns, or does it require recalibration? A strong stress-test would: (1) calibrate RTPurbo on web text (FineWeb) as in the paper, (2) evaluate on out-of-domain long-context tasks (code completion on long repositories, multilingual document QA, legal contract analysis), (3) measure accuracy degradation relative to in-domain performance, and (4) test whether per-domain recalibration (running the calibration on one code document, one legal document) recovers performance. Negative results—e.g., catastrophic degradation on code due to different attention patterns—would establish that RTPurbo requires domain-aware calibration, which is a practical limitation but not a fundamental flaw. Positive results—e.g., web-text calibration transferring well to code—would significantly strengthen the claim that head specialization is a universal property of pretrained LLMs.

Practical Applications and Downstream Use Cases

Cost-efficient long-context API serving. For providers serving long-context LLM inference (document QA, code repository analysis, multi-turn dialogue with long history), RTPurbo's 9.36× prefill speedup and 2.01× decode speedup at 1M context (Figure 1) directly translate to infrastructure cost reduction. If a service processes 1M inference requests per day at an average context length of 256K tokens, the prefill speedup alone reduces GPU-hours for the attention computation by roughly 7.5×, and the decode speedup reduces autoregressive generation cost by ~1.8×. Since attention dominates the prefill phase at long contexts, the end-to-end throughput improvement is likely in the 3–5× range (depending on the MLP-to-attention FLOP ratio), meaning the same hardware can serve 3–5× more requests or the same request volume can be served with 3–5× fewer GPUs. The key deployment decision is whether to apply RTPurbo at the model level (permanently sparsifying a model checkpoint) or at the inference engine level (as a just-in-time optimization). The per-model approach (as in the paper) requires the 600-step training pipeline but yields the highest accuracy; the engine-level approach would avoid model modification but might require online calibration and could be less stable.

On-device or edge deployment of long-context models. The combination of compute sparsification (up to 97.4% at 512K, Figure 6) and potential memory savings (from local-head KV cache reduction) makes RTPurbo attractive for deploying long-context models on memory-constrained edge devices. The Qwen3-30B-A3B model used in the paper is a Mixture-of-Experts architecture with 3B activated parameters per token—already designed for efficiency. Adding RTPurbo's attention sparsification reduces the memory bandwidth pressure during decode by attending to only ~3% of cached tokens at 512K, which could enable real-time long-context inference on hardware that would otherwise be bottlenecked by KV cache size (e.g., consumer GPUs with 8–16 GB VRAM, or mobile NPUs). The practical benefit is that applications requiring long-context processing (document summarization on-device, privacy-preserving email thread analysis, local codebase understanding) become feasible without cloud offloading. The limitation, as the paper notes, is that retrieval heads still store full KV caches from prefill, so the memory reduction applies primarily to local-head KV storage and decode-time memory bandwidth, not to total KV cache size.

Self-improvement data generation pipelines with sparse models. When using LLMs to generate training data for self-improvement loops (e.g., generating reasoning traces for distillation into smaller models, or producing synthetic long-context QA pairs), the generation cost is the dominant expense. RTPurbo's decode speedup of up to 2.01× at 1M context directly reduces this cost: for a pipeline generating 100K reasoning traces averaging 16K tokens each (similar to the AIME traces reported in Table 5), the total decode time would be roughly halved. More importantly, RTPurbo preserves near-lossless accuracy on reasoning tasks (Table 5: 86.67 on AIME24/25, perfectly matching full attention), meaning the generated data quality does not degrade. This makes RTPurbo a drop-in replacement for the full-attention model in data generation workflows, with no accuracy penalty and substantial cost savings. The self-distillation paradigm also suggests a bootstrap: generate data with the sparse model, use it to train a new model, then sparsify that new model with RTPurbo, iterating.

Batch inference for evaluation and benchmarking at scale. For research labs and companies that regularly evaluate models on long-context benchmarks (LongBench, RULER, HELMET, etc.), the cost of running full-attention inference on thousands of test instances at 32K–128K context length is substantial. RTPurbo's prefill speedup (2.83–5.92× at 32K–128K, Figure 1) directly reduces evaluation cost without compromising metric validity, since the accuracy on these benchmarks is near-lossless (Table 3: 54.24% vs. 53.80% on LongBench; Table 4: 90.06% vs. 89.65% on RULER 32K). This is particularly valuable for rapid iteration during model development, where many candidate checkpoints must be evaluated. A lab evaluating 10 candidate checkpoints per week on RULER at 128K context could reduce GPU-hours by ~6×, freeing resources for additional experiments or larger-scale ablations.