Inference Optimization2025~111 minarXiv:2508.07101

Less Is More: Training-Free Sparse Attention with Global Locality for Efficient Reasoning

score 9Lijie Yang, Zhihao Zhang, Arti Jain +5↑ 14★ 29Aug 9, 2025cs.CLhttps://github.com/DerrickYLJ/LessIsMore

🎯 Pitch

Sparse attention methods typically collapse on long reasoning chains due to accumulated per-head selection errors—but LessIsMore flips this by aggregating head-level choices into a single unified ranking, letting the model attend to over half as many tokens with zero accuracy loss. It achieves a 1.13× end-to-end speed-up over prior sparse methods simply by introducing a stable recency budget and cross-head token voting, no retraining required.

1. Executive Summary

This paper proposes LessIsMore, a training-free sparse attention mechanism that leverages global attention patterns — specifically Unified Attention Head Selection (aggregating per-head top-k token choices into a single cross-head ranking) and Stable Recency Window (reserving a fixed proportion of the token budget for recently generated tokens) — to address the accumulation of selection errors that degrades existing sparse attention approaches on long-generation reasoning tasks. Evaluated on Qwen3-8B and Qwen3-4B across AIME-24/25, GPQA-Diamond, and MATH500, LessIsMore preserves lossless accuracy at substantially higher sparsity than prior methods, achieving a 1.1× average decoding speed-up over full attention and a 1.13× end-to-end speed-up over the next-best sparse baseline while attending to at least 2× fewer tokens, establishing that unified global selection and recency-aware budgeting can maintain reasoning fidelity where per-head local optimization fails.

2. Context and Motivation

The Core Problem: Sparse Attention Degrades Unacceptably on Long-Generation Reasoning

This paper addresses a specific and increasingly consequential mismatch in the ML systems landscape: existing sparse attention methods, which work well for standard long-context tasks like retrieval and summarization, suffer catastrophic accuracy degradation when applied to the decode-heavy reasoning tasks that define modern large reasoning models (LRMs). The computational profile of reasoning is inverted relative to traditional LLM workloads. Rather than processing long input documents and generating short answers, reasoning models consume relatively concise problem statements (e.g., a few paragraphs of math contest text) and then generate extensive multi-step derivations that can span tens of thousands of output tokens. The paper reports a concrete example in Section 1: using full attention in the standard HuggingFace framework, DeepSeek-R1-Distill-Llama-8B requires more than 20 minutes on a single NVIDIA RTX A5000 GPU to generate 32,768 tokens for a single AIME problem. This is not an edge case — it is the design philosophy of test-time scaled reasoning (Wei et al., 2023; DeepSeek-AI, 2025; OpenAI, 2025), where deliberate, extended chain-of-thought generation is the mechanism by which models achieve higher accuracy on challenging problems.

The gap the paper identifies is therefore not about finding yet another sparse attention method, but about understanding why existing sparse attention methods fail on reasoning specifically, and what structural properties of the reasoning attention distribution can be exploited to fix this failure without resorting to expensive retraining or high token retention rates.

Why This Problem Matters: Real-World Impact and Theoretical Significance

The practical stakes are straightforward and quantifiable. As the paper notes in Section 1, test-time scaling — generating large numbers of tokens to enhance reasoning accuracy — is the dominant paradigm across all major LRM families (DeepSeek-R1, Gemini-2.5-pro, OpenAI-o3, Qwen3, gpt-oss). Each of these models generates output sequences that are orders of magnitude longer than their input prompts. The decode phase therefore dominates the inference cost, both in FLOPs and in wall-clock time. Any technique that can reduce the per-token cost of decoding without sacrificing accuracy translates directly into lower latency, higher throughput, and reduced operational expenses for deployments of these models.

However, the problem has theoretical significance as well. Reasoning tasks impose a stricter requirement on attention selection than retrieval or summarization tasks: logical consistency must be maintained across the entire reasoning chain, not just within a local window of context. As the paper observes in Section 1, "step-by-step reasoning requires that crucial contextual information to be preserved throughout the entire process to maintain logical consistency" (Lee & Hockenmaier, 2025). A retrieval task can tolerate moderate information loss because the model only needs to locate and extract a specific piece of information. A reasoning task, by contrast, builds each subsequent step on the conclusions of all preceding steps — if an early intermediate result is lost due to sparse attention selection error, the entire reasoning chain downstream of that error may be logically unsound.

This means that reasoning tasks serve as a stress test for the fundamental tradeoff in sparse attention design: the tension between the sparsity level (how many tokens to attend to) and the attention recall (what fraction of the ground-truth attention mass is captured). On standard tasks, existing methods can achieve very high sparsity with acceptable recall degradation. On reasoning tasks, the paper demonstrates that this tradeoff collapses — methods like TidalDecode (Yang et al., 2024), which achieve over 99.9% sparsity with no accuracy loss on retrieval, must reduce sparsity below 50% to preserve accuracy on AIME-24 (Section 1). Understanding why this collapse happens and how to design around it has implications for our understanding of attention itself — specifically, whether the long-standing assumption that attention heads are highly specialized and require independent token subsets is actually true in the reasoning regime, or whether it is an artifact of the tasks on which these models have been historically evaluated.

A third consequence the paper highlights is that inaccurate sparse attention not only degrades accuracy but also lengthens the generation process itself (Section 1, corroborated by Gao et al., 2025). When the model loses access to earlier reasoning steps due to selection errors, it may engage in redundant re-derivation, generating even more tokens and further eroding the efficiency gains that sparse attention was intended to provide. This creates a vicious cycle: sparse attention causes errors, errors cause longer generations, longer generations amplify sparse attention errors. Breaking this cycle requires a selection mechanism that maintains high attention recall throughout the entire generation trajectory, not just at the beginning.

Where Existing Approaches Fall Short

The paper identifies specific failure modes in both major families of sparse attention methods — eviction-based and selection-based — and in the underlying assumptions they share. Walking through these systematically reveals why prior work is insufficient.

Eviction-Based Methods: Permanent Information Loss

Eviction-based approaches (StreamingLLM by Xiao et al., 2023; H₂O by Zhang et al., 2023; SnapKV by Li et al., 2024; Keyformer by Adnan et al., 2024) permanently discard tokens from the KV cache based on predefined criteria — typically a combination of recent tokens (a sliding window) and a small set of "attention sink" tokens that receive consistently high attention scores regardless of semantic content. Once a token is evicted, it is gone forever; no subsequent attention computation can recover it.

For reasoning tasks, this permanent information loss is particularly damaging because the set of tokens that matter for future reasoning steps is not fully knowable at the time of eviction. An intermediate algebraic simplification that seems unimportant at step 10 might become critical at step 50 when the model needs to substitute it into a later expression. Once evicted, that intermediate result is unavailable, and the model either produces a wrong answer or wastes tokens re-deriving it. Figure 1(a) in the paper quantifies this: eviction-based StreamingLLM achieves below 65% cumulative average attention recall on AIME-24, and this recall degrades further as generation length increases — the exact opposite of what is needed for test-time scaling.

A subtler issue is that eviction-based methods typically maintain a fixed-size cache, which means that as generation length grows, the fraction of the total context that is preserved shrinks. For a 32K-token generation with a 4K-token cache size, only 12.5% of the context is available at any time. For reasoning problems where the derivation grows roughly linearly with reasoning depth, this means the model progressively loses access to earlier steps, effectively operating with a form of amnesia that worsens as the problem gets harder.

Selection-Based Methods: Local Optimization, Global Degradation

Selection-based methods (TidalDecode by Yang et al., 2024; Quest by Tang et al., 2024; RetrievalAttention by Liu et al., 2024) retain the full KV cache — meaning no information is permanently lost — but dynamically choose which tokens to actually attend to during each attention computation. The selection is typically optimized per attention head: each head independently selects the top- $k$ tokens that maximize an approximation of that head's attention scores. The intuition, inherited from the multi-head attention literature, is that different heads learn different functional roles (Xiao et al., 2024) and therefore benefit from head-specific token subsets.

This approach avoids the permanent information loss of eviction, but it introduces a different failure mode: the accumulation of per-step selection errors over long generation sequences. Each decoding step, each head makes an independent top- $k$ selection based on an approximation of the attention scores. These approximations are imperfect — they use efficient heuristics (e.g., the inner product between query and key as a proxy for the full softmax) rather than exact attention computation. Over thousands of decoding steps, these small per-step approximation errors compound. The paper shows this quantitatively in Figure 1(a): selection-based TidalDecode reaches only 75% attention recall on AIME-24, and this recall degrades as decoding length increases.

A critical additional detail from Figure 1(b) is that the absolute attention recall of selection-based methods might look comparable between reasoning and simpler tasks, but the much longer generation sequences in reasoning mean that any fixed per-step error rate produces far more total accumulated error. A method that loses 0.1% recall per step will lose 5% over 50 steps (typical retrieval) but 32% over 3,200 steps (typical AIME reasoning). The paper's key insight is that this interaction between per-step error and generation length makes reasoning tasks qualitatively harder for sparse attention, not just quantitatively harder.

The Hidden Assumption: Head-Specific Specialization

Both eviction and selection methods share a deeper assumption that the paper directly challenges: that attention heads are highly specialized and require distinct token subsets for optimal performance. This assumption is encoded in the design of these methods — selection-based approaches perform per-head token selection with different subsets per head, and even eviction-based approaches often compute per-head importance scores to decide which tokens to retain.

The paper's motivating observation (Section 2.2.1, Figure 2) is that on reasoning tasks, this assumption breaks down. When visualizing the top-4K tokens across all 32 attention heads at a representative decoding step, the authors find "high overlap among selected tokens across consecutive groups of four attention heads within the same key-value group" and even "broader overlaps spanning all attention heads." Rather than each head attending to a specialized subset of tokens, there is substantial consensus in token importance rankings. This means that per-head top- $k$ selection — the dominant paradigm in existing methods — is optimizing for a local optimum: it finds the best $k$ tokens for each specific head's query pattern, but it potentially misses tokens that are globally important across many heads and would be valuable for future decoding layers where the selection decisions made now will still be in effect.

Training-Required Approaches: Costly and Not Always Effective

A more recent line of work attempts to address the reasoning-specific degradation problem by fine-tuning models specifically for sparse attention on reasoning tasks. The paper's primary comparison point here is SeerAttention-r (Gao et al., 2025), which requires post-training adaptation. While this approach mitigates some of the accuracy loss, the paper's results in Figure 6 show that SeerAttention-r still suffers notable accuracy degradation at reduced token budgets — on AIME-24 with Qwen3-8B at 2K token budget, SeerAttention-r achieves roughly 60% accuracy compared to LessIsMore's near-lossless ~73% (vs. ~74% full attention). Moreover, training-required methods incur the computational cost of the adaptation process itself and may not generalize to new models or tasks without re-training.

The paper's position is that a training-free approach is preferable not just for convenience, but because it demonstrates that the failure mode is not a fundamental limitation of sparse attention for reasoning — it is a consequence of a specific design choice (per-head local optimization) that can be corrected by exploiting the attention patterns that actually characterize reasoning.

How This Paper Positions Itself

The paper frames its contribution through a specific lens: the failure of existing sparse attention on reasoning is caused by an incorrect assumption about attention head specialization, and correcting this assumption — by switching from per-head local optimization to cross-head global optimization — is sufficient to recover lossless accuracy at high sparsity without any training or architectural modification.

This positioning has several important characteristics:

1. It is a conceptual intervention, not just an algorithmic one. The paper's two core techniques — Unified Attention Head Selection and Stable Recency Window — are implementations of a deeper insight: that on reasoning tasks, the attention distribution exhibits spatial locality across heads (tokens that are important for one head tend to be important for many heads) and recency locality across steps (recently generated tokens are consistently and repeatedly attended to by all heads). Prior work either missed these patterns (because they were analyzing simpler tasks where head specialization may be more pronounced) or observed related phenomena (e.g., StreamingLLM's attention sinks and recency windows) without connecting them to a unified selection strategy.

2. It is training-free by design, not by constraint. The paper is not claiming that training-free is inherently better; it is claiming that the observed localities are fundamental properties of the reasoning attention distribution that can be exploited without learning. If the patterns are truly universal to reasoning tasks, then no training should be necessary — a well-designed selection heuristic that respects these patterns should maintain accuracy across models and tasks. The empirical results across Qwen3-8B and Qwen3-4B on four different reasoning benchmarks (AIME-24/25, GPQA-Diamond, MATH500) provide evidence for this universality claim.

3. It generalizes across sparse attention backbones. The paper explicitly builds LessIsMore on top of TidalDecode's pipeline (Section 3) but notes in Section 5.2 that "LessIsMore's underlying techniques can be incorporated into any sparse attention using approximation algorithms." The ablation study in Figure 8 demonstrates that the unified selection principle improves attention recall even when applied to simple GQA aggregation schemes outside the TidalDecode framework. This positions LessIsMore as a design principle (unified global selection + recency budgeting) rather than a single algorithm, making it applicable to future sparse attention architectures as they evolve.

4. It directly addresses the accuracy-efficiency-length trilemma. Prior sparse attention methods face a three-way tension: increase sparsity (higher efficiency) at the cost of accuracy, or maintain accuracy at the cost of longer generation sequences (as the model compensates for missing context by re-deriving it). LessIsMore is positioned as breaking this tradeoff — achieving higher sparsity (2× fewer tokens than TidalDecode at iso-accuracy), maintaining or improving accuracy, and simultaneously shortening generation length by 7% (Table 1). This is possible because more accurate token selection prevents the model from entering redundant reasoning loops, so the efficiency gains from sparsity are not eaten up by increased token generation.

5. The paper's ambition is bounded but generative. It does not claim to solve all sparse attention problems for all tasks. The scope is explicitly reasoning tasks on GQA-based models. But within that scope, the paper aims to establish a new baseline: lossless accuracy at substantially higher sparsity than was previously thought possible without retraining. The future work section (5.2–5.4) then sketches how the underlying principles could extend to other attention architectures (MLA, MoE), be integrated into pretraining (native hybrid attention), and be improved with adaptive ratio selection — all directions that build on the paper's core insight rather than requiring its replacement.

In summary, the paper positions LessIsMore as a necessary corrective to the per-head local optimization paradigm that dominates sparse attention design. The evidence it marshals — attention recall degradation curves (Figure 1), visualization of cross-head token overlap (Figure 2), and stable recency patterns across decoding steps (Figure 3) — collectively makes the case that the conventional wisdom about attention head specialization does not hold in the reasoning regime, and that this explains why existing methods fail. The proposed solution is therefore not an incremental improvement but a principled redesign that aligns the selection mechanism with the actual attention structure of reasoning.

3. Technical Approach

3.1 Reader Orientation

LessIsMore is a plug-in token selection policy that sits between the KV cache and the attention computation in a transformer decoder, deciding at each decoding step which subset of all previously generated tokens the model should actually attend to when computing the next token. It solves the problem that existing sparse attention methods lose critical information during long reasoning chains — not by inventing a new way to approximate attention scores, but by recognizing that on reasoning tasks, attention heads largely agree on which tokens matter, so aggregating their individual preferences into a single unified ranking produces more robust and more efficient sparse attention than letting each head choose its own separate tokens.

3.2 Big-Picture Architecture (Diagram in Words)

The LessIsMore system has four major components that operate within a standard transformer decoder loop:

Full Attention Layers (the "sensing" layers): The first two layers and a small number of designated middle layers compute standard, dense attention over the entire KV cache. These layers serve two purposes — they provide the model with complete context at strategic points (preventing the accumulation of errors from ever-compounding partial views), and they generate the exact query-key dot products that the selection layers need to decide which tokens are important.
Token Selection Layers (the "decision" layers): Designated layers (typically Layer 12 for Qwen3-8B, Layer 20 for Qwen3-4B) compute full attention to produce exact attention scores, then execute the LessIsMore selection logic — per-head top- $k$ extraction, cross-head unification and ranking, and concatenation with the recent token window — to produce a single set of token indices $\rho$ that will be shared across all subsequent sparse attention layers.
Sparse Attention Layers (the "execution" layers): All remaining layers between selection layers load only the KV cache entries for the tokens in $\rho$ and compute attention exclusively over this reduced set. Critically, all attention heads in these layers share the same set of selected tokens — this is the key departure from prior work and the source of both accuracy and efficiency gains.
Stable Recency Window (the "budget" component): A fixed fraction $r = 0.25$ of the total token budget $K$ is permanently reserved for the most recently generated tokens. This window size scales proportionally with the total budget — if $K = 4000$ , then $1000$ tokens are always recent tokens, and the remaining $3000$ are selected via the unified ranking mechanism.

Information flows as follows: at each decoding step, the input hidden state passes through full attention layers (standard computation) → at the token selection layer, full attention is computed, the query-key dot products are extracted, each head selects its top- $k$ tokens, these are aggregated into a unified ranked list, the top $K \cdot (1 - r)$ from this list are combined with the $K \cdot r$ most recent tokens to form the selection set $\rho$ → all subsequent sparse attention layers in that decoding step load only the KV entries indexed by $\rho$ and compute attention over them → the process repeats at the next token selection layer.

3.3 Roadmap for the Deep Dive

First, the formal attention recall objective (Equation 3), which defines what sparse attention methods are trying to maximize and establishes the metric by which LessIsMore's design choices are evaluated — this is the loss function of the selection problem, even though LessIsMore itself is training-free.
Second, the backbone pipeline (TidalDecode's inter-layer hybrid architecture), because LessIsMore inherits a specific structural choice — two full attention layers, two token selection layers, sparse attention everywhere else — and understanding this inherited structure clarifies exactly where LessIsMore's innovations are inserted and what they replace.
Third, Unified Attention Head Selection, the first of LessIsMore's two core mechanisms, which aggregates per-head token selections into a single cross-head ranking. This sub-section explains the full procedure: per-head top- $k$ extraction, the UnionFlatten operation, global re-ranking, and pruning to the budget.
Fourth, Stable Recency Window, the second core mechanism, which reserves a fixed ratio $r$ of the token budget for recent tokens. This sub-section explains why a ratio rather than a fixed count is chosen, how it interacts with the unified selection, and the specific value $r = 0.25$ .
Fifth, the end-to-end decoding algorithm (Algorithm 1), which ties all components together into a single structured procedure. This sub-section walks through the pseudocode line by line, explaining what happens at each layer type and how state is maintained across layers within a single decoding step.
Sixth, the design choices and their justifications, a consolidated discussion of why per-head local optimization was replaced, why a ratio-based budget was chosen, why $r = 0.25$ , and why the GQA structure matters.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems + empirical analysis paper whose core idea is that reasoning tasks exhibit two attention localities — cross-head overlap in token importance (spatial locality) and persistent attention to recent tokens (recency locality) — and that a sparse attention mechanism designed to exploit these localities via unified cross-head ranking and proportional recent-token budgeting can outperform per-head local selection methods without any training.

The Attention Recall Objective: What Sparse Attention Is Trying to Maximize

Before diving into LessIsMore's architecture, it is essential to understand the formal objective that all sparse attention methods are implicitly optimizing: attention recall. The paper defines this precisely because it serves as the measurable quantity that connects a selection mechanism's decisions to downstream accuracy.

Standard attention for a single head $i$ computes:

$W_i = \frac{Q_i K_i^T}{\sqrt{d}}, \quad O_i = \text{softmax}(W_i) V_i$

where $Q_i$ is the query tensor for head $i$ (derived from the current token's hidden state), $K_i$ is the key tensor for head $i$ (derived from all previous tokens' hidden states and stored in the KV cache), $V_i$ is the value tensor, $d$ is the head dimension, $W_i$ is the matrix of raw attention scores before softmax, and $O_i$ is the head's output.

What this computes: for each head, the current token computes a compatibility score (dot product, scaled by $\sqrt{d}$ ) with every previous token. These scores are normalized via softmax into a probability distribution, and the head's output is a weighted sum of the value vectors of all previous tokens, with weights equal to these probabilities. The computation requires loading all $K_i$ and $V_i$ entries and computing $O(N^2)$ dot products where $N$ is the sequence length — this is the cost that sparse attention aims to reduce.

Why this form matters for sparsity: the softmax normalization means that a small number of tokens typically dominate the attention distribution — most of the probability mass concentrates on a few highly-relevant tokens, while the vast majority of tokens receive near-zero weight. Sparse attention exploits this by only computing attention over the tokens that are likely to receive significant weight, skipping the computation and memory access for the rest.

The core approximation in sparse attention is formalized as selecting a subset $\rho$ of token indices from the KV cache, constrained by a fixed budget $k = |\rho|$ , that maximizes some approximation function of the attention scores:

$\arg\max_{\rho} f(Q_i, K_i[\rho], V_i[\rho], k), \quad |\rho| = k$

where $\rho$ is the selected subset of token indices, $k$ is the token budget (a hyperparameter controlling the sparsity level), and $f$ is an efficient approximation to the ground-truth attention scores — different sparse attention methods differ primarily in how they define $f$ .

What this computes: given that we can only afford to attend to $k$ tokens out of the total $N$ available, which $k$ tokens should we choose? The answer depends on $f$ , which approximates how much attention each token would receive if we computed the full softmax. For example, a common choice for $f$ is simply the query-key dot product magnitude — tokens with large dot products are likely to receive high post-softmax attention. The constraint $|\\rho| = k$ enforces the sparsity budget.

Why this is an optimization problem: we cannot simply compute exact attention and pick the top- $k$ tokens, because computing exact attention requires loading all keys and values — exactly the work we are trying to avoid. Instead, we compute a cheaper proxy $f$ and hope that it ranks tokens similarly to the true attention scores.

The quality metric for this approximation is attention recall:

$R_i = \frac{\sum(\text{softmax}(W_i)[\rho])}{\sum(\text{softmax}(W_i))}$

where $\text{softmax}(W_i)[\rho]$ selects only the softmax-normalized attention weights for the tokens in the chosen subset $\rho$ , and the denominator is 1 (since softmax outputs sum to 1), making $R_i$ simply the sum of attention weights on the selected tokens.

What it computes: if we compute full attention (loading all keys), we get a probability distribution over all tokens. Attention recall $R_i$ is the fraction of that probability mass that falls on the tokens we actually selected — it measures how much of the "true" attention we captured with our sparse approximation. A recall of 1.0 means the selected tokens captured all attention mass (essentially, no important tokens were missed). A recall of 0.75 means we missed 25% of the attention mass, which means the head's output will be a distorted version of the true output.

Why this form matters for reasoning: on retrieval tasks, a single important token (e.g., the needle in a haystack) might carry most of the attention mass, so even imperfect selection can achieve high recall. On reasoning tasks, attention is distributed across many tokens spanning the entire reasoning chain — earlier intermediate results, definitions, and problem constraints all matter. Achieving high recall requires selecting tokens from across the entire context, not just a local window. The compounding error that Figure 1 illustrates is essentially the phenomenon where per-step recall losses accumulate: if each step loses 1% recall, after 1000 steps the effective recall may be far lower because tokens that were important but not selected at step $t$ may be unavailable at step $t + \Delta t$ when they become critical.

The Backbone Pipeline: TidalDecode's Inter-Layer Hybrid Architecture

LessIsMore does not define its own pipeline from scratch; it inherits TidalDecode's structural design (Section 3, Figure 4) and replaces only the token selection logic within that structure. Understanding this backbone is necessary because it determines when and how often token selection decisions are made, which in turn determines how much the quality of those decisions matters.

The TidalDecode pipeline operates a hybrid full/sparse attention architecture across the layers of the transformer decoder. For a single decoding step (generating one new token), the processing proceeds as follows:

Layer 1 and Layer 2: Full Attention. The first two layers always compute standard dense attention over the entire KV cache. The paper states this explicitly: "two full attention layers" at the start. The motivation (inherited from TidalDecode and broader sparse attention design intuitions) is that the earliest layers perform foundational context integration — they need to establish the basic relationships between the current query token and the full context. Restricting these layers to a subset of tokens could compromise the model's ability to even begin forming useful representations of the input and recent context. These two layers serve as a "safety net": they ensure that at least some layers have complete information, limiting how far errors from sparse attention can propagate.

Layer 3: Token Selection Layer with Full Attention. The third layer (or, more generally, the first designated selection layer — the paper uses Layer 12 for Qwen3-8B, Layer 20 for Qwen3-4B, as determined by the needle-in-the-haystack optimization procedure described in Appendix A.1) computes full attention and simultaneously extracts the query-key dot products $P = q \cdot \mathcal{C}.K^T$ . This dual purpose is critical: the layer serves its normal function in the network (producing output representations for the next layer) and acts as a sensor that provides the raw signal the selection mechanism needs. Computing full attention here is unavoidable — the selection needs exact dot products to make accurate decisions — but the cost is amortized because this information enables sparse attention in all subsequent layers until the next selection point.

All Remaining Layers Until the Next Selection Layer: Sparse Attention. Every layer between the current selection layer and the next one (or the end of the decoder stack) loads only the KV entries indexed by $\rho$ , the token subset selected at the most recent selection layer. These layers never see tokens outside $\rho$ . This is where the computational savings come from: instead of loading and computing dot products with $N$ keys, these layers load and compute with only $k \ll N$ keys.

The Next Selection Layer: Full Attention + Re-Selection. At a designated middle layer (for the models used in the paper, typically only two selection layers total — one early and one middle — with sparse attention everywhere else), the process repeats: full attention is computed, new dot products are extracted, and a new selection set $\rho'$ is produced for the remaining layers.

Why this structure matters for LessIsMore's design: the quality of the token selection $\rho$ at a selection layer determines the quality of attention in all subsequent sparse layers until the next re-selection. There is no per-layer adjustment — once $\rho$ is chosen, every sparse attention head in every subsequent layer uses exactly the same set of tokens. This is the key difference from prior selection-based methods like TidalDecode, where each head independently selects its own tokens and can do so at every layer if desired. Making a single selection set work for all heads and all subsequent layers is exactly what forces the selection to be globally robust rather than locally optimal — and this is precisely what LessIsMore's unified selection mechanism is designed to achieve.

The paper notes that for Qwen3-8B, the selection layer is Layer 12; for Qwen3-4B, it is Layer 20. These were chosen via a needle-in-the-haystack calibration (Appendix A.1): "With a token budget of 256, Layer 12 achieves the best accuracy of 86% on Qwen3-8B; Layer 12 and 20 achieve the best accuracies of 86% and 84% on Qwen3-4B, respectively." The choice to use different selection layers for different models (rather than a fixed layer index) reflects the fact that the optimal point for re-selection depends on the model's internal representation dynamics — too early, and the selection has insufficient context to make good decisions; too late, and too many layers have already been constrained by a stale selection.

Unified Attention Head Selection

This is the first of LessIsMore's two core innovations, and it directly operationalizes the spatial locality observation from Section 2.2.1 — the finding that attention heads within the same decoding layer exhibit substantial overlap in which tokens they consider important.

The problem with per-head selection. In standard selection-based sparse attention (Equation 2), each attention head independently solves its own optimization problem: find the $k$ tokens that maximize its own approximation function $f(Q_i, K_i[\rho], V_i[\rho], k)$ . This produces up to $H$ different token subsets (one per head), where $H$ is the number of attention heads. When these subsets are different — as they are under the assumption that heads are specialized — implementing sparse attention requires either (a) loading different KV entries for each head (complex memory access patterns, poor hardware utilization) or (b) taking the union of all per-head selections and loading those for all heads (increased effective budget, reduced sparsity). The paper's insight is that on reasoning tasks, the subsets are actually not that different — heads substantially agree — which means that per-head selection is solving $H$ nearly-identical optimization problems separately, wasting both selection computation and the opportunity to use a single, more robust, globally-ranked set.

The unified selection procedure (Algorithm 1, lines 11–14). LessIsMore replaces per-head independent selection with a three-step pipeline:

Step 1: Per-head top- $k$ extraction. At the token selection layer, after full attention is computed, the query-key dot product matrix $P$ is available. This matrix has dimensions $[H, N]$ , where $H$ is the number of attention heads and $N$ is the current sequence length. For each head $h$ , the system identifies the indices of the $k$ tokens with the largest dot product magnitudes, but excluding the most recent $K \cdot r$ tokens:

$\rho_{\text{head}} = \text{TopKIndices}(P[:, :-(K \cdot r)], k = K \cdot (1 - r))$

where $P[:, :-(K \cdot r)]$ takes all heads but only columns up to $N - K \cdot r$ (excluding the recent window), $K$ is the total token budget, and $r$ is the static ratio for the recency window. The parameter $k = K \cdot (1 - r)$ means each head selects exactly $K \cdot (1 - r)$ tokens from the non-recent portion of the context.

Why exclude recent tokens here: the recent tokens will be added separately by the Stable Recency Window. If they were included in the per-head selection, they would consume slots in the budget that are already guaranteed to them, effectively double-counting. This exclusion ensures that the per-head selection focuses entirely on identifying important historical tokens — tokens from earlier in the reasoning chain that might otherwise be forgotten — while the recent window handles the recency locality pattern.

Step 2: Cross-head aggregation via UnionFlatten. The per-head selections $\rho_{\text{head}}$ are aggregated into a single unified list:

$\rho_{\text{unified}} = \text{UnionFlatten}(\rho_{\text{head}})$

What UnionFlatten does: each head produces an ordered list of $K \cdot (1 - r)$ token indices, ranked from highest to lowest dot product magnitude for that head. UnionFlatten takes all $H$ of these lists — a total of $H \cdot K \cdot (1 - r)$ indices, which may include duplicates if the same token was selected by multiple heads — and flattens them into a single list, preserving the within-head ranking order. A token that was ranked first by head 1 and third by head 2 appears in this flattened list at positions corresponding to both rankings. The operation is conceptually similar to concatenating all per-head ranked lists.

Why flatten rather than deduplicate immediately: the ranking information is preserved because tokens that appear early in multiple heads' lists (indicating broad consensus on their importance) will appear multiple times near the front of the unified list, which matters for the next step. If deduplication were done first, a token selected by only one head (but ranked highly by that head) would be indistinguishable from a token selected by all heads (but ranked moderately by each). The flattening retains the signal of cross-head consensus.

Step 3: Global pruning to budget. The unified list is sorted globally — tokens are re-ranked based on how many heads selected them and at what rank — and then pruned to the budget size:

$\rho = \rho_{\text{unified}}[:K \cdot (1 - r)] \cup \rho_{\text{recent}}$

What this means operationally: the globally-sorted unified list is truncated to exactly $K \cdot (1 - r)$ entries (keeping only the highest-ranked tokens across all heads), and these are concatenated with the $\rho_{\text{recent}}$ indices (the $K \cdot r$ most recent tokens from the Stable Recency Window) to form the final selection set $\rho$ of size exactly $K$ . All sparse attention layers between this selection layer and the next selection layer will load only the KV entries for tokens in $\rho$ .

Why global pruning matters: a token that is the single most important token for head 7 but irrelevant to all other heads will be ranked globally based on its rank within head 7's list. If it is head 7's top-ranked token, it appears at the very front of head 7's contribution to the unified list and is likely to survive pruning. However, tokens that are moderately important across many heads — appearing in the top-10 for, say, 20 out of 32 heads — will generate 20 entries near the front of the unified list, crowding out tokens that are critically important to only one head. The global ranking therefore implicitly favors tokens with broad, cross-head relevance (which aligns with the spatial locality observation) over tokens that are highly specialized to a single head (which are more likely to reflect noise or head-specific artifacts).

Computational complexity of unified selection. The per-head top- $k$ extraction operates on the full $P$ matrix and costs $O(H \cdot N)$ to find top- $k$ per head (practically, this is done via a partial sort or selection algorithm). The UnionFlatten and sort operations operate on $O(H \cdot k)$ elements. Compared to per-head selection, which would require selecting and storing $H$ different lists of size $k$ , the unified approach actually reduces the total number of unique token indices that must be loaded (since duplicates are eliminated in the pruning step), which is what enables the efficiency gains reported in Section 4.3.

Stable Recency Window

This is the second core innovation, operationalizing the recency locality observation from Section 2.2.2 — the finding that recent tokens are consistently and repeatedly attended to by all attention heads throughout the decoding process, and that the size of this important recency window stays relatively constant relative to the total number of selected tokens.

The observation driving the design. Figure 3 shows that across decoding steps 10K, 15K, 20K, and 25K, the most recently generated tokens consistently appear in the top-4K selections of all attention heads — they form a "band" at the right edge of the attention matrix that is uniformly dark (indicating high overlap in selection). The paper refers to this as "recency locality." Prior work (StreamingLLM by Xiao et al., 2023) had already identified the importance of recent tokens and attention sinks, but maintained a fixed-size sliding window — e.g., always keep the last 256 tokens regardless of total budget or generation length.

Why a fixed-size window is suboptimal for reasoning. If the number of recent tokens that matter scales with the reasoning context, a fixed-size window becomes proportionally too small as the budget shrinks. With a budget of $K = 4000$ and a fixed recent window of 256 tokens, only 6.4% of the budget goes to recency. But at $K = 1000$ , that same 256-token window consumes 25.6% of the budget — inconsistent, and potentially crowding out historical tokens at small budgets when they are most needed. Conversely, for very long generations where the recent reasoning context might span hundreds of tokens (encompassing multiple reasoning steps), a fixed window of 256 might be too small.

The ratio-based design. LessIsMore instead allocates a fixed ratio $r$ of the total token budget $K$ to the recent window:

$\rho_{\text{recent}} = \text{Recent}(K \cdot r)$

where $\text{Recent}(m)$ returns the indices of the $m$ most recently generated tokens. With $r = 0.25$ (the value used throughout the paper's experiments, Section 4.1: "The static ratio $r$ of LessIsMore is set to 0.25"), the recent window size scales proportionally with the budget: at $K = 4000$ , the recent window is 1000 tokens; at $K = 2000$ , it is 500 tokens; at $K = 1000$ , it is 250 tokens.

Why a ratio rather than an absolute count: the paper's observation (Section 2.2.2) is that "the ratio between critical recency window size and the number of selected tokens remains stable throughout the reasoning process." This is a statement about proportionality — the recent context that matters for the current reasoning step grows roughly linearly with the total amount of context the model is maintaining. A ratio-based allocation directly encodes this proportionality, ensuring that the recency budget remains a consistent fraction of total attention resources regardless of the absolute sparsity level.

Why $r = 0.25$ : the paper provides empirical justification in the ablation study of Section 4.4.3 and Figure 9. Under a 4K token budget on AIME-24 with Qwen3-8B, configurations with $r = 25\\%$ , $r = 50\\%$ , $r = 75\\%$ , and $r = 100\\%$ (pure recency) are compared. The $r = 25\\%$ configuration — which is the full LessIsMore design (Unified Attention Head Selection + Stable Recency Window with $r = 0.25$ ) — achieves the highest cumulative attention recall throughout most of the generation process. Configurations with $r = 0\\%$ (Unified Attention Head Selection only, no recency window) and TidalDecode both fail to produce the correct answer on the evaluated problem. All configurations with $r > 0$ (25%, 50%, 75%) succeed in producing the correct answer, indicating that some recency allocation is necessary, but $r = 25\\%$ provides the best balance between recent and historical token budgets.

Interaction with attention sinks. The paper notes in Section 4.1 that "4 tokens are always reserved for attention sink in this section." Attention sinks are a small number of initial tokens (often the first few tokens of the sequence) that receive disproportionately high attention scores regardless of their semantic content (Xiao et al., 2023). By reserving 4 tokens explicitly for attention sinks, LessIsMore ensures that these tokens — which serve a structural role in the attention mechanism (likely related to the softmax's need for a "null" attention target) — are not crowded out by either the unified historical selection or the recency window. This is a minor but important detail: without explicit reservation, these sink tokens might not be selected by the unified ranking (since their importance is structural rather than content-based), and their absence could distort the softmax normalization.

Why the recency window and unified selection are complementary. The recency window guarantees coverage of the rightmost portion of the context, where step-to-step reasoning continuity lives. The unified selection covers the left and middle portions, where earlier intermediate results, problem constraints, and key definitions live. Separating the budget this way — and excluding the recent window from the unified selection's domain — prevents the unified ranking from simply filling its budget with recent tokens (which would be ranked highly by most heads anyway, per the recency locality observation) and neglecting the historical context. The explicit partitioning ensures that both recency and history are represented in the final selection set.

The End-to-End Decoding Algorithm (Algorithm 1 Line-by-Line)

Algorithm 1 in the paper presents the full LessIsMore decoding pipeline as pseudocode. Walking through it line by line provides the complete picture of how the components interact.

Inputs (Line 1): The algorithm takes the current hidden state $h$ (the output of the previous decoding step or the embedding layer for the first token), the KV cache $\mathcal{C}$ (which stores the key and value tensors for all previously generated tokens, with new entries appended at each step), the token budget $K$ (a hyperparameter controlling sparsity level), and the static ratio $r$ ( $0.25$ in all experiments).

Output (Line 2): The algorithm produces logits — unnormalized scores over the vocabulary — from which the next token is sampled.

Initialization (Line 3): The token selection buffer $\rho$ is initialized as an empty list. This buffer holds the indices that sparse attention layers will use; it is populated at token selection layers and read (but not modified) at sparse attention layers.

The per-layer loop (Line 4): The algorithm iterates through each decoder layer $i$ of the transformer. At each layer, the same basic operations occur — query/key/value projection, KV cache update — but what happens after depends on the layer type.

Query, key, value projection (Line 5): The current hidden state $h$ is projected through the model's learned weight matrices to produce the query $q$ , key $k$ , and value $v$ tensors. This is standard in all transformer decoders. The projection $f(W_{qkv}, h)$ maps the hidden state from the model dimension to $H$ heads, each with dimension $d$ .

KV cache update (Line 6): The newly computed key $k$ and value $v$ are appended to the KV cache $\mathcal{C}$ . This means $\mathcal{C}$ grows by one entry per decoding step. In selection-based methods like LessIsMore, the full KV cache is retained — no tokens are permanently evicted. The sparsity comes from which entries are read, not from which entries are stored.

Full attention layers (Lines 7–8): If layer $i$ is a full attention layer (layers 1, 2, and potentially the very last layer), standard dense attention is computed: $o = \text{FullAttention}(q, \mathcal{C}[:])$ , where $\mathcal{C}[:]$ means the entire KV cache. This produces the layer output $o$ via the standard scaled dot-product attention mechanism. No token selection occurs; the output reflects attention over all available context.

Token selection layers (Lines 9–15): If layer $i$ is a designated token selection layer, two things happen simultaneously:

First (Line 10): Full attention is computed exactly as in a full attention layer, producing the output $o$ that will be passed to the next layer. Simultaneously, the query-key dot product matrix $P = q \cdot \mathcal{C}.K^\top$ is extracted. This $P$ has shape $[H, N]$ , where $H$ is the number of attention heads and $N$ is the current KV cache length.

Second (Lines 11–14): The LessIsMore token selection procedure is executed on $P$ to populate $\rho$ :

Line 11: Per-head top- $k$ extraction, excluding the recent window: each head selects the $K \cdot (1 - r)$ tokens with the largest dot product from $P[:, :-(K \cdot r)]$ .
Line 12: The per-head selections are flattened into a unified list $\rho_{\text{unified}}$ via UnionFlatten.
Line 13: The recency window indices $\rho_{\text{recent}}$ are set to the indices of the $K \cdot r$ most recent tokens.
Line 14: The final selection set $\rho$ is formed by taking the first $K \cdot (1 - r)$ entries from the globally-ranked $\rho_{\text{unified}}$ and concatenating them with $\rho_{\text{recent}}$ .

The selection set $\rho$ is now populated and will be used by all subsequent sparse attention layers in this decoding step.

Sparse attention layers (Lines 15–16): If layer $i$ is a sparse attention layer (all layers between selection layers that are not full attention layers), attention is computed only over the tokens in $\rho$ : $o = \text{SparseAttention}(q, \mathcal{C}[\rho])$ . This means that only the KV cache entries indexed by $\rho$ are loaded from memory, and the attention dot products are computed over $k$ keys rather than $N$ keys. All attention heads in these layers share the same $\rho$ — there is no per-head variation.

Feed-forward network (Line 18): After attention (whether full or sparse), the output $o$ passes through the standard feed-forward network (FFN) to produce the hidden state for the next layer: $h = \text{FFN}(o)$ .

After all layers (Line 20): The final hidden state is projected to vocabulary size via the language model head: lm_head(h), producing logits for next-token prediction.

What happens at the next decoding step: The entire process repeats for the next token. On the first step after a selection layer, the $\rho$ from the previous selection is stale (it was computed using dot products from the previous token's query). The first sparse attention layer encountered in the new step will still use this stale $\rho$ until the next selection layer is reached, at which point fresh dot products are computed and a new $\rho$ is generated. This is why the paper emphasizes that the selection must be robust enough to remain useful across multiple layers and even across the boundary into the next decoding step — a locally-optimal selection that overfits to the specific query at the selection layer may not generalize to queries in subsequent layers.

The 4 attention sink tokens. The paper does not specify in the pseudocode but notes in Section 4.1 that "4 tokens are always reserved for attention sink." In practice, this likely means that the first 4 tokens of the sequence are always included in $\rho$ , either by explicit addition before the final selection or by being treated as always-present. This ensures that the structural role these tokens play in the softmax normalization is maintained regardless of selection decisions.

Design Choices and Their Justifications

Why replace per-head local optimization with unified global selection? The paper's motivating observation (Figure 2, Section 2.2.1) directly challenges the conventional wisdom that attention heads are highly specialized. On reasoning tasks, the top-4K tokens show "high overlap among selected tokens across consecutive groups of four attention heads within the same key-value group" and even "broader overlaps spanning all attention heads." This means that per-head selection is solving $H$ nearly-redundant optimization problems. The unified approach replaces $H$ separate decisions with one consensus decision, which is both more efficient (fewer unique indices to manage, simpler KV cache access patterns) and more robust (tokens that are broadly important across many heads are prioritized over tokens that are important to only one head). The paper validates this choice in the ablation study of Section 4.4.1 (Figure 8), where LessIsMore achieves substantially higher attention recall than head-to-head or randomized selection schemes, particularly when selection is performed infrequently (only at Layer 2 rather than all layers).

Why use a ratio rather than a fixed count for the recency window? The fixed-count approach (e.g., StreamingLLM's sliding window of 256 tokens) assumes that the absolute number of recent tokens needed is constant regardless of total budget or generation length. The paper's observation (Section 2.2.2) is that "the ratio between critical recency window size and the number of selected tokens remains stable throughout the reasoning process." A ratio-based approach encodes this proportionality directly. At small budgets, it prevents the recency window from dominating the selection (a fixed 256-token window at $K = 1000$ would consume 25.6% of the budget for recency alone). At large budgets, it ensures that the recency allocation grows to encompass the full recent reasoning context rather than truncating it. The ratio $r = 0.25$ is empirically validated in Figure 9 as providing the highest cumulative attention recall.

Why $r = 0.25$ specifically? The ablation in Section 4.4.3 and Figure 9 compares $r = 0\%$ , $25\%$ , $50\%$ , $75\%$ , and $100\%$ on an AIME-24 problem with a 4K budget. The $25\%$ setting achieves the highest attention recall throughout most of the generation and is the only configuration that combines Unified Attention Head Selection with a non-zero recency window. Configurations at $0\%$ (unified selection only, no recency window) fail to produce the correct answer, as does TidalDecode (which also lacks an explicit recency allocation), confirming that some recency budget is necessary. Configurations at $50\%$ and $75\%$ succeed but achieve lower recall than $25\%$ , suggesting they over-allocate to recency at the expense of historical context. The $25\%$ value thus represents the empirically optimal balance for the evaluated models and tasks.

Why does GQA matter for this design? The paper explicitly evaluates on Qwen3-8B and Qwen3-4B, both of which use Grouped Query Attention (GQA). In GQA, multiple query heads share a single key-value head — for Qwen3-8B, there are 32 query heads but only 8 KV heads, meaning each KV head is shared by 4 query heads. Under standard per-head selection, each of the 32 query heads would independently select its own tokens, but the KV cache is organized by KV head, meaning that 4 query heads' selections must be aggregated to determine which entries to load for each KV head. This aggregation is non-trivial: if the 4 query heads disagree on which tokens to select, loading the union of their selections reduces sparsity. LessIsMore's unified approach naturally handles this by producing a single selection set across all heads, eliminating the per-KV-group aggregation problem entirely. The paper includes an ablation (Figure 8) that directly evaluates different GQA aggregation strategies, confirming that unified selection outperforms alternatives.

Why is LessIsMore training-free? The paper argues (and empirically demonstrates) that the localities it exploits — cross-head spatial overlap and recency persistence — are properties of the reasoning attention distribution, not learned features that require training to discover. If these properties hold across reasoning tasks and model architectures, then a heuristic that respects them (unified ranking + ratio-based recency) should work without training. The results across two model sizes and four task benchmarks support this universality claim. The training-free nature is valuable not just for convenience (no adaptation cost) but as evidence that the observed localities are fundamental — if training were necessary, it would suggest the patterns are model-specific or task-specific artifacts rather than structural properties.

Why build on TidalDecode rather than a simpler baseline? TidalDecode is the paper's "backbone" (Section 3, paragraph 1): it provides the inter-layer hybrid architecture (full attention layers + selection layers + sparse attention layers) and the concept of position-persistent selection (selection decisions made at one layer persist across subsequent layers). LessIsMore inherits this structural design and replaces only the token selection logic within selection layers. This modular approach makes the contribution clean: any accuracy or efficiency improvement over TidalDecode can be directly attributed to the unified selection and recency window, since the surrounding pipeline is identical. The paper notes that LessIsMore "adopts TidalDecode ... as a backbone; specifically, it starts with two full attention layers, includes two dedicated token selection layers, and employs sparse attention in the remaining layers" (Section 3, first paragraph). The choice of which layers serve as selection layers is validated through the needle-in-the-haystack calibration in Appendix A.1.

Why a token budget rather than a sparsity ratio? The paper uses absolute token budgets $K$ (1K, 2K, 4K, 6K) rather than specifying sparsity as a fraction of sequence length. This is a practical choice: with generation lengths varying widely across problems (from ~4K tokens for MATH500 to ~20K tokens for AIME-24), a fixed budget provides predictable latency and memory characteristics. At generation length 20K with a 4K budget, the sparsity is 80% (only 20% of tokens attended). At generation length 4K with the same 4K budget, sparsity is 0% (full attention). The budget serves as a computational cap: regardless of how long the generation becomes, the per-token attention cost is bounded by the fixed budget $K$ .

4. Key Insights and Innovations

Innovation 1: The Cross-Head Consensus Observation — A Diagnostic That Reframes the Sparse Attention Problem for Reasoning

The paper's most intellectually distinctive move is not an algorithm but a diagnosis: the reason existing sparse attention methods fail on reasoning tasks is that they inherit an incorrect assumption about attention head specialization, and the evidence for this misdiagnosis is directly visible in the attention distributions themselves. This reframes the problem from "sparse attention is inherently lossy on long reasoning sequences" (an inevitable-tradeoff framing that would motivate training-based mitigation) to "sparse attention is being applied with the wrong structural prior" (a fixable-design framing that motivates a training-free correction).

What the field assumed before this paper. The dominant paradigm in sparse attention design — encoded in both selection-based methods like TidalDecode (Yang et al., 2024) and Quest (Tang et al., 2024) and eviction-based methods that compute per-head importance scores (Li et al., 2024; Zhang et al., 2023) — is that attention heads are functionally specialized and therefore benefit from maintaining distinct token subsets. This assumption traces back to the multi-head attention literature (Voita et al., 2019; Michel et al., 2019) showing that different heads learn different linguistic functions. Xiao et al. (2024) explicitly characterize heads into "retrieval heads" and "streaming heads" with different attention patterns. Under this worldview, per-head top-k selection is not just efficient — it is necessary for optimality, because a token that is critical for a retrieval head might be irrelevant to a streaming head, and forcing both to share the same tokens would degrade at least one head's attention quality.

What this paper shows instead. The visualization in Figure 2 (Section 2.2.1) provides direct counterevidence for reasoning tasks: the top-4K tokens across 32 attention heads at a representative decoding step show "high overlap among selected tokens across consecutive groups of four attention heads within the same key-value group" and "broader overlaps spanning all attention heads." The pattern is not that heads are interchangeable — some heads do attend to slightly different token subsets — but that there is substantial consensus in token importance rankings. On a reasoning problem, the intermediate algebraic result that matters for logical continuity matters to most heads, not just one specialized head.

This is not a minor empirical tweak. It fundamentally reframes the optimization landscape that sparse attention methods are navigating. Per-head selection solves $H$ separate local optimization problems:

$\arg\max_{\rho_h} f(Q_h, K_h[\rho_h], V_h[\rho_h], k) \quad \text{for each head } h$

where $f$ approximates that head's attention scores. If heads substantially agree on which tokens matter, then these $H$ optimization problems have overlapping solutions — the global optimum (selecting tokens that maximize recall summed across all heads) is close to each head's local optimum. But per-head selection cannot find the global optimum because it never aggregates information across heads: head 1 might select its locally-optimal tokens, head 2 its locally-optimal tokens, and the union may be larger than the budget or may miss tokens that are moderately important to many heads but not top-k for any single head. Unified selection, by aggregating rankings before pruning, directly targets the global objective.

Why this matters beyond raw accuracy gains. The cross-head consensus observation provides a principled explanation for a previously puzzling empirical phenomenon: why sparse attention methods that achieve excellent recall on retrieval tasks collapse on reasoning tasks. The collapse is not because reasoning tasks are "harder" in some vague sense — it is because the structural assumption of head specialization that underlies per-head selection is less true on reasoning tasks than on retrieval tasks. On retrieval, a specialized "retrieval head" might genuinely attend to tokens that no other head cares about, making per-head selection appropriate. On reasoning, where the logical chain creates broad cross-head relevance, forcing each head to solve an isolated selection problem wastes budget on head-specific noise while missing consensus-important tokens.

This diagnosis has generative implications. It predicts that any sparse attention method that makes a per-head independence assumption will underperform on reasoning regardless of how clever its approximation function $f$ is — and Figure 6 confirms this, with Quest, TidalDecode, and SeerAttention-r all showing accuracy degradation at low budgets while LessIsMore (which drops the per-head independence assumption) does not. It also suggests that the search for better sparse attention methods on reasoning should focus on cross-head information sharing rather than better per-head approximation functions — a reorientation of the research agenda that the paper enables but does not exhaust.

Evidence anchor. Figure 2 (visualization of cross-head token overlap), Figure 8 (ablation showing unified selection substantially outperforms head-to-head and randomized aggregation, especially under infrequent re-selection), and the consistent accuracy advantage in Figure 6 across four benchmarks and two model sizes.

Innovation 2: The Proportional Recency Budget — From Fixed-Size Sliding Windows to Ratio-Based Allocation

The Stable Recency Window is conceptually distinct from prior recency-aware techniques because it replaces an absolute-size heuristic with a proportional allocation principle grounded in an empirical observation about reasoning dynamics. This shift from "always keep the last N tokens" to "always allocate r fraction of the budget to recent tokens" has implications for how sparse attention budgets should be designed across varying sparsity levels and generation lengths.

What the field did before. The importance of recent tokens for language modeling is well-established. StreamingLLM (Xiao et al., 2023) demonstrated that maintaining a fixed-size sliding window of recent tokens alongside a small set of attention sink tokens is sufficient to stabilize perplexity on long sequences, even when all other tokens are evicted. Subsequent eviction-based methods (Zhang et al., 2023; Li et al., 2024) typically retain a fixed number or fixed proportion of recent tokens alongside importance-selected historical tokens. In the selection-based family, implicit recency emerges from the approximation function itself (since recent tokens often have high query-key dot products), but no explicit budget allocation is made.

The fixed-size approach implicitly assumes that the absolute amount of recent context needed is independent of the total budget, the generation length, and the task complexity. For a retrieval task where the model only needs to maintain local syntactic coherence, this assumption is reasonable — 256 recent tokens might always be enough. The paper does not challenge this assumption for those tasks; it argues that it breaks down for reasoning.

What this paper shows instead. Section 2.2.2 and Figure 3 establish a more specific claim: on reasoning tasks, the ratio between the size of the important recent context and the total number of tokens being attended to "remains stable throughout the reasoning process." This is visible in the consistent width of the high-overlap band at the right edge of the attention matrices in Figure 3(a)–(d). The observation is not just that recent tokens matter (trivial), or that they matter more as generation length grows (expected), but that they matter in stable proportion to the total attention budget — a much stronger claim that implies a specific mathematical relationship between recency allocation and total budget.

Why this is a conceptual upgrade rather than a parameter tweak. A fixed-size recent window: (a) becomes proportionally larger as the total budget shrinks, crowding out historical tokens at low budgets when they are most needed; (b) does not automatically scale with generation length, so a window optimized for short generations may be too small for the extended reasoning context of a 20K-token AIME problem; and (c) requires per-task tuning — the right window size for retrieval is likely wrong for reasoning. A ratio-based allocation solves all three problems simultaneously by making the recency budget self-scaling: it is always the same fraction of total resources regardless of absolute budget or generation length.

More subtly, the ratio-based design encodes an assumption about why recency matters in reasoning. If recency were just about local syntactic coherence (maintaining grammatical continuity), a fixed window would suffice — grammatical structure depends on a roughly constant number of preceding tokens regardless of task. But in step-by-step reasoning, each new logical step builds on the conclusions of the immediately preceding steps (Lee & Hockenmaier, 2025). As the reasoning chain grows longer, the "immediately preceding context" that is logically relevant to the current step may itself grow — the model needs to see not just the last equation, but the last several equations that together form the sub-conclusion being used. A ratio-based allocation naturally accommodates this by scaling the recent window with the total context the model is maintaining.

Significance beyond the specific value r = 0.25. The paper's ablation (Figure 9) finds that r = 0.25 is optimal for the evaluated setting, and that any r > 0 (combined with unified selection) is sufficient to solve the test problem while r = 0 (unified selection only) fails. But the deeper contribution is the ratio-as-design-principle, not the specific value. The optimal ratio likely depends on model architecture, task distribution, and budget level — and the paper's future work section (5.1) explicitly calls for adaptive ratio determination. What the paper establishes is not "always use 25%" but "always think in ratios, not absolute counts, and validate the ratio empirically" — a methodological contribution that applies to any future sparse attention method for reasoning.

Evidence anchor. Figure 3 (visualization of consistent recency band width across decoding steps), Figure 9 (ablation showing r = 0.25 achieves highest recall and that configurations with r = 0% fail to produce the correct answer), and the generation length analysis in Table 1 (showing LessIsMore maintains generation length comparable to full attention while TidalDecode generates 18% longer sequences at K = 2000, consistent with the idea that explicit recency allocation prevents redundant re-derivation).

Innovation 3: The Accuracy-Length-Efficiency Trilemma Is Breakable — Attending to Fewer Tokens Can Produce Shorter, More Accurate Generations

A counterintuitive empirical finding running throughout the paper's evaluation is that higher sparsity, when combined with better token selection, does not trade accuracy for length — it can improve both simultaneously while reducing latency. This challenges an implicit assumption in the sparse attention literature that sparsity is a compromise: you accept some accuracy loss and/or some increase in generation length in exchange for computational savings. The paper demonstrates that this trilemma is not fundamental — it is an artifact of poor token selection that can be resolved by a selection mechanism well-matched to the task's attention structure.

The vicious cycle in prior work. The paper documents (Section 1, corroborated by Gao et al., 2025) that existing sparse attention methods exhibit a tendency to extend generation lengths on reasoning tasks. Table 1 quantifies this: at K = 2000 tokens, Quest generates 30.0K tokens (103% increase over full attention's 14.8K), SeerAttention-r generates 19.8K (34% increase), and TidalDecode generates 17.4K (18% increase). The mechanism is straightforward: when sparse attention drops tokens that contain earlier intermediate results, the model loses access to its own prior reasoning steps. Rather than producing an incorrect answer immediately, the model often re-derives the missing information — generating additional tokens to reconstruct what it forgot. This extra generation increases latency (partially or fully offsetting the per-token speedup from sparsity) and provides more opportunities for further selection errors, potentially leading to even more re-derivation. The vicious cycle compounds.

What LessIsMore demonstrates instead. The rightmost columns of Table 1 show a different pattern: at K = 2000, LessIsMore generates 14.9K tokens (only 1% more than full attention, well within sampling variance). At K = 4000, it generates exactly the same 14.8K tokens as full attention while achieving higher accuracy (the exact values are in the paper's Table 1 in Appendix A.2, but the text reports "better accuracy"). At K = 6000, LessIsMore generates 14.4K tokens (slightly shorter than full attention). The generation length does not balloon as sparsity increases — if anything, it slightly decreases at high budgets.

This is not a marginal improvement. It breaks the perceived tradeoff structure that defined the problem space. Prior work implicitly accepted that sparse attention means either (a) accuracy loss, (b) length increase, or (c) very low sparsity (high token retention). LessIsMore achieves none of these penalties — accuracy at or above full attention baselines, generation length at or below full attention baselines, and up to 87.5% sparsity on AIME-24 (Section 1 mentions "up to 87.5% sparsity" with lossless accuracy, which at a generation length of ~15K implies a token budget of approximately 2K).

Why this finding is conceptually significant. The trilemma-breaking result implies that the failure mode of prior sparse attention methods was not "sparsity is lossy" in some information-theoretic sense — it was "sparsity with the wrong selection prior is lossy in ways that trigger pathological model behavior (redundant re-derivation)." When the selection prior is aligned with the actual attention structure (unified cross-head ranking respects spatial locality; ratio-based recency respects temporal locality), the model finds the information it needs on the first try and does not need to re-derive it. The efficiency gains from sparsity are therefore multiplicative rather than eroded: you save compute per token and you avoid generating extra tokens to compensate for lost context.

This has an important corollary: attention recall is a leading indicator of generation length. The paper measures attention recall as a diagnostic (Figure 1, Figure 8, Figure 9) but the downstream variable that matters for end-to-end efficiency is generation length. The strong correlation between high recall (LessIsMore) and stable generation length (Table 1) versus low recall (TidalDecode, Quest) and inflated generation length (Table 1) suggests that recall is not just an accuracy proxy — it is an efficiency proxy as well. Methods that maintain high recall avoid the vicious cycle of re-derivation, making attention recall a more practically meaningful metric than prior work might have assumed.

Evidence anchor. Table 1 (generation length comparison across methods and budgets), the efficiency-accuracy tradeoff plot in Figure 7 (showing LessIsMore achieves higher accuracy at lower latency than TidalDecode across all budgets), and the cumulative attention recall curves in Figures 1 and 9 (showing LessIsMore's recall advantage widens over the course of long generations, consistent with the avoidance of accumulating errors that trigger re-derivation).

Innovation 4: The Global-vs-Local Selection Distinction as a First-Class Design Axis for Sparse Attention

The paper introduces a conceptual distinction that transcends its specific method: global token selection (aggregating information across attention heads before making selection decisions) versus local token selection (each head independently maximizing its own objective). This distinction, while simple in retrospect, had not been articulated as a primary design axis for sparse attention methods prior to this work. The paper demonstrates that on reasoning tasks, this design choice dominates other considerations — including the choice of approximation function f, the inter-layer selection architecture, and whether the method requires training.

Where prior methods sit on this axis. Eviction-based methods (StreamingLLM, H₂O, SnapKV, Keyformer) are globally-selective in the sense that tokens are evicted from the KV cache for all heads simultaneously — there is one cache, not one per head. However, they typically compute importance scores on a per-head basis and then aggregate (e.g., by summing or taking the maximum) to decide which tokens to evict. This aggregation step is global-selective only in implementation, not in principle — the design intent is still to approximate per-head importance. Selection-based methods (TidalDecode, Quest) are explicitly and intentionally local — each head selects its own tokens, and these selections may differ substantially across heads. Quest (Tang et al., 2024) even emphasizes this as a feature, arguing that different heads have different sparsity patterns.

What LessIsMore changes. By making cross-head aggregation the first step in selection (ranking tokens globally based on consensus across heads) rather than a post-hoc reconciliation of independent per-head decisions, LessIsMore inverts the priority: consensus is the primary signal, head-specific preferences are secondary. This is a design philosophy shift, not just an algorithm swap — it says that on reasoning tasks, the tokens that matter are those that matter to most heads, not those that matter most to any single head.

Why this distinction is productive. Framing the design space around the global-vs-local axis generates testable predictions. It predicts that methods closer to the local end (Quest, TidalDecode) will degrade more severely at low budgets than methods closer to the global end, because local methods risk duplicating budget across heads (selecting the same tokens multiple times or, equivalently, selecting different tokens such that the union exceeds the effective budget). It predicts that the global-local distinction becomes more important as the re-selection interval grows (i.e., when selection decisions must persist across more layers), because locally-optimal selections overfit to the specific query at the selection layer while globally-consensus selections generalize better — a prediction confirmed in Figure 8, where LessIsMore's advantage over head-to-head and randomized schemes is much larger under infrequent (L2 only) selection than under per-layer selection. It predicts that training-based methods (SeerAttention-r) will still underperform training-free global selection if the training objective reinforces local optimization rather than teaching global consensus — and Figure 6 shows exactly this, with SeerAttention-r underperforming LessIsMore at low budgets despite the training cost.

The generative implication. The global-vs-local axis suggests a new research direction orthogonal to the approximation function f: how should information be aggregated across heads before selection, not just how should attention scores be approximated within each head? This opens a design space of aggregation operators — voting, weighted averaging, attention-score-based fusion — that the current paper only begins to explore with its specific UnionFlatten and ranking approach. Future work could design learned aggregation functions, task-adaptive aggregation, or hierarchical aggregation that first groups heads by functional similarity before aggregating.

Evidence anchor. Figure 8 (ablation comparing LessIsMore's unified selection to head-to-head and randomized aggregation, with the gap substantially larger under infrequent selection), Figure 6 (LessIsMore outperforming both training-free local methods and training-required SeerAttention-r), and the spatial locality visualization in Figure 2 (direct evidence of cross-head overlap that motivates the global approach).

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use four reasoning benchmarks: AIME-24 and AIME-25 (competition mathematics problems from the American Invitational Mathematics Examination), GPQA-Diamond (graduate-level science questions; Rein et al., 2023), and MATH500 (a 500-question subset of the MATH benchmark; Hendrycks et al., 2021). The AIME tasks are the most challenging and produce the longest generation sequences (averaging 14K–20K tokens), while MATH500 and GPQA-Diamond involve shorter reasoning chains (4K–9K tokens on average).
Base model(s). The primary models are Qwen3-8B and Qwen3-4B (Team, 2025), both using Grouped Query Attention (GQA) — 32 query heads sharing 8 key-value heads for the 8B model. These models are "specifically trained for reasoning tasks and perform the effective thinking process by generating extensive tokens" (Section 4.1). The paper also evaluates on Llama-3.1-8B (AI, 2024) for latency measurements, chosen as a GQA-based model with the same parameter count as Qwen3-8B, enabling direct efficiency comparisons within a unified FlashInfer-based (Ye et al., 2024) implementation.
Metrics. The paper tracks three categories: accuracy (pass@1 averaged over multiple samples — 16 for AIME-24/25, 8 for MATH500 and GPQA-Diamond), attention recall (cumulative average of Equation 3 over generation steps, measuring what fraction of ground-truth attention mass the selected tokens capture), and efficiency (per-token decoding latency in milliseconds, measured on a single NVIDIA RTX A5000 GPU, plus end-to-end generation length in thousands of tokens). The accuracy metric uses the grading function from Lightman et al. (2022) for MATH problems and exact-match for the others, consistent with the benchmark conventions.
Baselines. Four comparison points are used: TidalDecode (Yang et al., 2024), the best-performing training-free selection-based method that LessIsMore builds upon — configured with the same two full attention layers and same selection layer indices (Layer 12 for Qwen3-8B, Layer 20 for Qwen3-4B) as LessIsMore; Quest (Tang et al., 2024), another training-free selection-based approach, configured with block size 64 and sparse attention at all layers per the settings from Gao et al. (2025); SeerAttention-r (Gao et al., 2025), a training-required sparse attention method specifically designed for reasoning, configured with block size 64 and all layers performing sparse attention; and Full Attention, the unmodified model with standard dense attention over the entire KV cache, serving as the accuracy upper bound. Quest and SeerAttention-r results use the reported sampling configurations from Gao et al. (2025).
Generation budget / compute accounting. The primary control variable is the token budget $K$ — the maximum number of KV cache entries that sparse attention layers may attend to. Budgets of 1K, 2K, 4K, and 6K tokens are evaluated, corresponding to different sparsity levels depending on generation length (e.g., a 2K budget on a 16K-token sequence represents 87.5% sparsity). The paper also measures and reports the static ratio $r = 0.25$ (fraction of $K$ reserved for recent tokens) and reserves 4 tokens for attention sinks in all experiments. For fair comparison, all methods use the same maximum generation length of 32,768 tokens across all tasks and models. Latency measurements normalize by generation length to produce per-token decoding speed.
Cross-validation / statistical protocol. There is no explicit cross-validation or statistical significance testing reported. Accuracy is averaged over multiple independent samples (16 per AIME problem, 8 per MATH500/GPQA-Diamond problem) to reduce sampling variance, but no confidence intervals or standard errors are reported for any accuracy or latency measurements. The selection of the optimal re-selection layer for TidalDecode/LessIsMore (Layer 12 vs. Layer 20) is calibrated via a needle-in-the-haystack test (Appendix A.1), but this tuning is done once per model and applied uniformly across all tasks.

Main Quantitative Results

Accuracy Across Reasoning Benchmarks (Figure 6)

The central accuracy experiment compares LessIsMore against Quest, TidalDecode, SeerAttention-r, and Full Attention across four benchmarks on two model sizes. The headline result is that LessIsMore consistently achieves the highest accuracy among all sparse methods, closely matching or equaling full attention even at the smallest token budgets, while all competitors suffer substantial degradation.

For Qwen3-8B on AIME-24: at the smallest budget of 2K tokens, LessIsMore achieves approximately 73–74% accuracy (near-lossless compared to full attention's ~74%), while Quest falls to roughly 30%, TidalDecode to roughly 53%, and SeerAttention-r to roughly 60%. At 4K budget, LessIsMore reaches full-attention accuracy while TidalDecode remains roughly 10 percentage points lower. At 6K budget, LessIsMore remains at or above full-attention performance while TidalDecode still trails. The full numerical values are reported in Appendix A.2 (Tables 2 and 3).

The same pattern holds across all benchmarks and both model scales. On AIME-25 with Qwen3-8B at 2K budget, LessIsMore substantially outperforms all baselines. On the easier MATH500 and GPQA-Diamond tasks (where generation lengths are shorter, 4K–9K tokens), LessIsMore achieves lossless accuracy at 1K token budgets where competitors again show clear degradation. On Qwen3-4B, the absolute accuracies are lower across all methods (the smaller model is less capable overall), but the relative pattern is identical: LessIsMore tracks full attention much more closely than any baseline.

A particularly striking pattern visible in Figure 6 is that the accuracy gap between LessIsMore and competitors widens as the token budget decreases — LessIsMore's advantage is largest precisely in the regime (small $K$ ) where efficient sparse attention is most valuable. At 6K budgets, several methods approach full-attention accuracy; at 2K budgets, only LessIsMore remains competitive with full attention.

Efficiency-Accuracy Tradeoff (Figure 7)

The efficiency evaluation measures per-token decoding latency on Llama-3.1-8B (same architecture as Qwen3-8B) using custom CUDA kernels within FlashInfer. The headline finding: LessIsMore achieves higher accuracy than TidalDecode at every token budget while simultaneously maintaining lower per-token latency, and at 2K budget achieves a 1.10× speed-up over full attention with near-lossless accuracy (73.75 vs. 74.48).

Figure 7 plots accuracy against average decoding latency for LessIsMore (orange squares), TidalDecode (blue circles), and Full Attention (triangle) at budgets of 1K, 2K, 4K, and 6K tokens. The "closer to top-left corner is better" framing shows LessIsMore points consistently above and left of TidalDecode points at corresponding budgets — meaning LessIsMore is simultaneously more accurate and faster.

Specific comparisons from Figure 7 and Table 1: at 2K budget, LessIsMore achieves 73.75% accuracy with a 1.10× speed-up over full attention (which achieves 74.48%). TidalDecode at the same 2K budget achieves only 53.33% accuracy. Even at 6K budget — double the tokens — TidalDecode reaches lower accuracy (the exact value is in Table 1) than LessIsMore at 2K, while generating 15.9K tokens vs. LessIsMore's 14.9K at 2K or 14.4K at 6K.

The end-to-end speed-up over TidalDecode is computed as the product of two factors: LessIsMore's 1.06× average per-token decoding speed-up at 6K vs. TidalDecode at 6K, and LessIsMore's 7% shorter generation length (14.4K vs. 15.9K tokens). This compounds to a 1.13× end-to-end speed-up — LessIsMore processes fewer tokens total and each token is processed faster.

The token index aggregation stage (the UnionFlatten and ranking operations from Algorithm 1) is implemented using PyTorch primitives rather than optimized CUDA kernels (Section 5.3), meaning the reported latency advantage is conservative — a fully kernel-optimized implementation would likely widen the gap.

Generation Length Analysis (Table 1)

Table 1 reports the average generation length (in thousands of tokens) alongside accuracy for each method and budget on AIME-24 with Qwen3-8B. The key finding: existing sparse attention methods substantially inflate generation length compared to full attention, while LessIsMore maintains generation lengths closely aligned with or slightly shorter than full attention.

Under the most restrictive 2K token budget, the generation lengths are: Full Attention 14.8K, Quest 30.0K (103% increase), SeerAttention-r 19.8K (34% increase), TidalDecode 17.4K (18% increase), and LessIsMore 14.9K (1% increase). The pattern of inflated generation length for baseline methods is interpreted as compensatory re-derivation — when important tokens are dropped, the model generates additional tokens to reconstruct the lost context.

At 4K budget, LessIsMore generates exactly 14.8K tokens — identical to full attention — while achieving the highest accuracy among all sparse methods. At 6K budget, LessIsMore generates 14.4K tokens, slightly shorter than full attention. Meanwhile, TidalDecode at 6K generates 15.9K tokens with lower accuracy, demonstrating that even with a generous budget, imprecise token selection produces both accuracy and efficiency penalties.

The practical consequence is that the raw per-token speed-up numbers understate LessIsMore's real efficiency advantage: baseline methods not only process each token slower (at equivalent budgets) but also generate more tokens total, compounding the latency penalty.

Ablation Studies and Robustness Checks

Effect of aggregation strategy on GQA models (Figure 8): This ablation tests whether LessIsMore's unified selection principle generalizes beyond the TidalDecode pipeline by applying three different token aggregation schemes — LessIsMore's unified top- $k$ with 25% recency window, Randomized Top- $k$ (randomly applying one query head's selection to the entire KV group), and Head-to-Head (per-head independent selection) — in a simplified setting on Qwen3-8B. When selection is applied at all decoding layers, all three schemes achieve relatively similar attention recall, with Randomized Top- $k$ being competitive. However, when selection frequency is reduced to only Layer 2 — a more efficient scenario that stresses generalization — LessIsMore's unified approach "significantly outperforms alternative aggregation schemes." The recall of Head-to-Head and Randomized Top- $k$ drops considerably under infrequent re-selection, while LessIsMore's recall remains high. This demonstrates that locally optimal per-head selections overfit to the specific query at the selection layer and do not generalize across subsequent layers, while unified global selection produces more robust token rankings.

Effect of recent window ratio on attention recall and answer correctness (Figure 9): This ablation sweeps the static ratio $r$ (fraction of budget allocated to recent tokens) across values of 0% (Unified Attention Head Selection only, no recency window), 25% (full LessIsMore design), 50%, 75%, and 100% (only recent tokens), plus TidalDecode as a baseline, on an AIME-24 problem with a 4K token budget and Qwen3-8B. The key findings: (1) 100% recent (pure sliding window, no historical selection) achieves the lowest attention recall, confirming that historical tokens are essential; (2) both TidalDecode and the 0% recent variant (unified selection without explicit recency) fail to produce the correct answer, even though the 0% variant improves recall over TidalDecode — explicit recency allocation is necessary for correctness; (3) all configurations with non-zero recency (25%, 50%, 75%) succeed in producing the correct answer; (4) among them, $r = 25\%$ achieves the highest cumulative attention recall throughout most of the generation process, validating this as the chosen ratio. The curves in Figure 9 are annotated with "(T)" or "(F)" to indicate whether the configuration yields the correct final answer.

Choice of re-selection layer (Appendix A.1): The optimal layer for token re-selection was determined via a needle-in-the-haystack test at 5K context length with a 256-token budget. For Qwen3-8B, Layer 12 achieved 86% accuracy. For Qwen3-4B, both Layer 12 (86%) and Layer 20 (84%) performed well; Layer 20 was chosen to demonstrate generalization across different selection layer choices. The paper notes that "in the same model family, the optimal re-selection layer is similar," so Layer 12 for Qwen3-8B and Layer 20 for Qwen3-4B were used for both TidalDecode and LessIsMore to ensure fair comparison.

Comparison against training-required methods (Figure 6): LessIsMore, which is training-free, is compared directly against SeerAttention-r, which requires post-training adaptation for reasoning tasks. Across all benchmarks and budgets, LessIsMore matches or exceeds SeerAttention-r's accuracy without any training cost. At the smallest budgets where the accuracy gap is largest (e.g., 2K on AIME-24 for Qwen3-8B), LessIsMore outperforms SeerAttention-r by approximately 13–14 percentage points. This is a meaningful negative result for the training-required approach: the paper's interpretation is that training-based adaptation, if it reinforces locally-optimal per-head selection rather than teaching global consensus, cannot overcome the fundamental misalignment that LessIsMore addresses structurally.

Generalization across model scales: The accuracy pattern in Figure 6 is replicated across Qwen3-8B and Qwen3-4B, with similar relative advantages for LessIsMore over baselines at both scales. This suggests the spatial and recency locality patterns are not specific to a particular model size within the Qwen3 family, though the paper does not test on models from other families (e.g., Llama, DeepSeek) or other attention architectures (MLA, MoE).

Critical Assessment

Claim: LessIsMore achieves lossless accuracy at substantially higher sparsity than prior methods.

Supported with qualifications. Figure 6 demonstrates that LessIsMore achieves accuracy comparable to full attention at 2K–4K budgets on AIME tasks and 1K budgets on MATH500/GPQA-Diamond, while all baselines (Quest, TidalDecode, SeerAttention-r) show clear degradation at these budgets. The "2× fewer tokens" claim from the abstract is validated by comparing LessIsMore at 2K budget to TidalDecode at 4K or 6K budget — LessIsMore matches or exceeds TidalDecode's accuracy while attending to half or fewer tokens.

However, the claim of "lossless" accuracy requires careful interpretation. The paper's figures do not report confidence intervals, so "lossless" means point estimates are within a few percentage points of full attention — not that the difference is statistically indistinguishable. At 2K budget on AIME-24 with Qwen3-8B, LessIsMore scores roughly 73–74% vs. full attention's ~74%, which is indeed nearly identical within reasonable sampling error. But this is based on 16 samples per problem on a 30-question benchmark — the effective sample size is modest, and small differences could be real but undetectable.

A more significant qualification: "lossless" is demonstrated only within the Qwen3 model family on a specific set of math and science reasoning benchmarks. Whether LessIsMore is lossless on, say, code generation reasoning (which also involves long chain-of-thought), or on models from other families (Llama, DeepSeek), or at even lower budgets (512 tokens), is entirely untested.

Claim: LessIsMore achieves a 1.1× average decoding speed-up over full attention.

Supported, but measured on a reference model, not the evaluation model. The latency measurements in Figure 7 use Llama-3.1-8B — not Qwen3-8B — as the measurement platform. The paper states this is "as a reference to the evaluated Qwen3-8B with the same parameter size and attention mechanism GQA." While the architectures are comparable, they are not identical (different training data, potentially different layer counts or hidden dimensions), and the speed-up on actual Qwen3-8B may differ. The speed-up is real and substantial, but the exact 1.10× figure is measured on a proxy model.

Additionally, the paper acknowledges (Section 5.3) that "the union operation in selection is implemented using PyTorch primitives rather than optimized CUDA kernels," meaning the reported speed-up is a lower bound — the selection overhead is not yet optimized. This is a strength (the method is already faster despite unoptimized components) but also means the 1.1× figure is not the final achievable speed-up. A countervailing concern: the paper does not report the prefill (input processing) latency, only decoding latency. Since LessIsMore uses full attention for the first two layers (and selection layers), the prefill phase is identical to full attention — the speed-up is purely in the decode phase. For reasoning tasks with short inputs and long outputs, this is the right metric, but for mixed workloads the effective speed-up would be lower.

Claim: LessIsMore achieves a 1.13× end-to-end speed-up over TidalDecode.

Supported, but computed from two quantities that are each subject to measurement variance. The 1.13× figure is the product of 1.06× per-token decoding speed-up (from Figure 7's latency measurements at the 6K budget) and 7% shorter generation length (from Table 1: LessIsMore generates 14.4K tokens vs. TidalDecode's 15.9K, a 9.4% reduction — the paper rounds this to 7%). The exact compounding depends on how prefill cost is factored in, which the paper does not detail. The qualitative conclusion (LessIsMore is meaningfully faster end-to-end than TidalDecode) is robust; the precise 1.13× multiplier should be treated as an estimate.

Claim: The generation length inflation observed in prior methods is caused by inaccurate token selection, and LessIsMore avoids it.

Supported by the data pattern, but the causal mechanism is inferred rather than directly tested. Table 1 shows a clear inverse correlation between attention recall (LessIsMore highest, Quest lowest) and generation length inflation (LessIsMore lowest, Quest highest), which is consistent with the "inaccurate selection → missing context → redundant re-derivation → longer generation" causal chain. However, the paper does not provide direct evidence of this mechanism — e.g., by showing that tokens generated during inflated-length sequences actually contain re-derived content that was present earlier in the context but not selected. The correlation is strongly suggestive but does not rule out alternative explanations (e.g., different methods might affect the model's "verbosity" through mechanisms unrelated to missing context).

Claim: LessIsMore's principles (unified selection, ratio-based recency) generalize across GQA-based architectures.

Partially supported. The ablation in Figure 8 demonstrates that the unified selection principle improves attention recall under infrequent re-selection compared to per-head or randomized aggregation schemes, even in a simplified setting outside the TidalDecode pipeline. This is evidence that the principle generalizes, but it is limited: the test still uses Qwen3-8B, still uses GQA, and still uses the same reasoning task. There is no evaluation on non-GQA architectures (e.g., standard multi-head attention, multi-query attention, or the MLA used in DeepSeek models), which limits the "generalize" claim to GQA-based models specifically.

What is missing — experiments that would strengthen the paper.

Standard errors or confidence intervals on all accuracy and latency measurements. The paper reports point estimates throughout but never quantifies uncertainty. For AIME-24 with 30 questions and 16 samples each, the standard error of the mean pass@1 is roughly $\sqrt{p(1-p)/n}$ where $n = 30 \times 16 = 480$ independent samples (though samples from the same question are not independent, so the effective sample size is smaller). Accuracies around 74% have a standard error of roughly 2%, meaning the "lossless" claims for differences of 1–2 percentage points are within noise.

Evaluation on a non-Qwen model family. All accuracy experiments use Qwen3-8B and Qwen3-4B. A single experiment on, e.g., Llama-3.1-8B (which is only used for latency) or DeepSeek-R1-Distill-Llama-8B (mentioned in the introduction as a motivating example but never evaluated) would substantiate the claim that LessIsMore's principles are task-general rather than model-specific.

Evaluation at extreme sparsity levels. The smallest budget tested is 1K tokens (for MATH500/GPQA-Diamond) and 2K tokens (for AIME). The paper claims "up to 87.5% sparsity" but does not push to see where LessIsMore breaks — does accuracy collapse at 512 tokens? At 256? Characterizing the failure boundary would be practically useful and would test whether the unified selection principle has a fundamental lower bound or degrades gracefully.

Direct measurement of redundant re-derivation. The generation length analysis would be stronger if accompanied by a qualitative or quantitative analysis showing that the extra tokens generated by baseline methods actually contain duplicated content, rather than simply being longer for other reasons (more verbose reasoning, different solution strategies, etc.).

Ablation on the number of full attention layers. LessIsMore inherits TidalDecode's "two full attention layers at the start" design. What happens with one full attention layer? With three? With full attention only at the very first layer? This ablation would clarify how essential the full-attention safety net is to the method's robustness.

Wall-clock measurements on the actual evaluation hardware and model. The latency experiments use Llama-3.1-8B rather than Qwen3-8B. Running the same latency benchmarks on Qwen3-8B would close this minor but noticeable gap between the accuracy evaluation platform and the efficiency evaluation platform.

6. Limitations and Trade-offs

1. The Static Ratio r = 0.25 Is Empirically Tuned on a Narrow Setting, Not Adaptively Determined

The assumption or constraint. LessIsMore hardcodes the fraction of the token budget allocated to recent tokens as $r = 0.25$ in all experiments (Section 4.1: "The static ratio $r$ of LessIsMore is set to $0.25$ "). This value is validated via a single-problem ablation on AIME-24 with Qwen3-8B at a 4K budget (Figure 9), where it achieves the highest cumulative attention recall among tested ratios and produces the correct answer. The paper explicitly acknowledges that this is not adaptive: "Currently, we employ a pre-defined token budget and a fixed 25% short-context ratio based on our observation of the consistent size of important recent context across reasoning tasks" (Section 5.1).

The deeper assumption is that the optimal ratio is stable across models, tasks, budget levels, and generation lengths. The ablation in Figure 9 tests $r \in \{0, 0.25, 0.5, 0.75, 1.0\}$ at a single budget (4K) and a single model (Qwen3-8B) on a single task (AIME-24). The paper does not evaluate whether $r = 0.25$ remains optimal at 2K budgets, 6K budgets, on Qwen3-4B, or on GPQA-Diamond and MATH500 — it is simply carried forward as a fixed constant.

The consequence. If the optimal recency ratio varies with task characteristics, model architecture, or budget level, then LessIsMore leaves accuracy on the table in those regimes — or worse, may actively harm performance by misallocating the budget. Consider two failure modes:

Over-allocation to recency on short-generation tasks. On MATH500, where average generation lengths are 4K–9K tokens (Section 4.2), a 1K token budget with $r = 0.25$ reserves 250 tokens for recency. If the "recent context" that matters for local reasoning continuity on these shorter problems is actually smaller (perhaps only 100 tokens), then 150 tokens of the historical budget are wasted — the model loses access to early problem context it genuinely needs.
Under-allocation to recency on extremely long generations. On AIME problems that generate 20K+ tokens, the "recent context" encompassing the last few reasoning steps might span considerably more than 25% of a small budget. At $K = 2000$ , the recency window is 500 tokens — if the model's current reasoning step depends on the last 800 tokens of context, critical intermediate results are dropped.

A subtler concern: the recency ratio interacts with the generation length behavior documented in Table 1. If a particular problem triggers longer-than-average generation (even under LessIsMore), the recency window's absolute size remains fixed (tied to the budget $K$ ), while the amount of context that is "recent" relative to the current reasoning step may grow. A ratio that is optimal at the average generation length may be suboptimal at the tails.

What evidence exists in the paper. Figure 9 provides the only direct evidence, showing that at $K = 4000$ on one AIME-24 problem, $r = 0.25$ achieves higher recall than $r = 0.5$ or $r = 0.75$ , and that $r = 0$ (no recency) fails to produce the correct answer. This is solid evidence that some recency allocation is necessary and that 25% is better than 50% or 75% in this specific setting. It does not establish that 25% is optimal across settings, nor does it characterize the sensitivity of accuracy to small perturbations around $r = 0.25$ (e.g., $r = 0.15$ vs. $r = 0.35$ ). The full benchmark results in Figure 6 use $r = 0.25$ throughout without any per-task or per-budget tuning — so we cannot tell whether some of the remaining accuracy gap to full attention at the smallest budgets could be closed by adjusting $r$ .

Mitigation status. The paper partially acknowledges this limitation in Section 5.1 ("Adaptive Short-Context Ratio") and suggests that "it would be optimal to determine this ratio adaptively during generation," citing top-p sampling-inspired approaches (Chen et al., 2024; Lin et al., 2025) as possible solutions. However, no adaptive method is developed, evaluated, or even sketched in algorithmic terms. The ratio remains a statically-tuned hyperparameter that a practitioner deploying LessIsMore on a new model or task would need to re-calibrate, with no guidance beyond "run an ablation like Figure 9." Given that the ablation requires ground-truth answer evaluation to determine which configuration "succeeds in solving the task" (the (T)/(F) annotations in Figure 9), this calibration itself requires a labeled benchmark — circular in deployment.

2. Difficulty Estimation Is Implicit and Uniform: All Problems Get the Same Budget, Regardless of Reasoning Depth

The assumption or constraint. LessIsMore applies a single, globally-fixed token budget $K$ to all problems in a benchmark. There is no mechanism for estimating a problem's difficulty or required reasoning depth and adjusting $K$ accordingly. An AIME geometry problem that requires a 20K-token derivation and a GPQA Diamond biology question that requires a 4K-token answer both receive the same $K$ in their respective evaluations — but more critically, within the AIME-24 benchmark, all 30 problems receive the same $K$ , even though some may require much longer reasoning chains than others.

This is not an oversight specific to LessIsMore — it is the standard evaluation protocol for sparse attention methods (TidalDecode, Quest, and SeerAttention-r are all evaluated identically). But it constitutes an implicit assumption: that uniform sparsity is the right resource allocation strategy across problems of varying difficulty. The paper's own analysis (Section 2.2.2, Figure 3) shows that the recency window size that matters is stable in proportion to the total context — but the total context varies dramatically across problems. A problem that naturally requires 6K tokens of reasoning and one that requires 20K tokens of reasoning have very different context scales, yet both are capped at the same absolute budget $K$ .

The consequence. A uniform budget creates two failure modes that pull in opposite directions:

Over-sparsification of hard problems. The hardest AIME problems (which this paper does not separate by difficulty within AIME) may require maintaining much more context than easier ones. A 2K budget on a 20K-token problem forces 90% sparsity, which may be below the threshold needed to preserve all necessary intermediate results. The paper reports average accuracy across all problems in each benchmark but does not break out performance by problem difficulty — so we cannot tell whether LessIsMore's near-lossless accuracy at 2K budget is achieved by maintaining near-perfect accuracy on medium problems while silently failing on the hardest ones (pulling the average up through easy problems where 2K is generous). The difficulty-binned analysis from the reference example paper (Snell et al., 2024) — where bin 5 problems showed near-zero improvement regardless of test-time compute — is exactly the kind of diagnostic that is absent here.
Under-sparsification of easy problems. Conversely, problems that require short reasoning chains are given more budget than they need, leaving efficiency gains on the table. A MATH500 problem that the model can solve in 2K tokens receives the same $K$ budget as one requiring 8K tokens. The fixed budget means LessIsMore is not compute-optimal in the sense of matching resource allocation to problem difficulty — it is uniform-optimal (finding the best uniform $K$ across the benchmark), which is a weaker guarantee.

What evidence exists in the paper. None directly. The paper reports only aggregate accuracy per benchmark, not per-problem accuracy or accuracy conditioned on generation length. The generation length analysis in Table 1 reports average generation lengths — LessIsMore averages 14.9K tokens at $K = 2000$ — but does not report the distribution (standard deviation, percentiles), which would reveal whether some problems balloon in length while others stay compact. A high variance in generation lengths under sparse attention would indicate that some problems are being forced into inefficient re-derivation while others are unaffected — the very problem-dependence the uniform budget ignores. The paper's efficiency-accuracy tradeoff curve (Figure 7) shows only aggregate points, not per-problem or per-difficulty breakdown.

Mitigation status. Not addressed. The paper's future work section (5.1) discusses adaptive ratio determination for the recency window but does not discuss adaptive budget determination across problems. A natural extension — which the paper does not mention — would be to estimate required context size from the problem statement or early decoding behavior and scale $K$ accordingly, directly analogous to the difficulty-conditioned test-time compute allocation in the reference example. The fact that this is not discussed, even as future work, suggests the paper views uniform budgeting as an acceptable evaluation protocol rather than a limitation to be overcome.

3. All Accuracy Evaluations Are on a Single Model Family (Qwen3) With No Demonstration of Cross-Architecture Generalization

The assumption or constraint. Every accuracy experiment in Section 4.2 and Appendix A.2 uses Qwen3-8B or Qwen3-4B — two models from the same family (Team, 2025), sharing the same architecture (GQA with 32 query heads and 8 KV heads), the same training data distribution, and the same reasoning-focused training procedure. The latency experiments use Llama-3.1-8B, but only as a measurement platform for speed comparisons — accuracy on Llama is never reported. The paper hypothesizes that the spatial and recency locality patterns it exploits are universal properties of reasoning tasks, not artifacts of Qwen3's specific training: "these localities are fundamental properties of the reasoning attention distribution that can be exploited without learning" (implied by the paper's position, Section 1 and the training-free nature). But this hypothesis is tested only within Qwen3's attention distribution.

The paper's observation section (2.2) uses Qwen3-8B exclusively — Figures 2 and 3 visualize attention patterns on this single model. The cross-head overlap pattern (spatial locality) is demonstrated on Qwen3-8B's 32 attention heads; the recency locality is demonstrated on Qwen3-8B's decoding steps. The paper's entire design rationale — unified selection because heads overlap, ratio-based recency because the recent band is stable — is derived from data collected on one model. If Qwen3 happens to have unusually high cross-head attention consensus compared to other reasoning models, LessIsMore's advantage would be exaggerated on the evaluated benchmarks and might not transfer.

The consequence. A practitioner reading this paper and wanting to deploy LessIsMore on, for example, DeepSeek-R1-Distill-Llama-8B (the model mentioned in the introduction as a motivating example for the computational cost problem) has no direct evidence that LessIsMore works on that model. The failure modes are multiple:

Different attention architectures. DeepSeek models use Multi-head Latent Attention (MLA), which compresses the KV cache into a latent space — a fundamentally different attention mechanism from GQA. The spatial locality observation (cross-head overlap in token-level top-k selections) is physically meaningful in GQA where query heads within a KV group attend to the same keys; in MLA, the relationship between query heads and the compressed latent KV representation is different, and the concept of "cross-head token overlap" may not even be well-defined.
Different training procedures. Qwen3 models are "specifically trained for reasoning tasks" (Section 4.1). Models trained with different objectives — e.g., general-purpose models fine-tuned for reasoning via RL (like DeepSeek-R1) versus reasoning-native models (like Qwen3) — may have different attention distributions. A model that learned reasoning as a post-training adaptation rather than a native capability might exhibit less cross-head consensus, making unified selection less effective.
Different scales. The paper evaluates 4B and 8B models. At larger scales (30B, 70B, 405B), attention head specialization may be more pronounced (larger models have more capacity for specialized functions), potentially reducing cross-head overlap and weakening LessIsMore's advantage over per-head methods. Conversely, at very small scales (1B), attention heads may be less specialized to begin with, making cross-head overlap high but also making per-head methods less harmful — potentially narrowing the gap.

What evidence exists in the paper. The Qwen3-4B results in Figure 6 provide a within-family scale comparison — LessIsMore maintains its advantage at both 4B and 8B sizes, showing the principles are not specific to one model capacity. The GQA aggregation ablation in Figure 8 demonstrates that the unified selection principle works for the specific GQA configuration of Qwen3-8B and outperforms alternative aggregation strategies. No cross-family data exists. The paper's Section 5.2 ("Generalization of LessIsMore") explicitly flags this: "future work should prioritize extending LessIsMore's locality-based principles to models beyond GQA, such as MLA or MoE architectures that employ different attention mechanisms."

Mitigation status. The paper is transparent about this limitation in Section 5.2, but the transparency does not mitigate the limitation — it merely acknowledges it. The claim in the abstract ("we introduce LessIsMore, a training-free sparse attention mechanism for reasoning tasks") and introduction is stated without qualification to GQA-based models, which overstates the demonstrated scope. A practitioner reading only the abstract and introduction would reasonably assume the method works on "reasoning tasks" broadly; only in Section 5.2 is the architecture constraint clarified.

4. The Token Selection Overhead Is Implemented in PyTorch Primitives, Partially Offsetting the Reported Speed-ups in Deployment

The assumption or constraint. LessIsMore's reported speed-ups (1.1× over full attention, 1.13× end-to-end over TidalDecode) are measured with the token index aggregation stage (the UnionFlatten, global sorting, and pruning operations from Algorithm 1 lines 12–14) implemented in PyTorch rather than optimized CUDA kernels. The paper states this explicitly: "The token index aggregation stage illustrated in Figure 5 is implemented using PyTorch primitives" (Section 4.3), and again in limitations: "The union operation in selection is implemented using PyTorch primitives rather than optimized CUDA kernels" (Section 5.3).

The reported latency numbers therefore include an unoptimized component whose cost scales with $H \cdot K \cdot (1 - r)$ (the size of the flattened per-head selections) — which, for $H = 32$ heads and $K \cdot (1-r) = 3000$ tokens at a 4K budget, means sorting roughly 96,000 indices at each selection layer, for each decoding step. In a pure-PyTorch implementation, this CPU-GPU synchronization and host-side sorting could be a significant fraction of the per-token latency, especially for small models where the attention computation itself is fast.

The consequence. There are two interpretations of this limitation, pulling in opposite directions:

Optimistic interpretation: The reported speed-ups are a lower bound. If the selection aggregation were implemented as fused CUDA kernels (operating entirely on GPU, without PyTorch overhead), the per-token latency would decrease further, and the speed-up over full attention and TidalDecode would be larger than currently reported. This is the framing the paper suggests (Section 5.3: "system-level optimizations in LessIsMore remain unexplored that could translate token savings into substantial memory savings").
Pessimistic interpretation: The reported speed-ups include an unknown amount of PyTorch overhead that may not be representative of actual optimized deployment. A naive PyTorch torch.topk followed by CPU-side list operations could be artificially slow, making TidalDecode look worse and the full attention baseline look relatively more expensive than they would be in a fully-optimized implementation of each method. If TidalDecode's selection logic is already kernel-optimized (it was published with custom CUDA support in FlashInfer; Yang et al., 2024), then the 1.06× per-token speed-up of LessIsMore over TidalDecode may partly reflect the optimization gap between TidalDecode's optimized selection and LessIsMore's unoptimized selection — the "true" speed-up at equal optimization levels could be smaller.

The practical consequence for a practitioner: the paper provides no guidance on the engineering effort required to realize the reported speed-ups in a production deployment. Implementing efficient CUDA kernels for the UnionFlatten + sort + prune pipeline is non-trivial — it involves managing GPU memory for variable-sized per-head output lists, avoiding atomic operations on the global ranking, and potentially fusing the sort with the subsequent KV cache gather operation. The paper's code release (GitHub link in the abstract) may or may not include these optimizations, but the paper text does not report speed-ups using optimized kernels.

What evidence exists in the paper. Figure 7 reports per-token latency measurements comparing LessIsMore, TidalDecode, and Full Attention on Llama-3.1-8B within FlashInfer. The paper does not break down latency by component (selection overhead vs. attention computation vs. FFN), so a reader cannot assess what fraction of the total latency is attributable to the unoptimized selection stage. The paper also does not compare against a hypothetical version of TidalDecode with similarly unoptimized selection (though TidalDecode's selection is simpler — no cross-head aggregation — so the overhead would be lower regardless). The 1.06× per-token speed-up over TidalDecode at 6K budget is small enough that optimization-level differences could explain part of it.

Mitigation status. The paper acknowledges this as a limitation (Section 5.3: "Currently, several implementation optimizations limit the full potential of LessIsMore") and frames it as future engineering work. No optimized kernel implementation is provided or benchmarked. A practitioner implementing LessIsMore from the paper's description would need to invest significant engineering effort to reproduce the reported latencies, and might achieve different results depending on their CUDA optimization skill.

5. LessIsMore Inherits and Does Not Ablate the TidalDecode Backbone Design Choices, Leaving Key Structural Hyperparameters Unexplored

The assumption or constraint. LessIsMore is explicitly built on TidalDecode's pipeline: "LessIsMore adopts TidalDecode ... as a backbone; specifically, it starts with two full attention layers, includes two dedicated token selection layers, and employs sparse attention in the remaining layers" (Section 3). The selection layers are placed at specific indices (Layer 12 for Qwen3-8B, Layer 20 for Qwen3-4B) determined by a needle-in-the-haystack calibration on TidalDecode (Appendix A.1: "With a token budget of 256, Layer 12 achieves the best accuracy of 86% on Qwen3-8B"). The paper does not ablate any of these inherited design choices in the context of LessIsMore — no experiments vary the number of full attention layers, the number of selection layers, or the placement of selection layers.

The paper attributes LessIsMore's accuracy gains entirely to the two novel mechanisms (Unified Attention Head Selection and Stable Recency Window), implicitly treating the TidalDecode backbone as a neutral substrate. But the backbone's design choices interact with the novel mechanisms: unified selection may be more or less beneficial depending on how many layers share each selection, and the recency window's importance may depend on how far apart selection layers are placed.

The consequence. Several practically important design decisions are left to guesswork:

How many full attention layers are necessary? The paper uses two at the start plus the selection layers. Could one suffice? Would three improve accuracy at the cost of speed? The "safety net" intuition (Section 3's description) says full attention layers prevent error propagation, but the strength of that safety net is untested. A deployment that can tolerate a small accuracy loss in exchange for higher speed might want to reduce full attention layers; the paper provides no guidance on this tradeoff.
Is Layer 12 (for Qwen3-8B) optimal for LessIsMore specifically, or only for TidalDecode? The selection layer was tuned on TidalDecode's needle-in-the-haystack task (Appendix A.1), not on LessIsMore's reasoning benchmarks. Since LessIsMore's unified selection produces different (and more robust) token rankings than TidalDecode's per-head selection, the optimal re-selection frequency and placement might differ. LessIsMore's selections are claimed to generalize better across subsequent layers (Figure 8), which might mean selection could be done less frequently — saving the cost of full attention at selection layers — without accuracy loss.
Could LessIsMore work with only one selection layer? Appendix A.1 tests TidalDecode with a single re-selection layer and finds Layer 12 optimal. The paper does not test whether LessIsMore's more robust selections could eliminate the need for a second selection layer entirely, which would improve speed by removing one full-attention computation per decoding step.
The 4 attention sink tokens are fixed but never ablated. The paper reserves exactly 4 tokens for attention sinks (Section 4.1) without testing whether this number matters. If the model's attention sink pattern uses a different number of initial tokens (some models show a single dominant sink token; others distribute sink behavior across multiple early tokens), the fixed reservation of 4 might be either wasteful or insufficient.

What evidence exists in the paper. The needle-in-the-haystack calibration in Appendix A.1 provides evidence that Layer 12 is an important layer for Qwen3 models, but this evidence is collected on TidalDecode with a 256-token budget on a retrieval-style task — not on LessIsMore on reasoning tasks. The paper's primary evaluation in Figure 6 uses this inherited configuration without questioning it. The ablation in Figure 8 tests selection frequency (All layers vs. Layer 2 only), but does so in a simplified setting that does not match the full LessIsMore pipeline (it evaluates aggregation schemes directly, not the complete TidalDecode backbone with LessIsMore's selection logic). The selection layer count and placement are never ablated.

Mitigation status. Not addressed. The paper does not acknowledge the backbone design choices as limitations or suggest future ablation work on them. A practitioner deploying LessIsMore would need to either trust the inherited TidalDecode configuration or run their own (potentially expensive) hyperparameter sweep over full attention layer count, selection layer count, and selection layer placement — with no guidance from the paper on expected sensitivity.

6. The Paper Does Not Evaluate on Non-Mathematical Reasoning Tasks, Leaving Domain Generality Untested

The assumption or constraint. All four evaluation benchmarks are in the mathematical or scientific reasoning domain: AIME-24 (competition mathematics), AIME-25 (competition mathematics), MATH500 (competition mathematics), and GPQA-Diamond (graduate-level biology, physics, chemistry). These tasks share structural properties: they involve symbolic manipulation, formal logical derivation, and exact-answer verification. The paper's motivating observations about attention localities (Figures 2 and 3) are collected on AIME-24 — a math task — and the entire design rationale (cross-head consensus because logical steps are broadly relevant; recency locality because each step builds on the last) is argued to apply to reasoning generally, but is only demonstrated on math and science.

The paper does not evaluate on code generation reasoning (e.g., HumanEval, MBPP, LiveCodeBench), commonsense reasoning (e.g., StrategyQA, Date Understanding), multi-hop question answering (e.g., HotpotQA), or any non-English reasoning task. This is a significant scope limitation because different reasoning domains may exhibit different attention patterns:

Code generation involves maintaining variable bindings, function definitions, and control flow context — the "recency" that matters might be the last function definition or the current loop scope, which could be arbitrarily far back in the context if the model is implementing a function defined at the start of the file. The ratio-based recency window, validated on math where "recent" means the last few equations, might misallocate budget for code tasks where "recent" in token space is not "recent" in logical dependency space.
Multi-hop reasoning over documents requires integrating information from multiple passages that may be far apart in the input context, followed by a relatively short reasoning chain. The cross-head consensus pattern might differ because the attended tokens are drawn from the input (which is long and static) rather than from the generation (which is long and growing). LessIsMore's design — which separates the budget into historical and recent tokens — implicitly assumes the important context is in the generation; for tasks where the input is the primary source of critical information, the input tokens might need a different budget allocation than "historical" tokens.

The consequence. A practitioner deploying LessIsMore on a code generation model (e.g., using DeepSeek-Coder or Qwen-Coder for repository-level code synthesis) has no evidence that the 1.1× speed-up with lossless accuracy transfers. The failure mode is not just lower accuracy — it is that LessIsMore's specific budget allocation (75% historical tokens selected by cross-head consensus, 25% recent tokens) might be structurally mismatched to the attention patterns of code tasks, where the distinction between "recent" and "historically important" does not align with the distinction between "current reasoning step" and "earlier reasoning steps." A poorly-matched budget allocation could simultaneously degrade accuracy and increase generation length (as the model compensates for missing context), yielding worse performance than full attention while still incurring the implementation complexity of sparse attention.

What evidence exists in the paper. None beyond the math/science benchmarks. The paper's title and abstract claim applicability to "reasoning tasks" without qualification, and the introduction discusses reasoning models broadly (Section 1 mentions "complex reasoning tasks" and lists DeepSeek-R1, Gemini-2.5-pro, OpenAI-o3, Qwen3, gpt-oss as examples — several of which are general-purpose reasoners, not math specialists). The empirical scope does not match the claimed scope. The paper's future work section (5.2) discusses architecture generalization (MLA, MoE) but does not discuss task domain generalization.

Mitigation status. Not addressed. The paper does not acknowledge the task domain limitation as such — it treats "reasoning tasks" as a coherent category defined by the benchmarks it evaluates, rather than as a broader category of which mathematical reasoning is one important but specific subtype. A practitioner reading the paper should understand that LessIsMore is validated on mathematical and scientific reasoning specifically, and that its applicability to other reasoning domains (code, commonsense, multi-hop QA, logical deduction) is an open question.

7. Implications and Future Directions

How This Work Changes the Landscape

LessIsMore is not a paradigm shift in the sense of introducing a new attention mechanism or a new training objective. It does not propose a fundamentally different way to compute attention, nor does it claim to replace dense attention entirely. Its contribution is more precise and, in certain respects, more actionable: it identifies a specific, correctable assumption in the dominant design paradigm for sparse attention — that attention heads are functionally specialized and require independent token subsets — and demonstrates that on reasoning tasks, correcting this assumption via cross-head consensus and proportional recency budgeting is sufficient to recover lossless accuracy at substantially higher sparsity than was previously thought achievable without training.

This is a diagnostic reframing with an algorithmic consequence. The diagnostic component is the observation (Figures 2 and 3) that reasoning attention distributions exhibit spatial locality across heads (consensus in token importance rankings) and recency locality across steps (stable proportional importance of recent tokens). The algorithmic consequence is LessIsMore's two mechanisms — Unified Attention Head Selection and Stable Recency Window — which directly operationalize these observations. Neither mechanism is individually complex, but together they invert the selection priority from "each head independently maximizes its own recall" to "the system maximizes global recall, and heads share the result."

The practical significance of this reframing is that it reopens the design space for training-free sparse attention on reasoning tasks, which had appeared to be hitting a wall. Prior to LessIsMore, the trajectory of the field suggested that training-free methods were insufficient for reasoning: TidalDecode required sparsity below 50% to preserve accuracy on AIME-24 (Section 1), Quest suffered even larger degradation (Figure 6), and the state-of-the-art for reasoning-specific sparse attention was a training-required method (SeerAttention-r). The implicit narrative was that reasoning's strict logical consistency requirements demanded learned adaptation — that no static selection heuristic could be robust enough. LessIsMore falsifies this narrative. It demonstrates that the failure was not inherent to training-free sparsity, but specific to the per-head local optimization assumption that previous methods inherited from the broader sparse attention literature without questioning whether it applied to reasoning.

This finding has several downstream effects on the research landscape:

It redirects research attention from approximation functions to aggregation strategies. Prior work on sparse attention predominantly focused on improving the per-head approximation function f — better ways to estimate which tokens each head will attend to without computing full attention (Quest's chunked estimates, TidalDecode's position-persistent selections, RetrievalAttention's vector search). LessIsMore's results suggest that on reasoning tasks, the choice of aggregation strategy (how information is combined across heads before selection) dominates the choice of approximation function. This is a reorientation of the research agenda: the next generation of sparse attention methods for reasoning should prioritize cross-head information sharing, not better per-head scoring heuristics.

It establishes attention recall as not just an accuracy proxy but an efficiency proxy. The paper demonstrates (Table 1, Figure 7) that higher attention recall correlates with shorter generation lengths — when the model reliably finds the context it needs, it does not waste tokens re-deriving lost information. This transforms attention recall from a diagnostic metric into a leading indicator of end-to-end efficiency, which has implications for how sparse attention methods should be designed and evaluated. A method that achieves high accuracy but inflated generation lengths (like Quest at 2K budget generating 30K tokens vs. full attention's 14.8K) may have worse end-to-end latency than a method with slightly lower per-token accuracy but stable generation lengths. Future evaluations of sparse attention should report the accuracy–latency–length trilemma explicitly, as LessIsMore's Figure 7 and Table 1 together do.

It resolves the apparent contradiction between retrieval and reasoning sparse attention performance. Prior work showed methods like TidalDecode achieving >99.9% sparsity with no accuracy loss on retrieval tasks (Yang et al., 2024) while collapsing on reasoning tasks. One interpretation was that reasoning is fundamentally harder — that the longer generation sequences make error accumulation inevitable. LessIsMore demonstrates that the collapse is not inevitable: it is a consequence of per-head selection producing locally-optimal but globally-fragile token rankings that degrade over long sequences. The unified selection approach maintains high recall over 32K-token generations (Figure 9), showing that the fundamental limit is not sequence length per se, but the mismatch between the selection mechanism and the attention structure. This reconciliation is valuable because it converts a discouraging empirical result ("sparse attention doesn't work for reasoning") into a tractable design problem ("sparse attention works for reasoning when the selection mechanism respects cross-head consensus and recency proportionality").

It makes training-free sparse attention competitive with — and in key regimes superior to — training-required methods. The comparison against SeerAttention-r in Figure 6 is striking: LessIsMore, with no training, consistently matches or exceeds the accuracy of a method that requires post-training adaptation specifically for reasoning. At the lowest budgets (2K on AIME-24), LessIsMore outperforms SeerAttention-r by approximately 13–14 percentage points. This does not mean training is useless — SeerAttention-r at higher budgets approaches full-attention accuracy — but it does mean that the structural prior encoded in LessIsMore (cross-head consensus + ratio-based recency) captures something essential about reasoning attention that training-based adaptation, if it reinforces per-head local optimization, does not learn. Future training-based methods should consider incorporating cross-head aggregation objectives into their training loss, rather than treating per-head optimization as the default.

Less attractive research directions. Several lines of work become less promising in light of LessIsMore's results. (1) Purely per-head approximation function improvements — better ways to score tokens within a single head — are unlikely to close the gap on reasoning tasks if the structural problem is that per-head scores are noisy and don't generalize across layers. The ablation in Figure 8 shows this directly: when selection must persist across many layers (L2 only), head-to-head and randomized schemes collapse while unified selection remains robust. Better per-head scoring cannot fix this overfitting problem. (2) Eviction-based methods that permanently discard tokens face an uphill battle on reasoning tasks, because Figure 1(a) shows their recall degradation is even more severe than selection-based methods on long generations — the permanent information loss is fundamentally at odds with the need to access arbitrary earlier reasoning steps. LessIsMore's selection-based design (retaining the full KV cache) is likely the right architectural choice for reasoning, and future work on reasoning-specific sparse attention should focus on improving selection rather than revisiting eviction. (3) The paper incidentally demonstrates that simply adding a fixed-size sliding window (the StreamingLLM approach) is insufficient — Figure 9 shows that 100% recent (pure sliding window) achieves the lowest recall of any configuration, confirming that historical context cannot be ignored for reasoning.

Follow-Up Research This Work Enables

Adaptive ratio determination using attention score distribution statistics. The paper identifies the static ratio r = 0.25 as a limitation (Section 5.1) and gestures at top-p sampling-inspired approaches. A concrete follow-up would design and evaluate an adaptive ratio mechanism that analyzes the distribution of attention scores at each selection layer and dynamically sets r based on what fraction of the total attention mass falls on the most recent tokens. For example: compute the cumulative attention mass on the last 1%, 5%, 10%, 25%, and 50% of the context at the selection layer; if the last 10% captures >40% of total attention mass, allocate r = 0.25; if it captures >60%, allocate r = 0.4; if it captures <20%, allocate r = 0.1. This would be evaluated on AIME-24 and GPQA-Diamond against the static r = 0.25 baseline, measuring both accuracy and generation length. The key question: does adaptive ratio determination close the remaining accuracy gap to full attention at the smallest budgets (1K–2K), and does it reduce generation length variance across problems? A negative result — adaptive ratio determination performs no better than static r = 0.25 — would suggest that the optimal ratio is genuinely stable across generation contexts and that the static choice is sufficient, which would be a practically useful finding in itself.

Difficulty-conditioned token budget allocation. LessIsMore currently applies a uniform token budget K to all problems in a benchmark. A natural extension, directly inspired by compute-optimal test-time scaling frameworks, would estimate problem difficulty from early decoding behavior and allocate K accordingly. The concrete experiment: on AIME-24, run LessIsMore with K = 2048 for the first 512 generated tokens, then use the average attention recall over those 512 tokens as a difficulty signal. If recall is high (>0.95), the problem is "easy" — the model is finding the context it needs in 2K tokens, and K can remain at 2K. If recall is moderate (0.85–0.95), increase K to 4K for the remainder of the generation. If recall is low (<0.85), increase K to 6K. Compare this adaptive-budget strategy against the best uniform K on AIME-24, measuring the tradeoff between total FLOPs (which accounts for the variable budget across problems) and accuracy. The hypothesis is that adaptive budgeting achieves higher accuracy than any uniform budget at equivalent average FLOPs, because hard problems get the extra context they need while easy problems don't waste compute on unnecessary tokens. A negative result — adaptive budgeting provides no benefit over the best uniform K — would indicate that LessIsMore's selection is already extracting nearly all available information at 2K budgets, and that additional budget goes primarily to tokens the model doesn't actually need, which would be a strong finding about the sufficiency of cross-head consensus selection.

Cross-architecture validation on DeepSeek's Multi-head Latent Attention (MLA). The paper acknowledges (Section 5.2) that all evaluations are on GQA models, and specifically calls out MLA as a target for future work. A concrete follow-up would replicate the full Figure 6 experiment (LessIsMore vs. TidalDecode vs. Quest vs. Full Attention across AIME-24, MATH500, GPQA-Diamond) on DeepSeek-R1-Distill-Llama-8B or DeepSeek-V2-Lite, both of which use MLA. The key adaptation challenge: MLA compresses the KV cache into a latent space, so the concept of "token-level top-k selection" must be translated — does LessIsMore operate on the latent representation, or on the reconstructed keys? The experiment would answer two questions simultaneously: (1) Do the spatial and recency locality patterns that LessIsMore exploits generalize to MLA's compressed attention representation, or are they artifacts of GQA's explicit key-value structure? (2) If the patterns do generalize, does LessIsMore's training-free unified selection outperform the training-required adaptation that SeerAttention-r requires? A positive result on MLA would substantially strengthen the paper's universality claim and open deployment on the DeepSeek model family, which dominates current reasoning leaderboards. A negative result — LessIsMore fails to transfer, with significant accuracy degradation even at high budgets — would precisely characterize the boundary of the method's applicability and motivate architecture-specific designs for MLA.

Ablation on the number and placement of full attention layers. LessIsMore inherits TidalDecode's "two full attention layers + two selection layers" design without ablating it. A systematic sweep would evaluate LessIsMore with {1, 2, 3, 4} initial full attention layers and {1, 2, 3} selection layers (placed at the optimal positions determined by the needle-in-the-haystack calibration), measuring the accuracy-efficiency Pareto frontier on AIME-24 with Qwen3-8B at K = 2048 and K = 4096. The hypothesis is that LessIsMore's more robust unified selections reduce the need for frequent re-selection: a single selection layer might suffice, saving the computational cost of additional full-attention passes. A finding that one full attention layer + one selection layer achieves nearly the same accuracy as the two-plus-two configuration would make LessIsMore substantially faster (reducing the full-attention overhead) and would demonstrate that the unified selection principle's robustness translates into architectural simplicity. Conversely, a finding that accuracy degrades sharply with fewer full attention layers would indicate that the full-attention safety net is doing more work than the paper's narrative suggests — that unified selection alone cannot fully compensate for the absence of dense attention at early layers.

Domain transfer: code generation reasoning. While the paper's evaluation is confined to mathematical and scientific reasoning, the motivating problem — decode-heavy generation with long chain-of-thought — applies equally to code generation, where models produce extensive reasoning before writing code. A concrete evaluation would apply LessIsMore to Qwen3-Coder (or another code-specific reasoning model) on LiveCodeBench or BigCodeBench, measuring pass@1 and generation length against full attention and TidalDecode at K ∈ {2048, 4096, 8192}. The key question is whether the spatial locality pattern (cross-head consensus in token importance) observed in math reasoning holds for code reasoning, where the "important context" might include function signatures, variable definitions, and API documentation that are textually distant from the current generation point but logically critical. If cross-head consensus is weaker for code (because different heads attend to different structural aspects of code — syntax vs. semantics vs. documentation), LessIsMore's unified selection might underperform per-head methods, precisely characterizing a domain boundary for the method. If cross-head consensus is equally strong, it strengthens the argument that the pattern is a general property of step-by-step reasoning, not a math-specific artifact.

Training-based integration: adding cross-head consensus objectives to sparse attention fine-tuning. LessIsMore demonstrates that a training-free heuristic exploiting cross-head consensus outperforms a training-required method (SeerAttention-r) that does not. This suggests a hybrid: fine-tune a model for sparse attention, but include an auxiliary loss that explicitly penalizes divergence in token importance rankings across heads. For example, during the fine-tuning process, compute a pairwise ranking consistency loss (e.g., Kendall's tau) between the top-k token rankings of different heads, and add this to the primary language modeling loss. This would encourage the model to learn attention patterns that are naturally amenable to unified selection, potentially achieving even higher sparsity than LessIsMore's training-free approach while maintaining accuracy. The evaluation would compare against both LessIsMore (training-free) and SeerAttention-r (training without cross-head objective) on AIME-24 across a range of token budgets. A positive result — the hybrid outperforms both LessIsMore and SeerAttention-r — would demonstrate that LessIsMore's structural insight can be productively combined with training-based adaptation, rather than being an alternative to it.

Practical Applications and Downstream Use Cases

Batch inference for reasoning model evaluation and data generation. Organizations that evaluate reasoning models at scale — running AIME, MATH, or GPQA benchmarks across multiple model checkpoints, or generating reasoning traces for distillation and fine-tuning datasets — face a direct tradeoff between accuracy and inference cost. LessIsMore's results provide a concrete recipe: deploy Qwen3-8B or similar GQA-based reasoning models with LessIsMore at K = 4096, achieving full-attention-equivalent accuracy on math and science reasoning benchmarks (Figure 6) while reducing per-token decoding latency by approximately 1.06× and avoiding the 18% generation length inflation that TidalDecode exhibits (Table 1, 17.4K vs. 14.8K tokens for full attention at K = 2000). For a batch evaluation of 500 MATH500 problems that would generate approximately 7.5M output tokens under full attention, LessIsMore at K = 4096 reduces total decode FLOPs by roughly 20–25% (reflecting both the per-token sparsity and the stable generation length) without measurable accuracy degradation. More aggressively, at K = 2048, LessIsMore achieves near-lossless accuracy on MATH500 and GPQA-Diamond (Figure 6, bottom rows) while attending to only 2K tokens per step, representing a roughly 2× reduction in attention compute relative to the 4K average generation length for these tasks. For organizations running nightly benchmark evaluations across dozens of model variants, this directly reduces GPU-hours and cloud compute costs.

Low-latency interactive reasoning assistants with on-device models. The paper's efficiency results on a single NVIDIA RTX A5000 GPU (a mid-range workstation card) are directly relevant to deployment scenarios where a smaller reasoning model runs locally on a user's device. The motivating example from the introduction — DeepSeek-R1-Distill-Llama-8B requiring 20+ minutes to generate 32K tokens on this GPU — defines the problem: interactive use is infeasible at full attention. LessIsMore at K = 4096 (the budget that achieves lossless accuracy on AIME-24 for Qwen3-8B) delivers a 1.10× decoding speed-up over full attention (Figure 7). On a 20-minute generation, this saves approximately 2 minutes — moving from unacceptable to merely slow. More importantly, the 87.5% sparsity achieved at K = 2048 with near-lossless accuracy implies that deployments willing to accept a small accuracy tradeoff (within 1–2 percentage points of full attention on AIME, per Figure 6) can achieve substantially larger speed-ups — potentially 1.2–1.3× over full attention — by pushing to lower budgets. The 7% shorter generation length relative to TidalDecode (Table 1) is particularly valuable in interactive settings, where perceived responsiveness depends on total wall-clock time, not just per-token speed. A reasoning assistant that generates 14.4K tokens instead of 15.9K tokens for the same answer finishes 1.5K tokens sooner, which at ~30ms/token translates to roughly 45 seconds of reduced wait time — perceptually meaningful for a user watching a chain-of-thought unfold.

Cost-efficient deployment of reasoning APIs with adaptive sparsity tiers. Cloud inference providers offering reasoning model APIs (e.g., together.ai, Fireworks, or first-party deployments of Qwen3-style models) can use LessIsMore to offer tiered service levels: a "high-accuracy" tier with K = 4096 that matches full-attention quality at a fraction of the compute cost, and a "fast" tier with K = 2048 or K = 1024 that offers substantially lower latency with a small accuracy tradeoff. The paper's results provide the numbers needed to set these tiers: K = 4096 yields lossless accuracy on AIME-24 and MATH500 (Figure 6); K = 2048 yields lossless accuracy on MATH500 and GPQA-Diamond, with a ~1% accuracy reduction on AIME-24 for Qwen3-8B (from ~74% to ~73%, well within sampling noise). A provider could route easy queries (short reasoning, simple domains) to K = 1024, medium queries to K = 2048, and only the hardest queries to K = 4096 or full attention — exactly the difficulty-conditioned allocation that Section 6 identifies as a missing feature but that can be approximated via query-length heuristics or lightweight difficulty classifiers. The per-token cost savings compound with the avoidance of generation length inflation: unlike Quest (which generates 30K tokens where full attention generates 14.8K at K = 2000, effectively negating the per-token savings), LessIsMore's generation length stability means the provider's cost-per-query scales nearly linearly with the sparsity level rather than being eroded by compensation behavior.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

LIMO: Less is More for Reasoning2025· reasoning← LLM-judged similar
- Chain of Draft: Thinking Faster by Writing Less2025· prompting← LLM-judged similar
- s1: Simple test-time scaling2025· reasoning← in-domain predecessor
- 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models2025· reasoning← similar topic
Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation2025· inference-optimization← in-domain predecessor
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding2025· inference-optimization← LLM-judged similar
- Parallelized Autoregressive Visual Generation2024· vision← LLM-judged similar
- LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs2025· inference-optimization← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

VibeVoice Technical Report2025· inference-optimization→ in-domain predecessor
- Qwen3-ASR Technical Report2026· training-methods→ LLM-judged similar
- Set Block Decoding is a Language Model Inference Accelerator2025· inference-optimization→ in-domain predecessor
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention2024· inference-optimization→ similar topic
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention2025· architecture→ LLM-judged similar
- Inference Performance Optimization for Large Language Models on CPUs2024· inference-optimization→ in-domain predecessor
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs2024· inference-optimization→ similar topic
- Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference2025· inference-optimization→ similar topic
- LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding2026· inference-optimization→ similar topic
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps2026· inference-optimization→ similar topic
- OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization2026· inference-optimization→ in-domain predecessor
- LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention2025· inference-optimization→ similar topic