FASA: Frequency-aware Sparse Attention

ArXiv: 2602.03152

🎯 Pitch

The paper uncovers that rotary position embeddings inherently create functional sparsity—only a handful of "dominant" frequency chunks actually determine which tokens matter to attention, the rest merely encode position. FASA exploits this zero-cost signal to dynamically predict token importance without any training, achieving near-lossless performance while keeping just 256 tokens and a 2.56× speedup on long reasoning tasks.

1. Executive Summary

This paper introduces FASA (Frequency-Aware Sparse Attention), a training-free framework for query-aware token eviction during LLM decoding, evaluated across Llama, Mistral, and Qwen model families on LongBench, long-sequence modeling (PG-19, WikiText, C4), and long chain-of-thought reasoning (MATH500, AIME24). The method is built on a novel discovery of functional sparsity at the frequency-chunk (FC) level induced by RoPE — a small, identifiable subset of "dominant FCs" consistently captures the contextual selection behavior of full attention heads (quantified via the paper's Contextual Agreement metric) — which FASA exploits through a two-stage pipeline: a Token Importance Predictor (TIP) that uses only the dominant FCs to dynamically estimate token saliency, followed by Focused Attention Computation (FAC) that performs full-dimensional attention exclusively on the pruned subset. FASA achieves near-lossless accuracy across tasks, reaching nearly 100% of full-KV performance on LongBench-V1 when retaining only 256 tokens and delivering a 2.56× speedup using just 18.9% of the cache on AIME24, while consistently outperforming all token-eviction baselines by substantial margins — for instance, on R1-Distill-Llama-8B with MATH500, FASA recovers 62.2% accuracy at a 300-token budget compared to SnapKV's 21.6% and the full-KV upper bound of 72.4%. The paper establishes that test-time token importance can be accurately predicted without training by exploiting the functional hierarchy inherent in rotary position embeddings, but only when the frequency chunk is treated as an indivisible unit and the FC-based scores are used strictly as a ranking proxy rather than as direct attention weights.

2. Context and Motivation

The Core Problem: The KV Cache Bottleneck in Long-Context LLM Inference

The fundamental problem this paper addresses is the prohibitive memory footprint and I/O bandwidth bottleneck of the Key-Value (KV) cache during auto-regressive decoding of Large Language Models, particularly when handling long input sequences. To understand why this matters, we need to understand how transformer attention works during generation.

During auto-regressive decoding, each new token the model generates must attend to every previous token in the sequence. The model stores precomputed key and value vectors for all past tokens in what is known as the KV cache, avoiding the need to recompute them at each step. For a sequence of length $t$, the KV cache stores $t$ key vectors and $t$ value vectors per layer and per attention head. As the sequence grows — whether from long input documents, multi-turn conversations, or extended chain-of-thought reasoning — the KV cache grows linearly with $t$. This creates two compounding problems:

Memory pressure. The KV cache can quickly exceed available GPU memory. For context lengths in the tens or hundreds of thousands of tokens, the cache alone can consume tens of gigabytes, making deployment on consumer-grade or resource-constrained hardware infeasible. As the authors note in Section 1, this is particularly acute for applications like repository-level code analysis, document summarization, and long-form reasoning, where sequence lengths routinely stretch to 32K tokens and beyond.

Memory bandwidth bottleneck during decoding. Perhaps less obvious but equally critical: the auto-regressive decoding phase is memory-bound, not compute-bound. At each generation step, the model must load the entire KV cache from GPU memory to compute attention scores. As Figure 3 demonstrates, at a 32K context length, decoding accounts for approximately 90% of total latency — the actual attention computation is fast, but the data movement from memory to the processor dominates wall-clock time. This means high-performance GPUs sit underutilized while waiting for data to arrive, fundamentally limiting throughput. As the authors put it (Section 3.2), this "memory-bound process underutilizes high-performance GPUs, ultimately limiting the overall throughput."

This problem is theoretically significant because it represents a fundamental inefficiency in the transformer architecture: the model spends the vast majority of its cycles accessing information that, for any given query, is mostly irrelevant. If only a small subset of tokens actually matters for each attention computation — a property known as attention sparsity — then loading the entire cache is wasteful in principle. The challenge is identifying which subset matters without paying the cost of examining every token first, which would defeat the purpose.

Why This Problem Matters: Real-World Impact

Democratization of long-context capabilities. Models with 128K+ context windows (like the Qwen2.5 series or Llama 3.1) are increasingly common, but running them at full context length requires expensive, high-memory GPUs. An effective token eviction method that preserves accuracy would enable researchers and practitioners with consumer-grade hardware to deploy long-context models, broadening access to state-of-the-art capabilities.

Economic and environmental costs. As the authors highlight in their Ethics Statement, making large-scale models "more accessible, affordable, and environmentally sustainable" is a primary positive impact. Reducing KV cache memory traffic directly translates to lower energy consumption per query and higher throughput per GPU, critical considerations as LLM inference scales to billions of daily queries.

Long chain-of-thought reasoning. The paper's emphasis on LongCoT tasks (MATH500, AIME24) points to an emerging challenge: modern reasoning models (like DeepSeek-R1) generate thousands of tokens of intermediate reasoning before arriving at a final answer. The KV cache for these chains grows to enormous sizes, and — unlike long-document tasks where the input is static — the entire generation history constitutes the cache that each subsequent reasoning step must attend to. This creates a doubly-challenging scenario: the cache is both large and dynamically evolving, with earlier reasoning steps potentially remaining critical for later inferences. A method that can selectively preserve this fragile "thought trace" while discarding irrelevant tokens is essential for practical deployment of reasoning models.

Prior Approaches and Their Shortcomings

The paper organizes existing KV cache optimization methods into several paradigms (Section 1), but focuses its critique on token eviction — the approach most closely related to FASA. The core idea of token eviction is simple: since only a small subset of past tokens significantly influences each new token's generation, we can permanently or temporarily remove the others from the KV cache, reducing both memory footprint and data transfer.

Existing token eviction methods fall into three categories, each with critical limitations:

Static strategies (e.g., StreamLLM; Xiao et al., 2024). These methods apply fixed rules to decide which tokens to keep, typically preserving a small number of initial tokens (the "attention sink" phenomenon, where the model learns to dump attention onto the first few tokens) and a sliding window of recent tokens. The approach is computationally cheap but fundamentally flawed: it risk irreversible information loss by discarding intermediate tokens that may be critical for answering a specific query. A question about a detail buried in paragraph 47 of a 100-paragraph document has no way to survive StreamLLM's eviction policy, which blindly retains only the beginning and end of the sequence. As Figure 4 shows, this leads to "a drastic increase in perplexity" on long-sequence modeling tasks because long-range dependencies are severed.

Adaptive strategies (e.g., SnapKV; Li et al., 2024, and Quest; Tang et al., 2024). These methods use heuristics to estimate token importance, typically based on accumulated attention scores during prefilling. SnapKV computes a one-time importance score for each token based on aggregated attention patterns from a small observation window and then permanently evicts tokens below a threshold. Quest organizes the KV cache into fixed-size "pages" and retrieves pages based on coarse query-page similarity during decoding.

The key shortcomings are threefold. First, SnapKV's estimation is static — it makes a single importance determination during prefilling and does not update it as decoding progresses. Token importance is inherently query-dependent: a token that is irrelevant to the first generated word may be critical for the tenth. SnapKV cannot adapt to this shifting relevance. Second, Quest's page-level granularity is too coarse — retrieving entire pages (of 16 tokens each) means that if only one token in a page is needed, the other 15 are loaded unnecessarily, wasting bandwidth and memory. Third, and most fundamentally, these heuristic rankings provide an imperfect proxy for the truly dynamic nature of token importance. They approximate "importance" through aggregated statistics rather than directly computing which tokens are most relevant to the specific query being processed at that decoding step. The paper states this explicitly: "such heuristic rankings provide an imperfect proxy for the truly dynamic nature of token importance" (Section 1).

Learning-based strategies (e.g., TokenButler; Akhauri et al., 2025). These methods train a separate predictor model to estimate token importance. While they can potentially learn more sophisticated importance patterns, they suffer from poor generalization across datasets and domains — a predictor trained on news articles may fail on code or mathematical reasoning — and require the overhead of training and deploying an auxiliary model. The paper frames this as the central question motivating FASA: "Can a token predictor achieve query-awareness without resorting to costly training?" (Section 1).

Beyond token eviction, the paper briefly acknowledges other paradigms — low-rank compression, quantization, KV merging, and budget allocation — but positions FASA as orthogonal to and compatible with these methods rather than competing with them directly. For example, Section 5.3 demonstrates that FASA can be combined with PyramidKV (a layer-wise budget allocation scheme) to yield additional gains.

Where Existing Token Eviction Falls Short: The Core Contradiction

The fundamental limitation shared by all prior token eviction methods is a failure to achieve query-aware, fine-grained, training-free token importance prediction at the decode stage. Let's unpack each of these requirements:

• Query-aware: Token importance is not a static property. A passage describing Einstein's early life may be irrelevant when the model is generating a summary of his scientific contributions but critical when it is generating his biographical details. Prior methods either ignore the query entirely (StreamLLM), approximate it through one-time prefill statistics (SnapKV), or approximate it through coarse aggregated heuristics (Quest).

• Fine-grained: The natural unit of attention is the individual token. Page-level retrieval (Quest) or sliding-window retention (StreamLLM) forces the model to either over-retrieve (wasting bandwidth on irrelevant tokens within a page or window) or under-retrieve (missing critical isolated tokens that fall outside the retention policy).

• Training-free: Learning-based methods require training data and risk distribution shift. A method that works out-of-the-box on any model without additional training is far more practical for deployment.

• At the decode stage: This is where the latency bottleneck lies (Figure 3). Methods that perform significant computation during prefilling (like SnapKV's observation window analysis) may add overhead to the initial processing but do not address the per-step cost of decoding — the dominant source of latency for long-context inference.

How FASA Positions Itself

FASA directly confronts this contradiction by asking: can we predict which tokens are important to the current query without training a predictor model, without resorting to coarse static heuristics, and without paying the full cost of computing attention over all tokens?

The paper's answer is rooted in a fundamental observation about how Rotary Positional Encodings (RoPE) structure the attention computation. RoPE encodes relative position by applying frequency-dependent rotations to query and key vectors — each pair of dimensions (a "frequency chunk" or FC) rotates at a characteristic angular velocity determined by its base frequency. The paper discovers that these FCs are functionally heterogeneous: a small subset of them (the "dominant FCs") consistently exhibits high Contextual Agreement — meaning their attention patterns closely match the full attention head's contextual selection behavior — while the vast majority construct robust but content-agnostic positional patterns (recency bias, attention sinks).

This insight is transformative because it means the full model's contextual awareness can be approximated using only a handful of frequency components. The dominant FCs form a "computationally free proxy" (Section 1) for token importance: by summing attention scores only over these sparse dimensions, you get a ranking of token saliency that closely matches what full attention would compute, at a fraction of the computational cost (since only $2 \times N_{\text{tip}}$ dimensions are used instead of the full $d$).

Crucially, the paper demonstrates that this functional sparsity is not an artifact of a particular model or task. The dominant FCs are:

• Sparse: In Table 9, dominant FCs with CA scores above 0.4 account for less than 1% of all FCs, while non-dominant FCs with CA scores below 0.15 comprise approximately 90% or more. This means the vast majority of the attention computation is effectively devoted to positional patterns that are largely independent of the specific content.
• Universal across architectures and scales: The heatmaps in Figure 1 (main text) and Figures 10–11 (Appendix A.1) show consistent dominant FC patterns across Llama, Mistral, and Qwen families, across model sizes from 3B to 32B, and even after long-context fine-tuning.
• Task-invariant: Figure 12 shows nearly identical saliency maps derived from question-answering (Qasper) and summarization (GovReport) datasets. Quantitatively, Table 10 reports that the overlap of dominant FCs across different calibration datasets consistently exceeds 70% across all tested models. This is critical: it means the dominant FCs can be identified once, on a single calibration sample, and then reused across diverse downstream tasks without recalibration.

This positions FASA as a principled alternative to prior token eviction methods. Rather than applying heuristics that approximate importance (attention scores, page similarity, recency), FASA leverages a structural property of the position encoding itself to do the prediction. The "offline calibration" step — identifying the dominant FCs for each attention head — is a one-time, negligible-cost process that requires only a single sample of text. The "online prediction" step — computing attention scores using only the dominant FC dimensions — is computationally cheap because it operates in a drastically reduced dimensional subspace ($2 \times N_{\text{tip}} \ll d$). And the prediction is inherently query-aware because the scores are computed from the actual query vector and key vectors at the current decoding step — they are not cached heuristics or precomputed statistics.

The paper frames this approach through the lens of two complementary stages: the Token Importance Predictor (TIP) acts as a "computationally frugal proxy" that estimates attention scores using only dominant FCs, and the Focused Attention Computation (FAC) performs the full-dimensional attention computation on only the top-ranked tokens. This coarse-to-fine strategy is analogous to how retrieval-augmented generation first identifies relevant documents and then processes them in detail — but here the "retrieval" happens at the token level within a single sequence, and the "document" is the set of past tokens that matter for the current query.

The Broader Significance

By demonstrating that functional sparsity in RoPE is both universal and task-agnostic, the paper provides a mechanistic understanding of why attention is sparse in the first place — the sparsity is not an emergent property of training but an architectural consequence of how positional information is encoded. This shifts the narrative around efficient attention from "we observe sparsity empirically and design heuristics to exploit it" to "we understand the structural basis of sparsity and can engineer methods that leverage it directly." The paper's discovery that the frequency chunk — not individual dimensions — is the indivisible functional unit for this prediction (Appendix D.2) further reinforces this mechanistic grounding: it is a direct consequence of how RoPE couples pairs of dimensions through 2D rotation matrices.

This positions FASA not merely as another token eviction method but as a framework grounded in a fundamental property of the most widely used position encoding scheme, with implications that extend beyond efficiency to how we understand the division of labor within transformer attention heads.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper is a systems and empirical analysis paper whose core idea is that Rotary Positional Encodings (RoPE) induce a functional sparsity at the level of frequency chunks — a small, identifiable subset of frequency components consistently captures the contextual selection behavior of full attention heads — which can be exploited to build a training-free, query-aware token importance predictor for efficient KV cache eviction during LLM decoding.

FASA is a two-stage inference framework that acts as a drop-in replacement for standard attention during auto-regressive decoding. It solves the problem of how to identify which past tokens are relevant to the current query without paying the full memory-bandwidth cost of loading the entire KV cache. The shape of the solution is a coarse-to-fine pipeline: first, a cheap proxy (using only a handful of frequency dimensions) ranks all past tokens by estimated relevance to the current query; then, full-dimensional attention is computed only on the top-ranked subset. The proxy is constructed from a structural property of RoPE itself, not from learned heuristics or auxiliary models, making it training-free, architecture-agnostic, and task-invariant.

3.2 Big-Picture Architecture (Diagram in Words)

The FASA system has four major stages, two offline (performed once, before any inference) and two online (executed at every decoding step):

Offline Stage 1: Dominant FC Calibration. Given a small calibration sample (a single example of text), the system computes attention scores for all heads and layers during a forward pass, then evaluates which frequency chunks consistently produce attention patterns that agree with the full head's attention pattern. This yields a set of dominant FC indices $I_{\text{dom}}$ for each attention head — typically a small fraction of all FCs (e.g., 8–16 out of 64). This calibration is performed once per model and reused across all tasks and datasets.

Offline Stage 2: FC Index Storage. These pre-computed dominant FC indices are stored in a globally accessible dictionary shared across all layers and heads. During inference, they are simply looked up — no recomputation occurs.

Online Stage 1: Token Importance Predictor (TIP). At each decoding step, for each attention head, the system computes a proxy attention score between the current query vector and all past key vectors, but using only the dimensions corresponding to the dominant FCs (a subspace of size $2 \times N_{\text{tip}}$ rather than the full $d$). These sparse scores are summed across dominant FCs to produce a single importance score per past token. The top $N_{\text{fac}}$ tokens by this importance score are selected as the contextually salient subset $\mathcal{T}_t$.

Online Stage 2: Focused Attention Computation (FAC). The system gathers the full-dimensional key and value vectors for only the selected tokens $\mathcal{T}_t$, preserving their original absolute positions, and computes standard scaled dot-product attention over this pruned set. The output is identical in form to standard attention output but computed over a drastically smaller key-value set — reducing both computation (fewer dot products) and memory I/O (fewer bytes loaded from the KV cache).

Information flow summary: A calibration sample → offline dominant FC identification → stored FC indices → (at each decode step) current query + past keys → TIP using only dominant FC dimensions → ranked token importance scores → top-k selection → FAC using full-dimensional attention on selected tokens → attention output → next token prediction.

3.3 Roadmap for the Deep Dive

• First, the functional sparsity hypothesis and the Contextual Agreement metric (Equation 4), because these are the conceptual foundation — they define what "dominant" FCs are and how we measure functional importance. Understanding this metric is prerequisite to understanding how dominant FCs are selected and why they work.

• Second, the offline calibration procedure (Equation 5), because it operationalizes the search for dominant FCs. We need to understand how $I_{\text{dom}}$ is identified from data before we can understand how it is used during inference.

• Third, the Token Importance Predictor (TIP) online stage, because it is the runtime mechanism that leverages the calibrated $I_{\text{dom}}$ to produce token rankings. This is where query-awareness is achieved — we need to understand exactly which dimensions are used, how scores are aggregated, and why the FC is the indivisible unit.

• Fourth, the Focused Attention Computation (FAC) online stage, because it is the output stage that consumes the TIP's token selection. We need to understand how selected tokens are gathered, why original positions are preserved, and how standard FlashAttention is reused.

• Fifth, the two hardware-aware variants (FASA-M and FASA-C) and their efficiency analysis, because these translate the algorithmic design into practical deployment choices — memory-optimized vs. computation-optimized — with different memory movement and latency tradeoffs.

• Sixth, the design principles and negative results (why FCs are indivisible, why FC scores cannot substitute for attention weights), because these are empirically validated constraints that define the boundary of what FASA can and cannot do.

3.4 Detailed, Sentence-Based Technical Breakdown

The Functional Sparsity Hypothesis and the Contextual Agreement (CA) Metric

The paper's central conceptual contribution is the discovery that Rotary Positional Encodings (RoPE) induce functional sparsity at the frequency-chunk level. To make this precise, we first need to understand how RoPE structures the attention computation, how frequency chunks are defined, and how the Contextual Agreement metric quantifies what it means for a frequency chunk to be "dominant."

RoPE from a Frequency-Chunk Perspective. In standard RoPE (Su et al., 2023), a $d$-dimensional vector $\mathbf{v} \in \mathbb{R}^d$ (such as a query or key) is partitioned into $d/2$ orthogonal 2D subspaces, each corresponding to a frequency chunk (FC). The paper defines the $i$-th FC as the vector pair:

$\mathbf{v}^{[i]} = (v_{2i}, v_{2i+1})^T \quad \text{for } i \in \{0, 1, \ldots, d/2 - 1\}$

where $\mathbf{v}^{[i]}$ is the 2D sub-vector from dimensions $2i$ and $2i+1$ of the original vector.

Each FC $i$ is associated with a unique base angular frequency calculated as:

$\theta_i = B^{-2i/d}$

where $B$ is the RoPE frequency base (typically 10,000 in Llama models, though the paper does not specify exact values per model) and $d$ is the head dimension. This formula means that lower index FCs (small $i$) have higher frequencies — they rotate faster with positional change — while higher index FCs rotate more slowly.

For a token at absolute position $m$, its $i$-th FC is rotated by an angle $m\theta_i$ through a $2 \times 2$ rotation matrix:

$\mathbf{R}_{m, \theta_i} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}$

The global rotation matrix for relative position $\Delta t$ is block-diagonal, applying independent 2D rotations to each FC:

$\mathbf{R}_{\Delta t} = \text{Diag}(\mathbf{R}_{\Delta t, \theta_0}, \mathbf{R}_{\Delta t, \theta_1}, \ldots, \mathbf{R}_{\Delta t, \theta_{d/2-1}})$

What this means operationally: the attention score between a query at position $t_1$ and a key at position $t_2$ can be decomposed as a sum of independent contributions from each FC:

$A_{t_1, t_2} = \mathbf{q}_{t_1} \mathbf{R}_{\Delta t} \mathbf{k}_{t_2}^T = \sum_{i=0}^{d/2-1} \mathbf{q}_{t_1}^{[i]} \mathbf{R}_{\Delta t, \theta_i} \mathbf{k}_{t_2}^{[i] T}$

where $\Delta t = t_1 - t_2$ is the relative position. This decomposition is exact — it simply expresses the full dot product as a sum over FC-wise dot products, each computed in a 2D subspace with its own rotation.

The Functional Heterogeneity Hypothesis. The paper's motivating insight (Section 3.2) is that FCs are not functionally uniform — different FCs contribute qualitatively different types of information to the attention computation. Drawing on prior work by Barbero et al. (2025) and Wei et al. (2025), the paper posits that:

• High-frequency FCs (low $i$ indices) are primarily responsible for constructing robust positional patterns, such as recency bias and attention sinks — these patterns are largely content-agnostic and depend mainly on relative position.

• Low-frequency FCs (high $i$ indices) specialize in carrying semantic information and modeling long-range dependencies — these patterns are content-dependent and capture which past tokens are contextually relevant to the current query.

The critical hypothesis is that the model's contextual awareness — its ability to identify which tokens are semantically relevant to the current query — is overwhelmingly driven by a small subset of "contextual FCs", while the majority of FCs contribute primarily to positional patterns that are relatively invariant to the specific content. If this hypothesis is true, then the full attention head's contextual selection behavior can be approximated by summing attention scores over only these dominant FCs:

$A_{t_1, t_2} \approx \sum_{i \in I_{\text{dom}}} \mathbf{q}_{t_1}^{[i]} \mathbf{R}_{\Delta t, \theta_i} \mathbf{k}_{t_2}^{[i] T}$

where $I_{\text{dom}} \subset \{0, 1, \ldots, d/2-1\}$ is the set of dominant FC indices.

The Contextual Agreement (CA) Metric. To test this hypothesis and identify which FCs are dominant, the paper introduces the Contextual Agreement (CA) metric. CA measures how well the attention pattern produced by a single FC aligns with the full attention head's pattern.

For a specific attention head $(l, h)$, query $\mathbf{q}_t$, and key matrix $\mathbf{K}_{1:t}$ (keys for all past tokens), we first compute two raw score vectors:

$\alpha_{l,h}(\mathbf{q}_t, \mathbf{K}_{1:t}) = [\mathbf{q}_t \mathbf{R}_{t-1} \mathbf{k}_0^T, \mathbf{q}_t \mathbf{R}_{t-2} \mathbf{k}_1^T, \ldots, \mathbf{q}_t \mathbf{R}_0 \mathbf{k}_t^T]^T$

This is the vector of full-head attention scores — the raw dot products (before softmax) between the query and every past key, which is a $t$-dimensional vector representing how much the current query "attends to" each past token according to the full attention head.

For each individual FC $i$, we compute the analogous single-FC scores:

$\alpha^{(i)}_{l,h}(\mathbf{q}_t, \mathbf{K}_{1:t}) = [\mathbf{q}_t^{[i]} \mathbf{R}_{t-1, \theta_i} \mathbf{k}_0^{[i] T}, \mathbf{q}_t^{[i]} \mathbf{R}_{t-2, \theta_i} \mathbf{k}_1^{[i] T}, \ldots, \mathbf{q}_t^{[i]} \mathbf{R}_0, \theta_i \mathbf{k}_t^{[i] T}]^T$

This is also a $t$-dimensional vector, but computed using only the 2D components of FC $i$ — it represents how much the $i$-th frequency chunk "thinks" the query should attend to each past token.

The CA score is defined as the normalized intersection of the top-K token index sets between these two score vectors:

$\text{CA}_{K}^{l,h,i}(\mathbf{q}_t, \mathbf{K}_{1:t}) = \frac{|\text{TopK-I}(\alpha_{l,h}(\mathbf{q}_t, \mathbf{K}_{1:t}), K) \cap \text{TopK-I}(\alpha^{(i)}_{l,h}(\mathbf{q}_t, \mathbf{K}_{1:t}), K)|}{K}$

where $\text{TopK-I}(\alpha, K)$ returns the set of indices of the $K$ largest values in vector $\alpha$ — in other words, the indices of the $K$ tokens that receive the highest attention scores.

What this equation computes in operational English: For a given query, the full attention head produces a ranking of all past tokens by relevance — its top-K set identifies which tokens the head considers most important. Each individual FC also produces its own ranking. The CA score is the fraction of these top-K sets that overlap — it answers the question: "if the full head thinks these K tokens are most important, how many of them does this particular FC also identify as most important?" A CA score of 1.0 means the FC perfectly agrees with the full head on which tokens matter; a score near 0 means the FC's importance ranking is essentially random relative to the full head.

Why this form: The paper uses top-K set intersection rather than, say, correlation between the full score vectors because FASA's downstream task — token eviction — is fundamentally about ranking: we need to identify which tokens to keep, not to perfectly reproduce the continuous attention weights. The FC scores will not be used as attention weights (and Appendix D.2 explicitly warns against doing so), only as a ranking proxy. Top-K overlap directly measures ranking quality without requiring score calibration. Alternative metrics like Spearman rank correlation or mean squared error would measure properties of the full distribution that are irrelevant to the eviction task. The choice of $K$ controls the "resolution" of the agreement measurement — smaller $K$ measures agreement on only the most highly attended tokens, while larger $K$ measures broader agreement.

The paper computes the mean CA score by averaging across multiple samples from a calibration dataset, providing a robust estimate of each FC's typical agreement with the full head. The heatmaps in Figure 1 visualize these mean CA scores per FC (x-axis) across all attention heads (y-axis) for representative layers, revealing the characteristic pattern: a few bright vertical bands (FCs with CA > 0.3–0.5) against a dark background (CA < 0.15 for most FCs). This visualization is the primary evidence for functional sparsity.

Key empirical properties established via CA:

• Sparsity (Table 9): Dominant FCs (defined as those with CA > 0.4) constitute less than 1% of all FCs across all tested models. Non-dominant FCs (CA < 0.15) constitute approximately 90% or more. This means the vast majority of the attention computation — when decomposed by frequency — is devoted to positional patterns that are largely content-agnostic.

• Universality (Figures 1, 10, 11): The pattern of dominant FCs is consistent across model families (Llama, Mistral, Qwen), model sizes (3B to 32B), and even persists after long-context fine-tuning. This suggests the functional division of labor is an architectural property of RoPE, not an artifact of specific training dynamics.

• Task-invariance (Figure 12, Table 10): Saliency maps derived from question-answering (Qasper) and summarization (GovReport) datasets are nearly identical. Quantitatively, the overlap of dominant FC sets across different calibration datasets exceeds 70% consistently. This means the dominant FCs are not adapting to specific task demands — they perform a fundamental, task-agnostic role.

Predictive capacity of dominant FCs (Table 1): To validate that dominant FCs can actually predict token importance, the paper computes a compound CA score — the agreement between the full head and the aggregated scores from only the selected dominant FCs $I_{\text{dom}}$. With just $F = 8$ FCs (1/8 of all 64 FCs), the compound CA reaches 43.0 at a tight budget of $K = 64$, surpassing SnapKV's 37.9 — and this advantage grows as more FCs are selected, reaching 55.3 with $F = 16$ FCs. This is the critical validation: a handful of frequency channels, identified through a one-time calibration, can predict token importance more accurately than sophisticated heuristic methods that require per-sample computation.

Predictive distribution across attention score ranges (Table 11): The paper further analyzes which tokens the dominant FCs successfully identify. For the top 20% most-attended tokens (the ones receiving the highest full-head attention scores), dominant FCs achieve 74–82% prediction accuracy across models — meaning they correctly identify roughly 4 out of 5 of the most important tokens. This accuracy degrades gracefully for lower-attention tokens, but this is acceptable because the eviction task needs to reliably identify the most important tokens, not perfectly rank the least important ones.

Offline Calibration: Identifying the Dominant FC Indices

The offline calibration procedure (Algorithm 1 in Appendix D.3) solves the problem: given a model and a small calibration dataset, identify the set of dominant FC indices $I_{\text{dom}}^{l,h}$ for each attention head $(l, h)$ that maximizes expected contextual agreement.

The calibration is formulated as a combinatorial search problem over frequency indices:

$I_{\text{dom}}^{l,h} = \underset{I \subseteq \{0,\ldots,d/2-1\}, |I| = N_{\text{tip}}}{\arg\max} \mathbb{E}_{\mathbf{q}, \mathbf{K} \sim \Omega} \left[ \sum_{i \in I} \text{CA}_{K}^{l,h,i}(\mathbf{q}, \mathbf{K}) \right]$

where $\Omega$ is the calibration dataset, $N_{\text{tip}}$ is the target number of dominant FCs to select (a hyperparameter controlling the precision of the TIP stage), and the sum inside the expectation is over the CA scores for the selected FCs on a particular query-key pair.

What this equation computes: For each attention head, we want to pick the $N_{\text{tip}}$ FC indices (out of $d/2$ possible) that, when their individual CA scores are summed, maximize the expected total agreement with the full head — averaged over all queries and key contexts in the calibration dataset. The expectation $\mathbb{E}_{\mathbf{q}, \mathbf{K} \sim \Omega}$ means we run the model on the calibration data, compute CA scores for every FC at every generation step, and then average per FC across all steps.

Why this form: The objective uses a sum over selected FCs rather than, say, taking the CA of the joint aggregated scores because the TIP stage sums FC contributions independently to produce the importance score vector (see next section). The sum-of-individual-CAs objective directly optimizes for the property that matters downstream: how well the selected FCs' scores will rank tokens when aggregated. An alternative would be to jointly optimize the FC set for compound CA — but this would require evaluating $\binom{d/2}{N_{\text{tip}}}$ combinations, which is computationally intractable. The paper implicitly uses a greedy approach (top K by individual CA) since individual CA scores are highly predictive of compound performance (as evidenced by the monotonic improvement in Table 1 as more FCs are added).

Calibration procedure in detail (Algorithm 1):

1. Initialize an empty map $\mathcal{M}$ to store CA scores for each $(l, h, i)$ triplet (layer, head, FC index).

2. For each example in the calibration dataset $\Omega$:

   • Run the model's forward pass (standard inference, no eviction).
   • At each token generation step $t$:
     • For each layer $l$ and head $h$:
       • Compute the full attention scores $\alpha_{l,h}(\mathbf{q}_t, \mathbf{K}_{1:t})$ — this requires the standard full-dimensional attention computation, but only during calibration.
       • For each FC index $i$:
         • Compute the single-FC scores $\alpha_{l,h}^{(i)}(\mathbf{q}_t, \mathbf{K}_{1:t})$ using only the 2D subspace for FC $i$.
         • Calculate $\text{CA}_{K}^{l,h,i}(\mathbf{q}_t, \mathbf{K}_{1:t})$ — the fraction of overlap between the full-head top-K and single-FC top-K.
         • Store this CA value in $\mathcal{M}[l][h][i]$ (appending to a list for later averaging).

3. After processing all data, compute the mean CA score for each $(l, h, i)$ by averaging over all stored values: $\bar{\mathcal{M}}[l][h][i] \leftarrow \text{Mean}(\mathcal{M}[l][h][i])$.

4. Select the top $N_{\text{tip}}$ FCs for each head by sorting by mean CA score and taking the indices with the highest values: $I_{\text{dom}}^{l,h} \leftarrow \text{TopK-Indices}(\bar{\mathcal{M}}[l][h], N_{\text{tip}})$.

Design choices and practical considerations:

• Calibration dataset size: The paper states that FASA's LongBench experiments used "just a single data sample from the Qasper dataset" for calibration (Appendix B.1), and for Long-CoT reasoning, "a similar single-instance calibration was performed on a question from the MATH500 dataset." This is possible because the dominant FCs are task-invariant — the calibration sample only needs to provide enough signal to compute reliable mean CA scores, and even one sample provides many query-key pairs (one per generation step per head per layer) for averaging.

• Uniform configuration across heads: The paper employs "a uniform configuration across all heads and layers" with $N_{\text{tip}} = 16$ consistently (Appendix B.1). This simplifies the architecture and "maximizes computational parallelism" — all heads use the same number of dominant FCs, even though individual heads may have slightly different optimal sets. The choice of 16 represents a balance between preserving contextual information and minimizing computational overhead; 16 FCs correspond to 32 dimensions, which is the amount of data accessed per token in the TIP stage.

• Calibration window $K$: The CA metric depends on $K$, the size of the top-K sets being compared. Table 5 shows an ablation where performance is "largely insensitive to $K$," with smaller $K$ values sometimes yielding slightly superior results. The paper's default is $K = 256$. The robustness to $K$ is attributed to the inherent sparsity of attention — even small calibration windows provide sufficient signal because most attention mass concentrates on few tokens.

• One-time cost: Crucially, the calibration is performed once per model, offline, and the resulting $I_{\text{dom}}$ indices are stored and reused across all tasks and datasets. The computational cost is negligible compared to the inference savings — it is a fixed overhead amortized over all future decoding steps.

• Storage: The dominant FC indices are stored in a globally accessible dictionary, shared across all layers and heads. Each entry is simply a list of $N_{\text{tip}}$ integer indices per head. For a model with 32 layers and 32 heads each, with $N_{\text{tip}} = 16$, the total storage is $32 \times 32 \times 16$ integers — effectively zero memory overhead.

Validation of the calibration approach (Tables 6 and 10):

• Cross-dataset robustness (Table 10): The overlap of dominant FC sets identified on different calibration datasets (Qasper, GovReport, Musique, NarrativeQA, 2WikiMQA) consistently exceeds 70% across all tested models. The average overlap is 80–86%. This means calibration on one dataset produces FC indices that are highly consistent with those from other datasets — a necessary condition for the one-time calibration to generalize.

• Performance stability (Table 6): When FASA is calibrated on different datasets and then evaluated on a fixed set of downstream tasks, the performance variation is minimal. The Coefficient of Variation (standard deviation / mean) across calibration datasets is 0.007–0.014 across tasks — i.e., less than 1.5% relative variation. For example, on Qasper, FASA calibrated on Qasper achieves 43.7, while FASA calibrated on NarrativeQA achieves 43.5 — a difference of 0.2 percentage points. This confirms that the calibration data choice is not a critical hyperparameter.

Why this calibration approach is justified: The key empirical finding is that functional sparsity is intrinsic to RoPE's mechanism, not emergent from specific training data distributions. The FCs' roles — high-frequency for positional patterns, low-frequency for semantic content — are predetermined by the frequency base $B$ and the dimension-to-frequency mapping. Training may refine which specific low-frequency FCs are most informative for a given model, but the overall structure is architecturally determined. This is why a single calibration sample suffices: the CA scores are measuring a stable property of the trained model, not a property of the calibration data distribution.

Online Stage 1: Token Importance Predictor (TIP)

The TIP stage operates at every decoding step to efficiently estimate which past tokens are most relevant to the current query, using only the pre-calibrated dominant FC indices. The core idea is that the full attention score can be decomposed into a sum of FC-wise contributions, and if we sum only over the dominant FCs, we get a cheap but accurate proxy for token importance ranking.

Score Computation. For a query $\mathbf{q}_t$ at decoding step $t$ and past keys $\mathbf{K}_{1:t}$, the full attention score vector (before softmax) is:

$\alpha_{l,h}(\mathbf{q}_t, \mathbf{K}_{1:t}) = \sum_{i=0}^{d/2-1} \alpha_{l,h}^{(i)}(\mathbf{q}_t, \mathbf{K}_{1:t})$

where $\alpha_{l,h}^{(i)}$ is the FC-$i$ contribution computed using only dimensions $(2i, 2i+1)$.

The TIP stage computes an importance score vector $\mathbf{S}_t^{l,h}$ by aggregating contributions from only the dominant FC indices $I_{\text{dom}}^{l,h}$:

$\mathbf{S}_t^{l,h} \triangleq \sum_{i \in I_{\text{dom}}^{l,h}} \alpha_{l,h}^{(i)}(\mathbf{q}_t, \mathbf{K}_{1:t})$

where $\mathbf{S}_t^{l,h} \in \mathbb{R}^t$ is a vector of length $t$ (one score per past token).

What this computes: For each past token, we compute a score that approximates how much the full attention head would attend to that token, but we do so by summing over only $|I_{\text{dom}}| = N_{\text{tip}}$ FCs instead of all $d/2$ FCs. Each FC contribution involves only 2-dimensional dot products between the query and key subspaces, rotated by the appropriate relative position matrix.

Why this works: The non-dominant FCs (the ~90% of FCs with low CA scores) contribute primarily to positional patterns — recency bias, attention sinks — that are largely independent of content. By excluding them, the TIP score emphasizes the content-dependent component of attention: which tokens are semantically relevant to the current query. The non-dominant FCs would add a roughly constant positional baseline to all tokens (e.g., uniformly boosting recent tokens or initial tokens), which does not change the relative ranking for eviction purposes — we are removing a signal that is largely query-independent and retaining the signal that discriminates between tokens based on content.

Token Selection. Given the importance score vector $\mathbf{S}_t^{l,h}$ for head $(l, h)$, the TIP stage identifies the set of top-$N_{\text{fac}}$ most important token indices:

$\mathcal{T}_t = \text{TopK-I}(\mathbf{S}_t^{l,h}, N_{\text{fac}})$

where $N_{\text{fac}}$ is the FAC token budget — a hyperparameter controlling how many tokens are retained for full attention computation. The paper uses a consistent notation: $N_{\text{tip}}$ is the number of dominant FCs (controlling TIP precision), and $N_{\text{fac}}$ is the number of retained tokens (controlling FAC fidelity).

Complexity of TIP. Each FC operates in a 2-dimensional subspace. Computing $\alpha_{l,h}^{(i)}$ requires $t$ dot products in 2D (one per past token), which is $O(t)$ per FC. With $N_{\text{tip}}$ FCs, the TIP complexity is $O(2 t N_{\text{tip}})$ — linear in both context length and the number of dominant FCs, but with a very small constant (2 dimensions per FC). In contrast, the full attention score computation is $O(t d)$, where $d$ is typically 128 (the head dimension for many models). With $N_{\text{tip}} = 16$ and $d = 128$, TIP uses 32 dimensions vs. 128 — a 4× reduction in computational FLOPs for the score computation.

Critical Design Principle: FCs Are Indivisible Units. The paper explicitly investigated whether individual dimensions (rather than FC pairs) could serve as the selection unit, and the answer is definitive: "A pipeline based on selecting 'dominant dimensions' suffers a catastrophic performance degradation" (Appendix D.2). This is because RoPE applies 2D rotations to coupled dimension pairs — splitting these pairs would sever the positional encoding, destroying the model's ability to interpret relative positions. The frequency chunk is the minimum functional unit for any operation that interacts with RoPE's position encoding.

Critical Design Principle: FC Scores Are a Ranking Proxy, Not Attention Weights. The paper explicitly warns that the TIP scores $\mathbf{S}_t^{l,h}$ should not be used as direct attention weights: "Although they provide a remarkably accurate relative ranking of token importance, their direct substitution for attention probabilities leads to a catastrophic performance degradation" (Appendix D.2). The TIP scores are not calibrated — they sum over only a subset of FCs, so their magnitudes do not match full attention scores, and their distribution is not suitable for softmax normalization. Their sole purpose is to produce a ranking for token selection. The FAC stage recomputes full-dimensional attention on the selected subset to produce properly calibrated attention weights.

Why the two-stage design is necessary: One might ask: why not simply use the TIP scores directly as attention weights? The answer is that the TIP operates in a severely reduced subspace (32 dimensions vs. 128) and has no access to the non-dominant FCs, which, while not contributing to token ranking, do contribute to the magnitudes needed for proper softmax normalization. Attempting to softmax-normalize the TIP scores would produce a distorted probability distribution that does not match the full head's output. The two-stage design separates the problems of (a) identifying which tokens matter (solved by TIP in low-dimensional space) and (b) computing how much they matter (solved by FAC in full-dimensional space).

Trade-off between $N_{\text{tip}}$ and $N_{\text{fac}}$ (Figure 5). The hyperparameters $N_{\text{tip}}$ (TIP precision) and $N_{\text{fac}}$ (FAC budget) govern a trade-off between selection accuracy and compute cost. Figure 5 (left, TREC dataset) and Figure 5 (right, MATH dataset) show that:

• With high-precision selection (large $N_{\text{tip}}$, e.g., 14 FCs), a small token budget (e.g., $N_{\text{fac}} = 300$) can match full-KV performance — because the TIP accurately identifies the truly important tokens.
• With lower-precision selection (small $N_{\text{tip}}$, e.g., 8 FCs), a larger token budget is needed to compensate — because the TIP may miss some important tokens, and retaining more tokens increases the chance that the missed ones are still included.
• Empirically, on TREC, using 10 dominant FCs with $N_{\text{fac}} = 500$ is sufficient to match FKV performance (80.5).

The paper's default configuration uses $N_{\text{tip}} = 16$ uniformly. At $N_{\text{tip}} = 16$, with a typical head dimension of 128, the TIP uses 25% of the dimensions (32/128) to achieve ranking accuracy that exceeds SnapKV's heuristic (Table 1).

Online Stage 2: Focused Attention Computation (FAC)

The FAC stage takes the token indices $\mathcal{T}_t$ produced by TIP and performs a standard, full-dimensional attention computation exclusively on this pruned subset, producing the final attention output for the head.

Token Gathering. The first step is to extract the key and value vectors for only the selected tokens from the full KV cache:

$\mathbf{K}_{\mathcal{T}_t} = \text{Gather}(\mathbf{K}_{1:t}, \mathcal{T}_t), \quad \mathbf{V}_{\mathcal{T}_t} = \text{Gather}(\mathbf{V}_{1:t}, \mathcal{T}_t)$

where $\text{Gather}(\cdot)$ selects the rows (token positions) specified by the index set $\mathcal{T}_t$ from the original key and value matrices. Critically, the original absolute positions of the tokens in $\mathcal{T}_t$ are preserved — the gathering operation extracts the key/value vectors but does not re-index them. This means that when RoPE is applied (which was already done during the prefill stage; the cached keys already encode their absolute positions), the positional information remains intact.

Why preserving original positions matters: If tokens were re-indexed to positions 0, 1, 2, ..., $|\mathcal{T}_t|-1$, the relative position between two selected tokens would be distorted — token at original position 47 and token at original position 52 would appear to be at relative position 5, but after re-indexing to positions 2 and 5, their relative position would be 3. Since the model was trained with standard RoPE, which encodes exact relative positions, such distortion would confuse the model and degrade performance. By preserving original positions, FASA ensures that the attention computation on the pruned set is identical in form to attention on the full set — only the set of keys and values differs, not their positional encodings.

Attention Computation. The standard scaled dot-product attention is then computed over the pruned set:

$\hat{\alpha}_{\text{FAC}}^{l,h} = \text{Softmax}\left(\mathbf{q}_t \mathbf{K}_{\mathcal{T}_t}^T / \sqrt{d}\right), \quad \mathbf{O}_t^{l,h} = \hat{\alpha}_{\text{FAC}}^{l,h} \mathbf{V}_{\mathcal{T}_t}$

where $\mathbf{O}_t^{l,h} \in \mathbb{R}^d$ is the output vector for head $(l, h)$ at step $t$. The output is then projected through the output projection matrix $\mathbf{W}_O$ (standard in multi-head attention) to produce the head's contribution to the next hidden state.

Why this form: This is exactly the standard attention formula, but with a restricted key-value set of size $N_{\text{fac}}$ instead of size $t$. The softmax normalizes over only the retained tokens, meaning the attention probability mass is distributed only among the contextually salient subset. The model cannot attend to evicted tokens — their contribution is implicitly zero. This is appropriate because the TIP stage has determined that those tokens are not contextually relevant to the current query.

Computational Complexity of FAC. The FAC stage computes:

• Query-key dot products: $O(N_{\text{fac}} d)$ — reduced from $O(t d)$ in standard attention.
• Softmax: $O(N_{\text{fac}})$ — reduced from $O(t)$.
• Value weighting: $O(N_{\text{fac}} d)$ — reduced from $O(t d)$.

Total FAC complexity: $O(2 N_{\text{fac}} d)$, compared to $O(2 t d)$ for standard attention. The speedup is approximately $t / N_{\text{fac}}$ — if $t = 10,000$ and $N_{\text{fac}} = 256$, the FAC is roughly 39× faster than full attention for the computation-bound portion.

Integration with FlashAttention. A key engineering detail (Figure 14) is that the FAC stage is designed to seamlessly integrate with FlashAttention (Dao et al., 2022). The gathered keys and values $(\mathbf{K}_{\mathcal{T}_t}, \mathbf{V}_{\mathcal{T}_t})$ form a contiguous tensor of shape $(1, N_{\text{fac}}, d)$ that can be passed directly to FlashAttention's forward function — the same API used for full attention, just with fewer key-value tokens. This means FASA inherits FlashAttention's memory-efficient tiling and kernel fusion without any modification to the attention implementation itself. The paper's implementation "intercepts the forward pass of the FlashAttention2 class within the model's modeling.py file" (Appendix B.4) via a monkey-patching approach, injecting the TIP logic before the attention call and passing the gathered KV subset to the standard FlashAttention kernel.

Correctness guarantee: Because FAC recomputes attention in full dimensionality using the original (unmodified) key and value vectors with their original positional encodings, the output $\mathbf{O}_t^{l,h}$ is identical to what standard attention would produce if the non-retained tokens received exactly zero attention weight. The only approximation is in the token selection — TIP may include some tokens that full attention would down-weight or exclude some that full attention would up-weight. The FAC computation itself is exact; the error comes entirely from the ranking quality of the TIP stage.

Hardware-Aware Variants: FASA-M and FASA-C

The paper introduces two specialized implementations that trade off between memory savings and computation speed, targeting different deployment scenarios.

FASA-M (Memory-Optimized). This variant is designed for VRAM-constrained environments such as consumer-grade GPUs. The key insight is that the full KV cache can be partitioned between GPU and CPU memory based on which components are needed at which stages of the two-stage pipeline.

The KV cache is split into three components:

1. Dominant key cache $\mathbf{C}_{\text{key}}^{\text{dom}}$: Only the dimensions corresponding to dominant FCs of all past keys, stored in GPU memory. Size: $t \times d_{\text{dom}}$ per head, where $d_{\text{dom}} = 2 N_{\text{tip}}$.
2. Non-dominant key cache $\mathbf{C}_{\text{key}}^{\text{nondom}}$: The remaining key dimensions for all past tokens, offloaded to CPU memory. Size: $t \times d_{\text{nondom}}$ per head, where $d_{\text{nondom}} = d - d_{\text{dom}}$.
3. Value cache $\mathbf{C}_{\text{val}}$: All value vectors for all past tokens, offloaded to CPU memory. Size: $t \times d$ per head.

At each decoding step during the TIP stage, only $\mathbf{C}_{\text{key}}^{\text{dom}}$ (already on GPU) is accessed — the TIP operates exclusively in the dominant FC subspace, so it never needs the non-dominant key dimensions or the values. After TIP identifies $\mathcal{T}_t$, the system transfers only the required subsets from CPU to GPU:

• $\mathbf{K}_{\mathcal{T}_t}^{\text{nondom}}$ (non-dominant key dimensions for selected tokens)
• $\mathbf{V}_{\mathcal{T}_t}$ (full value vectors for selected tokens)

These are combined with $\mathbf{K}_{\mathcal{T}_t}^{\text{dom}}$ (already on GPU) to reconstruct full-dimensional keys for FAC. The FAC then proceeds on GPU with all required data.

GPU memory footprint analysis (Equation 10):

$\text{Mem}_{\text{GPU}} \approx N_{\text{layers}} \times \left( \underbrace{L \times d_{\text{dom}}}_{\text{Dominant Keys}} + \underbrace{b \times d_{\text{nondom}}}_{\text{Non-dominant Keys}} + \underbrace{b \times d}_{\text{Values}} \right) \times \text{bytes\_per\_param}$

where $L$ is the total sequence length, $b = N_{\text{fac}}$ is the FAC token budget, $d_{\text{dom}}$ is the dominant FC dimension, $d_{\text{nondom}} = d - d_{\text{dom}}$, and $\text{bytes\_per\_param}$ is typically 2 for FP16.

What this equation reveals: The dominant key cache scales with total sequence length $L$ but only in a fraction $d_{\text{dom}} / d$ of the full dimension. The non-dominant keys and values scale only with the budget $b$, not with $L$ — because only the selected tokens' non-dominant parts and values need to be on GPU. For a typical configuration with $d_{\text{dom}} = 0.25d$ and $b = 0.1L$, the GPU memory is dominated by the dominant key cache ($0.25 L d$), while the non-dominant keys ($0.1 L \times 0.75 d = 0.075 L d$) and values ($0.1 L d$) together add $0.175 L d$. Total GPU memory: $0.425 L d$ per head, compared to $2 L d$ for a full KV cache — an approximately 4.7× reduction. The paper reports "approaching an 8× reduction in typical configurations" (Appendix D.1), which would correspond to more aggressive settings (smaller $d_{\text{dom}}/d$ ratio or smaller $b/L$).

Latency overhead mitigation. CPU-GPU data transfer for the selected subset introduces latency. The paper notes this "can be effectively mitigated by prefetching techniques that asynchronously load the required KV pairs in advance" — that is, the TIP stage can run concurrently with prefetching the non-dominant keys and values that were needed for previous queries, overlapping computation and data transfer.

FASA-C (Computation-Optimized). This variant is designed for scenarios where GPU memory is sufficient but inference latency is the bottleneck. The full KV cache is retained in GPU memory, and no data is offloaded to CPU. However, during the TIP stage, only the dominant FC dimensions of keys are accessed — the non-dominant key dimensions and values are never loaded from GPU memory during TIP. This drastically reduces memory I/O bandwidth consumption, which is the primary bottleneck in memory-bound decoding.

The key distinction from FASA-M is that FASA-C does not save GPU memory (the full cache is still allocated), but it saves memory bandwidth by only reading a fraction of the cache during TIP. The speedup comes from the fact that at each decoding step, standard attention loads $2 t d$ elements from memory (all keys and values), while FASA loads:

• TIP: $t \times d_{\text{dom}}$ elements (dominant key dimensions only)
• FAC: $2 N_{\text{fac}} d$ elements (full keys and values for selected tokens)

The fraction of data loaded is $(t d_{\text{dom}} + 2 N_{\text{fac}} d) / (2 t d) = d_{\text{dom}}/(2d) + N_{\text{fac}}/t$. When $N_{\text{fac}} \ll t$ (the typical case), this is approximately $d_{\text{dom}}/(2d)$. With $N_{\text{tip}} = 16$ and $d = 128$, $d_{\text{dom}} = 32$, the fraction is $32/(2 \times 128) + N_{\text{fac}}/t = 0.125 + N_{\text{fac}}/t$ — roughly 1/8 of the memory traffic, or an 8× bandwidth reduction.

Theoretical speedup (Equation 8):

$\text{Speedup} = \frac{2 t d}{2 t N_{\text{tip}} + 2 N_{\text{fac}} d} = \frac{1}{N_{\text{tip}}/d + N_{\text{fac}}/t}$

When $N_{\text{fac}} \ll t$, this simplifies to:

$\text{Speedup} \approx \frac{d}{N_{\text{tip}}}$

With $d = 128$ and $N_{\text{tip}} = 16$, the theoretical speedup is $128/16 = 8\times$. In practice, the paper reports a 2.56× speedup at 64K sequence length with $N_{\text{tip}} = 16$ (Figure 7) — less than theoretical due to overhead from the TIP computation itself, top-K selection, and gather operations, but still substantial.

Implementation details. FASA-C is "implemented with Triton (based on Ribar et al., 2024)" — a GPU kernel programming framework that allows fine-grained control over memory access patterns. The implementation leverages Triton's ability to write custom attention kernels that only load the required dimensions rather than the full cache.

Complementarity with other methods. The paper emphasizes (Section 5.3) that FASA is orthogonal to and compatible with other KV cache optimization paradigms. For example, FASA can be combined with PyramidKV (Cai et al., 2025b), a layer-wise budget allocation scheme. PyramidKV decides how many tokens to keep per layer (e.g., more tokens in early layers, fewer in late layers), while FASA decides which tokens to keep in each layer. The combination yields consistent performance gains (Table 4), with FASA + PyramidKV outperforming FASA alone by up to 1.1 percentage points in CA score.

Design Principles and Empirically Validated Constraints

The paper's experimental investigation uncovered two critical design principles that define what FASA can and cannot do. These are presented as negative results — approaches that were tried and failed — which are arguably as informative as the positive results.

Principle 1: Frequency Chunks Are Indivisible Functional Units. The paper investigated whether individual dimensions — rather than paired FCs — could serve as the selection unit for the TIP. The answer is categorical: "A pipeline based on selecting 'dominant dimensions' suffers a catastrophic performance degradation" (Appendix D.2).

Why this must be true: RoPE encodes position by applying 2D rotation matrices to pairs of dimensions $(2i, 2i+1)$. These rotations are geometric operations in a 2D plane — they mix the two dimensions through sine and cosine. If you select dimension $2i$ without dimension $2i+1$, you are computing $q_{2i} (\cos(\theta) k_{2i} - \sin(\theta) k_{2i+1})$ — which is missing the $\sin(\theta)$ term from $q_{2i+1}$. The result is not a valid rotation; it severs the positional encoding. The FC is the minimum unit that preserves the geometric structure of RoPE. This principle is "a direct corollary of RoPE's core mechanism" and holds for any RoPE-based model.

Principle 2: FC Scores Are a Ranking Proxy, Not a Substitute for Attention Weights. The paper tested whether the aggregated FC scores $\mathbf{S}_t^{l,h}$ could be directly softmax-normalized and used as attention weights (bypassing the FAC stage entirely). This also leads to catastrophic degradation, "revealing their fundamental role as a selector — a mechanism to identify salient tokens rather than an approximator of the final attention distribution" (Appendix D.2).

Why this must be true: The TIP scores sum over only $N_{\text{tip}}$ FCs while ignoring the remaining $d/2 - N_{\text{tip}}$ FCs. The non-dominant FCs, while not changing the relative ranking of tokens, contribute to the absolute magnitudes of attention scores. Softmax is sensitive to these magnitudes — it amplifies differences in logit space into probability differences. Using only a subset of FCs produces logits with the wrong dynamic range, and applying softmax distorts the distribution. The FAC stage is necessary to recompute properly calibrated attention weights in full dimensionality.

Why this matters for correctness: These two principles together define FASA's architecture as fundamentally two-stage. One cannot collapse TIP and FAC into a single stage — the two stages have different dimensionality requirements (sparse for ranking, full for computation), and violating either principle (using individual dimensions or using FC scores directly) breaks the model's positional encoding or attention calibration. These constraints are intrinsic to RoPE's design, not implementation limitations, and would apply to any method that exploits frequency sparsity.

4. Key Insights and Innovations

Innovation 1: Functional Sparsity as an Architectural Diagnostic — Moving from "We Observe Sparsity" to "We Understand Why It's There"

The field of efficient attention has long operated under the empirical premise that attention is sparse — most tokens receive negligible attention weight, so they can be safely discarded. This observation motivated an entire research program of token eviction heuristics (StreamLLM's attention sinks, SnapKV's aggregated attention scores, H2O's heavy-hitter tracking), all of which exploit sparsity as a phenomenon without interrogating its structural origin. The implicit assumption was: sparsity emerges from training dynamics and data distribution, so we design methods to detect it post-hoc.

FASA makes a fundamentally different intellectual move. Rather than treating sparsity as an emergent property to be measured, the paper asks: is sparsity encoded in the architecture itself? The discovery that frequency chunks within RoPE exhibit functional sparsity — that fewer than 1% of FCs capture contextual selection while ~90% construct positional patterns (Table 9) — reframes attention sparsity from a behavioral observation to an architectural invariant. This is a diagnostic contribution, not merely a method contribution. The paper doesn't just say "attention is sparse" — it says "here is which part of the computation produces the sparse contextual signal, and here is which part produces the dense positional baseline."

This distinction matters because it explains why prior methods have inconsistent failure modes. StreamLLM's fixed retention of initial and recent tokens succeeds because those tokens are heavily weighted by the positional FCs — the attention sink and recency bias patterns that the non-dominant FCs construct. But when a query requires attending to semantically relevant content buried in the middle of a document, StreamLLM fails catastrophically because the contextual FCs — the only ones that encode content-dependent relevance — would have flagged those intermediate tokens as important, but they were evicted by a rule that only respects position. The paper's CA metric (Equation 4) provides the diagnostic instrument to see this division of labor directly: the contextual FCs have high CA because they track the full head's content-dependent attention; the positional FCs have low CA because their patterns are query-independent and thus don't agree with the full head's contextual selection (though they contribute to the total attention magnitude).

The universality and task-invariance of dominant FCs (Figures 10–12, Table 10) further reinforce this as a mechanistic insight, not an empirical correlation. The dominant FC sets are consistent across model families (Llama, Mistral, Qwen), across model scales (3B to 32B), across training paradigms (including long-context fine-tuned variants), and across calibration tasks (QA vs. summarization, with >70% overlap). This is not what you would expect if dominant FCs were learning to solve specific tasks — it is what you would expect if they implement a fundamental computational primitive that the architecture bakes in. The paper's contribution here is analogous to the discovery of induction heads in transformer circuits: it identifies a functional component with a specific, interpretable role, grounded in the mathematical structure of the model rather than in training data statistics.

The significance beyond performance is that this insight provides a principled vocabulary for reasoning about attention. Prior work discussed "important tokens" as a monolithic concept; FASA decomposes this into (a) tokens that matter for contextual reasons (identified by dominant FCs) and (b) tokens that matter for positional reasons (supported by all FCs collectively). This decomposition is not a heuristic — it falls directly out of RoPE's frequency parameterization. And because RoPE is the dominant position encoding scheme across essentially all modern open-source LLMs, this vocabulary applies broadly.

Innovation 2: The Frequency Chunk as the Correct Indivisible Unit — A Negative Result with Architectural Teeth

The paper makes a methodological contribution that is both a negative result and a design constraint: the frequency chunk (a pair of RoPE-coupled dimensions) is the indivisible functional unit for any operation that interacts with positional encoding. Selecting individual dimensions leads to catastrophic failure.

This might seem like a minor implementation detail, but it is conceptually significant because it constrains the design space for every future method that exploits frequency-domain sparsity in RoPE. The reason individual dimensions fail is not an empirical quirk — it is a mathematical consequence of how RoPE encodes position via 2D rotation. A rotation in a 2D plane mixes coordinates through sine and cosine; if you only have one coordinate, you lose the geometric structure entirely. The paper positions this as a "direct corollary of RoPE's core mechanism" (Appendix D.2), but the corollary was not obvious before the paper made it explicit and empirically demonstrated it.

Why is this a contribution rather than an obvious fact? Because prior work on low-rank and sparse attention (SparQ, LoKi) did not respect this constraint. SparQ (Ribar et al., 2024) selects key dimensions based on query-vector magnitudes — an approach that operates on individual dimensions without regard for RoPE's coupling structure. The paper's comparison with SparQ (Figure 17) shows that this heuristic "proves to be a poor substitute for true contextual awareness" and collapses under constrained budgets. FASA's discovery that the FC is the minimum functional unit provides the diagnostic explanation for why SparQ underperforms: it severs RoPE's positional encoding by operating on individual dimensions rather than FC pairs.

This insight is a boundary condition for the broader research program of frequency-aware attention optimization. It tells the community: if you want to exploit frequency-domain structure in RoPE-based models, you must treat dimensions in their coupled pairs. This is not a recommendation; it is an empirically validated architectural constraint. The paper's language is appropriately strong: "a definitive no" to the question of whether individual dimensions can work (Appendix D.2), and "any optimization for RoPE-based models must respect the inherent coupling of dimension pairs, treating the Frequency Chunk as an indivisible functional unit." This is the kind of finding that prevents future researchers from going down dead-end paths and creates a shared vocabulary for the subfield.

Innovation 3: Query-Awareness Without Training — Reframing Token Importance Prediction as a Frequency-Subspace Projection Problem

Prior work on query-aware token eviction faced an unpleasant choice: either (a) use static heuristics that ignore the query (StreamLLM) and lose contextual fidelity, (b) use adaptive heuristics that approximate query relevance from prefill-stage statistics (SnapKV) and lose dynamic adaptation, or (c) train a separate predictor model (TokenButler) and suffer generalization failures and deployment overhead. The dominant assumption was that accurate, query-aware token importance prediction requires either expensive runtime computation or learned models.

FASA reframes the problem entirely. Instead of asking "how can we learn to predict which tokens a query will attend to?", it asks: "which dimensions of the attention computation contain the query-dependent information, and can we compute only those?" The answer — that the dominant FCs isolated by the CA metric encode exactly this information — converts token importance prediction from a learning problem to a structured subspace projection problem. The TIP stage is not learning a mapping from queries to token rankings; it is computing the query-key dot product in a specific low-dimensional subspace that, by construction, captures the content-dependent component of attention while filtering out the positional baseline.

This reframing is significant because it decouples quality from data. A learned predictor's quality depends on the distribution match between training and deployment data — a predictor trained on news articles may fail on code. FASA's TIP quality depends on the architectural property of RoPE, which is fixed once the model is trained. The one-time offline calibration identifies the subspace once, and the online prediction is simply a matrix multiplication in that subspace — no learning, no adaptation, no distribution shift. The paper demonstrates this through the calibration robustness experiments (Table 6): calibrating on Qasper vs. NarrativeQA vs. Musique produces coefficients of variation under 1.5%, meaning the predictor's quality is essentially invariant to calibration data choice.

This is a paradigm shift for the token eviction subfield. The question is no longer "how do we build a better importance heuristic?" but "what is the minimal subspace that captures the query-dependent attention signal, and how do we identify it?" The Contextual Agreement metric and the offline calibration procedure provide a complete answer for RoPE-based models. For non-RoPE architectures (Section 6), the paper demonstrates that the same principle — functional sparsity exists in ALiBi and Partial-RoPE, just at different granularities — suggesting the reframing may generalize beyond RoPE.

Innovation 4: The Two-Stage Coarse-to-Fine Architecture as a Necessary Consequence of Frequency-Domain Structure

On the surface, FASA's two-stage design (TIP then FAC) looks like a standard coarse-to-fine pipeline: cheap proxy, then expensive refinement on the proxy's output. But the paper's contribution is demonstrating that this architecture is not an engineering choice — it is forced by the structure of RoPE.

The key negative result is that FC-based scores cannot substitute for attention weights directly (Appendix D.2). This is not a calibration issue that could be fixed with a learned scaling factor; it is a fundamental consequence of the fact that the non-dominant FCs, while not contributing to token ranking, contribute to the magnitudes needed for proper softmax normalization. The TIP scores operate in a 32-dimensional subspace; softmax normalization in this subspace produces a different distribution than softmax in the full 128-dimensional space. The two stages therefore have fundamentally different requirements that cannot be collapsed: TIP needs only the ranking-relevant dimensions (the dominant FCs), while FAC needs all dimensions to produce properly calibrated attention weights.

This is conceptually important because it defines the minimal architecture for any frequency-sparsity-based method. A single-stage approach that attempted to do both ranking and weighting in the reduced subspace would fail — not due to insufficient optimization, but due to a structural mismatch. The two-stage design is not an implementation detail; it is the architectural signature of the fact that ranking and weighting are separable problems in RoPE's frequency decomposition.

The paper also provides a positive version of this insight through the N_tip vs. N_fac trade-off analysis (Figure 5). The fact that high-precision selection (large N_tip) can compensate for a small token budget (small N_fac), and vice versa, demonstrates that the two stages address genuinely different axes of the approximation: TIP quality governs ranking accuracy, while FAC budget governs how many tokens survive to full-dimensional computation. This is not a trivial parameter trade-off — it reflects the decomposition of the full attention computation into a selection problem (which dimensions encode the ranking signal?) and a computation problem (how many tokens can we afford full attention on?). Prior work conflated these into a single "how many tokens to keep" decision, missing the opportunity to invest compute in better selection rather than simply retaining more tokens.

Innovation 5: Extending the Paradigm Beyond RoPE — Functional Sparsity as a General Property of Position Encodings

Section 6 of the paper makes a brief but significant conceptual move: it demonstrates that functional sparsity is not unique to RoPE, but appears in other position encoding schemes — specifically ALiBi and Partial-RoPE (as used in MLA, Multi-head Latent Attention, in DeepSeek-V2). This is important because it suggests the paper's core insight — that position encoding mechanisms induce a functional division of labor — may be a general architectural principle rather than a RoPE-specific curiosity.

The evidence is preliminary — a few heatmaps (Figures 8–9) and two small evaluation tables (Tables 7–8) — and the paper does not claim to fully characterize functional sparsity in non-RoPE settings. But the conceptual move is significant: it transforms the paper's contribution from "RoPE has an interesting property we can exploit" to "position encodings have a functional structure that we are only beginning to understand, and RoPE's frequency-chunk decomposition makes this structure especially interpretable and exploitable."

This is a boundary-expanding contribution rather than a fully fleshed-out result. It tells the community: the approach of identifying which components of a position encoding carry semantic vs. positional information, and using only the semantic components for token selection, may generalize. For ALiBi, the functional sparsity manifests at the head level rather than the frequency-chunk level (since ALiBi uses additive biases rather than rotational encoding), but the principle — some heads specialize in contextual selection while others handle positional patterns — appears to hold. For Partial-RoPE (MLA), where RoPE is applied only to a portion of the dimensions, the sparsity manifests within the RoPE-applied subspace.

The theoretical significance is that this points toward a taxonomy of position encoding structures organized by how they distribute functional roles. Full-RoPE distributes across frequency chunks; ALiBi distributes across heads; Partial-RoPE distributes across both dimensional partitions and frequency chunks within the RoPE-applied partition. Understanding this taxonomy could enable designing position encodings that are explicitly optimized for separability — where the "contextual" and "positional" components are architecturally segregated from the start, making token eviction trivial. This is speculative and beyond the paper's scope, but the paper opens the door by demonstrating that the phenomenon exists across encoding schemes.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three distinct paradigms: (1) LongBench-V1 (Bai et al., 2024), a multi-task benchmark for long-context understanding with sequence lengths from 4K to over 100K tokens, spanning single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion — 16 tasks are reported in Table 2; (2) long-sequence modeling via perplexity on PG-19 (Rae et al., 2019), WikiText (Merity et al., 2017), and C4 (Raffel et al., 2019), evaluating generative fidelity over long dependencies; (3) long chain-of-thought reasoning on MATH500 (Hendrycks et al., 2021) and AIME24 (MAA, 2024), where models must maintain coherence across thousands of auto-regressively generated reasoning tokens. The MATH500 subset consists of 500 competition-level math problems; AIME24 uses pass@1 computed from 16 independent generations per question.

• Base model(s). The paper spans five model families and scales: Llama-3.2-3B-Instruct, Meta-Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-14B-Instruct-1M (a long-context fine-tuned variant), and Qwen2.5-32B-Instruct for the LongBench and perplexity experiments; plus three DeepSeek-R1 distilled models (Llama-8B, Qwen-14B, Qwen-32B) for long-CoT reasoning. The diversity spans three architectures (Llama, Mistral, Qwen), model sizes from 3B to 32B, and both standard and reasoning-specialized training paradigms. The authors state the models are chosen to be "representative of the capabilities of many contemporary LLMs" (Section 4).

• Metrics. For LongBench, the paper follows the official evaluation protocol: F1 score for QA tasks, ROUGE score for summarization tasks, code similarity score for code completion tasks, with the final reported score being the average across all constituent tasks. For long-sequence modeling, perplexity (PPL) is computed as the exponential of the average negative log-likelihood: PPL(W) = exp(-(1/N) Σ log P(w_i | w_{<i})). For long-CoT reasoning, pass@1 accuracy is reported — for MATH500, a single generation per problem; for AIME24, pass@1 computed over 16 independent generations per question. For the FLOPs-efficiency analysis (Figure 7), GPU memory (GB) and wall-clock latency (seconds) are measured directly.

• Baselines. The paper benchmarks against two groups of baselines (Section 5.1). Token eviction methods: (1) StreamLLM (Xiao et al., 2024) — preserves a fixed number of initial tokens ("attention sinks") plus a sliding window of recent tokens; configured with start_size=8 and recent_size=budget-8. (2) SnapKV (Li et al., 2024) — estimates token importance from aggregated attention scores during a prefill observation window, with maxpool strategy, window size 32, kernel size 7; for long-generation tasks, filtering is re-applied every n generated tokens following Cai et al. (2025a). (3) Quest (Tang et al., 2024) — organizes KV cache into pages (size 16) and retrieves pages based on query-page similarity. (4) H2O (Zhang et al., 2023) — retains "heavy hitter" tokens based on cumulative attention scores; evaluated on CoT tasks. (5) RKV (Cai et al., 2025a) — a specialized retrieval-based method for CoT compression with λ=0.1 balancing recent vs. important tokens. Upper bounds: (1) FKV (Full KV) — standard inference with uncompressed KV cache, the absolute performance ceiling. (2) Oracle — assumes ideal knowledge to retain only the most critical tokens based on full-head attention scores; establishes the maximum achievable performance for any sparse attention method at a given budget. Low-rank comparison: SparQ (Ribar et al., 2024) — selects key dimensions based on high query-vector magnitudes (Figure 17 only).

• Generation budget / compute accounting. For LongBench and perplexity experiments, the primary budget metric is retained token count — the number of tokens kept in the KV cache for attention computation (Tables 2 and 6, Figure 4 show budgets from 128 to 2048 tokens). For FASA, this corresponds to N_fac. All baselines are configured to the same token budget for fair comparison. For the efficiency analysis (Figure 7), compute is measured in GPU memory (GB) and decoding latency (seconds) at various sequence lengths from 1K to 64K tokens. The memory movement analysis (Section 4.4) defines the fraction of KV cache loaded as N_tip/d + N_fac/t ≈ N_tip/d when N_fac ≪ t. For the Long-CoT experiments (Table 3), budgets are reported as fixed token counts (300, 500, 700, 1000 for MATH500; 500–2500 for AIME24), with FASA using N_tip = 16 consistently.

• Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional train/validation/test sense. Instead, it relies on a one-time offline calibration on a single sample that is separate from the evaluation data: for LongBench, calibration is performed on "just a single data sample from the Qasper dataset" (Appendix B.1); for Long-CoT, calibration uses "a single-instance ... on a question from the MATH500 dataset." The task-invariance of dominant FCs (quantified in Table 10 with >70% overlap across calibration datasets) serves as the validation that calibration generalizes. For the Long-CoT experiments, pass@1 on AIME24 is based on 16 generations per question with the standard sampling-based estimation. The paper does not report confidence intervals or statistical significance tests for any result; all reported numbers are point estimates (means across the test set).

Main Quantitative Results

Long-Context Understanding on LongBench-V1

The headline result is that FASA achieves near-lossless performance relative to full KV cache across all tested models while retaining only 256 tokens, consistently outperforming all token-eviction baselines by substantial margins (Table 2).

Aggregate performance across 16 tasks: On Llama-3.2-3B, FASA achieves an average score of 41.5 across all LongBench tasks, compared to FKV's 42.2 — a drop of only 0.7 points. This is dramatically better than the best baseline, SnapKV, at 37.0 (a 5.2-point drop from FKV). On Meta-Llama-3.1-8B, FASA scores 48.2 vs. FKV's 48.7 (0.5-point drop), while SnapKV scores 45.0 (3.7-point drop). On Qwen2.5-14B-1M, FASA scores 49.2 vs. FKV's 50.3 (1.1-point drop), with SnapKV at 45.9 (4.4-point drop). Across all five model configurations in Table 2, FASA's average gap from FKV is 0.54 points, while SnapKV's average gap is 4.38 points — FASA is approximately 8× closer to full-KV performance.

Per-task analysis reveals where baselines fail catastrophically: On NarrativeQA (single-doc QA), Quest's performance on Llama-3.2-3B plummets to 8.7 compared to FKV's 26.0 — a collapse of 17.3 points (Table 2, column "NQA"). FASA achieves 25.6, within 0.4 points of FKV. On the Code task (Lcc, code completion), Quest scores 34.5 on Llama-3.2-3B vs. FKV's 52.0 — a 17.5-point gap — while FASA scores 53.2, actually exceeding FKV by 1.2 points. The paper attributes such occasional FASA > FKV results to "the mitigation of attentional distraction from irrelevant tokens" (Section 5.2), a hypothesis corroborated by the Oracle baseline also sometimes outperforming FKV (e.g., Oracle at 42.4 vs. FKV at 42.2 on Llama-3.2-3B).

Cross-model consistency: The performance pattern is remarkably stable across model families and scales. On Mistral-7B, StreamLLM catastrophically fails on multi-doc QA (2WikiMQA: 27.1 vs. FKV's 39.5) and code (Lcc: 44.5 vs. FKV's 76.0), while FASA recovers to within 0.4 points of FKV on 2WikiMQA (39.1) and within 2.0 points on Lcc (58.0). On Qwen2.5-7B, SnapKV loses 5.2 points on average; FASA actually exceeds FKV by 0.1 points (47.9 vs. 47.8). The largest relative gap is on the hardest tasks: on MultiFieldQA-zh (Qwen2.5-7B), SnapKV scores 45.6 vs. FKV's 50.4 (a 4.8-point gap), while FASA scores 49.9, a 0.5-point gap.

Performance under varying budgets (Figure 6 and 15–16): Across budget ranges from 128 to 2048 tokens on Qwen2.5-32B, FASA consistently tracks the FKV and Oracle upper bounds, while Quest, StreamLLM, and SnapKV exhibit substantial gaps that do not fully close even at 2048 tokens. For instance, on GovReport (Qwen2.5-32B), FKV scores approximately 0.27, FASA follows at roughly 0.25–0.27 across all budgets, while SnapKV plateaus near 0.20–0.24 and Quest near 0.18–0.22. On HotpotQA, the gap is wider: FKV around 0.58, FASA around 0.52–0.58, SnapKV around 0.42–0.52, and Quest around 0.34–0.42.

Comparison with low-rank baseline SparQ (Figure 17): Under a constrained budget of 256 tokens on LongBench, SparQ's performance collapses — achieving roughly 15–55% across budget levels — while FASA matches FKV's performance across the full budget range from 250 to 2000 tokens. The paper attributes SparQ's failure to its reliance on query-vector magnitudes as a heuristic, which "proves to be a poor substitute for true contextual awareness" (Appendix C.1).

Long-Sequence Modeling (Perplexity)

The headline finding is that FASA achieves perplexity nearly identical to full KV cache across diverse long-text corpora, while StreamLLM and Quest degrade sharply, especially at aggressive token sparsity ratios (Figure 4).

Quantitative results at token sparsity ~0.2 (20% retention): On Llama-3.2-3B with WikiText, FKV perplexity is approximately 10.0–10.5; FASA tracks this closely at roughly 10.5–11.0, while StreamLLM degrades to approximately 14.0 and Quest to roughly 12.0–12.5. On PG-19, FKV is approximately 15.0, FASA at 15.5–16.0, StreamLLM at 25.0+, and Quest at 17.5–20.0. On C4, FKV is approximately 16.0, FASA at 17.0–18.0, StreamLLM at 22.0–24.0, and Quest at 18.0–20.0.

Scaling with token sparsity: As the retention ratio decreases (more aggressive compression), FASA's perplexity degrades gracefully while baselines diverge sharply. On Meta-Llama-3.1-8B with WikiText (Figure 4, bottom row, left), at token sparsity 0.1 (10% retention), FKV is approximately 6.0, FASA is roughly 7.0, Quest is approximately 8.0, and StreamLLM degrades to roughly 7.5–8.0 (note that StreamLLM's fixed-rule retention of initial + recent tokens disproportionately hurts long-range dependency modeling, but the relative ordering depends on corpus). The key pattern: FASA's perplexity curve stays close to the FKV and Oracle lines across all sparsity levels and corpora, while Quest and StreamLLM diverge substantially at sparsity below 0.4.

Cross-model consistency: The pattern holds across Llama-3.2-3B (top row), Meta-Llama-3.1-8B (second row), Mistral-7B-v0.3 (third row), and Qwen2.5-14B (bottom row), across all three corpora. The absolute perplexity values differ by model capacity (larger models achieve lower perplexity), but the relative ordering of methods is consistent: FKV ≈ Oracle ≈ FASA > Quest > StreamLLM.

Mechanism of failure for baselines: StreamLLM's "fixed-rule approach ... severely compromises its ability to capture long-range dependencies" because it discards all intermediate tokens regardless of content, severing the model's access to context beyond the sliding window. Quest's "coarse, page-level granularity prevents it from adaptively retaining critical, non-contiguous tokens" — within a page of 16 tokens, if only 1–2 are relevant, Quest wastes budget on the other 14–15, and if a critical token spans a page boundary, it may be partially or entirely missed.

Long Chain-of-Thought Reasoning (MATH500 and AIME24)

This is the most challenging evaluation setting because it combines extreme sequence lengths (thousands of generated tokens, not just long input context) with the need to preserve dynamically shifting "thought traces" where the model's own intermediate reasoning steps must remain accessible for later steps.

MATH500 results (Table 3): On DeepSeek-R1-Distill-Llama-8B, FKV achieves 72.4% pass@1. At a 300-token budget, FASA achieves 62.2% — within 10.2 points of FKV and dramatically ahead of SnapKV (21.6%), StreamLLM (9.6%), H2O (6.8%), and RKV (24.0%). This is a 40.6-point advantage over the best baseline (RKV) at the tightest budget. As budget increases, FASA's performance rises to 68.8% (at 500 tokens), 69.4% (at 700 tokens), and 71.8% (at 1000 tokens) — essentially matching FKV's 72.4% with only 1000 tokens retained. In contrast, SnapKV at 1000 tokens reaches only 54.6%, and H2O reaches 42.8%.

On larger reasoning models: DeepSeek-R1-Distill-Qwen-14B shows FKV at 92.4%. FASA achieves 86.6% at 300 tokens, 88.8% at 500, 90.2% at 700, and 91.2% at 1000 — within 1.2 points of FKV. SnapKV at 1000 tokens reaches 79.4%, a 12-point gap from FKV. RKV — explicitly designed for CoT compression — achieves 86.4% at 1000 tokens, which FASA surpasses at only 500 tokens (88.8%). On DeepSeek-R1-Distill-Qwen-32B (FKV: 92.6%), FASA achieves 86.4% at 300 tokens and 91.2% at 1000 tokens, while RKV peaks at 83.6% at 1000 tokens — FASA reaches RKV's best performance with less than 1/3 the budget.

AIME24 results (Table 3): This is a substantially harder benchmark — FKV on Llama-8B is only 43.9% pass@1. FASA at 500 tokens achieves 20.6%, growing to 40.2% at 1500 tokens (within 3.7 points of FKV), compared to SnapKV at 500 tokens scoring only 8.0%. On Qwen-14B (FKV: 66.6%), FASA at 500 tokens achieves 54.0% and at 2500 tokens reaches 63.3%, compared to SnapKV's 23.3% at 500 tokens and RKV's 30.0% at 500 tokens. On Qwen-32B (FKV: 72.8%), FASA at 500 tokens achieves 60.7% and scales to 73.2% at 2500 tokens — exceeding FKV — while RKV peaks at 61.3% at 2500 tokens.

Output length analysis: A critical and often overlooked aspect of compression methods is their impact on generation length. Table 3 reports prefill, decoding, and total token counts. For DeepSeek-R1-Distill-Llama-8B on MATH500, FKV generates 3104 total tokens (127 prefill + 2977 decode). FASA generates 3298 total tokens (127 prefill + 3171 decode) — a 6.2% increase. In contrast, H2O generates 8370 total tokens (8244 decode) — a 2.7× increase in output length, indicating severe generative verbosity. StreamLLM generates 3647 total tokens — only 17.6% more than FKV, but with drastically worse accuracy (47.4% vs. 72.4%), suggesting premature termination and truncated reasoning. SnapKV generates 7174 total tokens (2.3× FKV) with poor accuracy. FASA is the only method that maintains output length nearly identical to FKV while preserving high accuracy, a behavior the authors describe as "demonstrating a superior balance" (Section 5.2).

On AIME24, output length patterns are more variable: For Qwen-32B, FKV generates 10,626 total tokens; FASA generates 11,891 (12% increase); RKV generates 18,243 (72% increase). The longer outputs from RKV and H2O represent an additional hidden computational cost: not only do they produce lower accuracy, but they also require more decoding steps, each of which loads the KV cache — partially negating any per-step efficiency gains from compression.

Efficiency Analysis: Memory and Latency

Figure 7 presents a head-to-head comparison of GPU memory usage and decoding latency for FKV, FASA-C, and FASA-M across sequence lengths from 1K to 64K tokens, with N_tip = 16.

Memory scaling: FKV memory grows approximately linearly with sequence length, reaching roughly 75 GB at 64K tokens. FASA-C memory is identical to FKV (it retains the full cache on GPU). FASA-M memory grows much more slowly — at 64K, it reaches approximately 10 GB, an approximately 7.5× reduction from FKV. The gap widens with sequence length because FASA-M's GPU-resident dominant key cache scales with L × d_dom (a fraction of the full dimension) while the CPU-offloaded components scale with budget, not sequence length.

Latency scaling: FKV decoding latency at 1K tokens is approximately 0.02 seconds and grows roughly linearly to approximately 0.14 seconds at 64K. FASA-C latency is consistently lower — at 1K, approximately 0.012 seconds (1.7× speedup); at 16K, approximately 0.05 seconds (2.0× speedup); at 32K, approximately 0.07 seconds (2.3× speedup); at 64K, approximately 0.10 seconds (2.56× speedup). The speedup increases with sequence length because the memory I/O savings (loading only dominant key dimensions during TIP) grow proportionally with t. FASA-M latency is slightly higher than FKV at short sequences (due to CPU-GPU transfer overhead) but crosses over around 4K–8K tokens; at 64K, FASA-M latency is approximately 0.11 seconds vs. FKV's 0.14 seconds — a 1.3× speedup, demonstrating that even with CPU offloading, the memory bandwidth savings eventually dominate.

Theoretical vs. empirical speedup: The paper's theoretical speedup formula (Equation 8) predicts approximately d / N_tip = 128 / 16 = 8× speedup when N_fac ≪ t. The empirical speedup of 2.56× at 64K is substantially lower because (a) TIP computation itself adds overhead (computing N_tip FC-wise dot products and top-K selection), (b) the gather operation for FAC has memory access costs not accounted for in the theoretical FLOPs model, and (c) FlashAttention's existing efficiency means the baseline is already highly optimized, leaving less room for improvement. The authors do not provide a detailed breakdown of where the remaining overhead lies.

Memory movement analysis (Section 4.4): The fraction of KV cache data loaded per decoding step is approximately N_tip/d + N_fac/t. With N_tip = 16, d = 128, N_fac = 256, and t = 10,000, this fraction is 16/128 + 256/10000 = 0.125 + 0.0256 ≈ 0.1506, meaning FASA accesses roughly 15% of the memory bandwidth of standard attention. As t grows, the N_fac/t term becomes negligible, and the fraction approaches N_tip/d = 16/128 = 0.125, an 8× bandwidth reduction. This is the source of the speedup — the decoding stage is memory-bound, so reducing memory traffic directly reduces latency.

Compatibility with Other KV Cache Methods

Table 4 demonstrates FASA's orthogonality to layer-wise budget allocation by combining it with PyramidKV (Cai et al., 2025b). On Qasper (Llama-3.1-8B) at a 256 token budget, FASA achieves 43.7 CA score; FASA + PyramidKV achieves 44.4, a gain of +0.7. At 2048 budget, the gain is +0.1. On Lcc (code completion) at 256 budget, FASA at 61.8, FASA + PyramidKV at 62.2 (+0.4); at 2048 budget, 64.9 (+0.1). The gains are modest but consistent, confirming that FASA's token selection (which tokens to keep) is complementary to PyramidKV's budget allocation (how many tokens per layer) — the two methods address orthogonal axes of the compression problem.

Ablation Studies and Robustness Checks

Robustness to calibration window size K: Table 5 examines how the CA-based dominant FC identification is affected by the choice of K (the top-K set size used in the Contextual Agreement metric). FASA's average performance across token budgets from 128 to 2048, for calibration K values ranging from 128 to 1024, varies between 43.9 and 44.5 — a range of only 0.6 points. The lowest average (43.9 at K=512) and the highest (44.5 at K=128) differ by only 1.4%. The paper notes that "smaller K values often yielding slightly superior results" and attributes robustness to "the inherent sparsity of attention" — even a small calibration window provides sufficient signal because attention concentrates on few tokens. This means the calibration procedure does not require careful tuning of K.

Trade-off between N_tip and N_fac: Figure 5 examines the interaction between the number of dominant FCs used for token prediction (N_tip, controlling TIP precision) and the number of tokens retained for full attention (N_fac, controlling FAC fidelity). On TREC (Qwen2.5-14B-Instruct-1M, left panel), FKV accuracy is 80.5. With N_tip = 8 (8 FCs, 16 dimensions), FASA requires N_fac = 700 to match FKV; with N_tip = 14 (14 FCs, 28 dimensions), FASA matches FKV at N_fac = 300. The pattern is a precision-budget trade-off: more dominant FCs (higher TIP precision) allows a smaller token budget, and vice versa. On MATH (DeepSeek-R1-Distill-Qwen-14B, right panel), FKV is 92.4. With N_tip = 8, FASA at N_fac = 600 achieves approximately 89.0; with N_tip = 14, FASA at N_fac = 300 achieves approximately 90.0. The paper's default N_tip = 16 sits near the high-precision end of this spectrum.

Robustness to calibration data: Table 6 examines whether the choice of calibration dataset affects downstream performance. On Llama-3.1-8B, FASA calibrated on the "Base" dataset (Qasper, the default) achieves average scores across six tasks (2WikiMQA, Musique, HotpotQA, Qasper, MultiFieldQA-en, NarrativeQA) ranging from 29.9 (NarrativeQA) to 55.8 (HotpotQA). Calibrating on alternative datasets (NQA, Qasp, Musi, or Self — where Self means calibrating on the evaluation dataset itself) produces scores within 0.2–1.5 points of the Base calibration on most tasks. The Coefficient of Variation (CV, standard deviation / mean) across calibration datasets ranges from 0.007 to 0.014 across tasks — less than 1.5% relative variation. For example, on Qasper, Base calibration achieves 43.7, while NQA calibration achieves 43.5, and Qasp calibration achieves 44.5. This confirms that the dominant FC identification is "stable and not reliant on a specific calibration source" — a practitioner can calibrate once on any small text sample and reuse the indices across tasks.

Extension to non-RoPE models (ALiBi and Partial-RoPE): Section 6 and Tables 7–8 evaluate FASA on architectures without standard full-RoPE. On Partial-RoPE (DeepSeek-V2-Lite-Chat, which applies RoPE to only a portion of head dimensions), Table 7 shows FASA matching or exceeding FKV across six tasks: on Qasper, FASA achieves 33.46 vs. FKV's 33.18; on 2WikiMQA, 20.25 vs. 19.83; on Lcc, 62.49 vs. 63.40 (a 0.91-point gap); on Samsum, 32.53 vs. 34.04 (a 1.51-point gap). On ALiBi (Baichuan-13B-Chat), Table 8 shows FASA achieving 7.80 vs. FKV's 9.11 on Qasper, 21.25 vs. 24.25 on LSHT, 21.70 vs. 23.18 on Dureader, 21.50 vs. 23.00 on TREC, and 16.46 vs. 17.30 on RepoBench. The gaps are consistently 1–3 points below FKV, which is larger than on RoPE models (where gaps are typically <1 point), but still substantially better than what StreamLLM or Quest would achieve (not directly reported for these architectures). This demonstrates that the functional sparsity principle generalizes, but the optimal method for exploiting it may need architecture-specific adaptation — the paper uses the same dimension-selection approach for ALiBi as for RoPE (selecting dominant dimensions based on CA scores computed at the dimension level rather than FC level, since ALiBi does not use RoPE coupling), and the slightly larger performance gap may reflect suboptimality of this direct transfer.

Comparison with SparQ on LongBench (Figure 17): This ablation directly tests whether SparQ's heuristic of selecting key dimensions by query-vector magnitude can match FASA's FC-based approach. Under token budgets from 250 to 2000 on LongBench, SparQ's performance ranges from roughly 15% (at 250 tokens) to roughly 55% (at 2000 tokens), while FASA consistently matches FKV (roughly 50–55% across the budget range). This is a negative result for the magnitude-based heuristic: high query-vector magnitudes do not reliably identify the dimensions that carry contextual selection information. The paper notes that SparQ also "incurs significant overhead as it must re-evaluate high-magnitude dimensions for every new query" while FASA uses a one-time calibration.

Ablation on FC vs. individual dimensions (Appendix D.2): The paper found that "a pipeline based on selecting 'dominant dimensions' suffers a catastrophic performance degradation" — quantitative results are not provided in a table, but the text describes the outcome as "catastrophic." This validates the FC as the indivisible functional unit.

Ablation on using FC scores as attention weights (Appendix D.2): Similarly, "direct substitution for attention probabilities leads to a catastrophic performance degradation" — the FC-based scores S_t^{l,h} cannot be softmax-normalized and used directly because they lack the magnitude calibration from non-dominant FCs.

Dynamic difficulty adjustment for FAC budget (Section 5.2): The paper does not ablate this, but the varying budget results in Figure 6 and the N_tip vs. N_fac trade-off in Figure 5 suggest that a dynamic policy — adapting N_fac based on query difficulty or attention entropy — could further improve the precision-efficiency trade-off. This is left to future work.

Critical Assessment

Does the Evidence Support the Central Claim of "Near-Lossless Performance Under Constrained Budgets"?

The paper's primary claim is that FASA "achieves performance comparable to that of full KV cache, with reduction of less than 0.7%" and "consistently achieves near-oracle accuracy in both long-context and long-generation tasks." The experimental evidence largely supports this claim, but with important qualifications about when and where it holds.

On LongBench (Table 2), FASA's average gap from FKV across five model configurations is 0.54 points — consistent with the "< 0.7%" claim. However, per-task variance is larger: on individual tasks like MultiFieldQA-en (Llama-3.2-3B), FASA achieves 49.9 vs. FKV's 50.4 (a 0.5-point gap), but on NarrativeQA (Mistral-7B), FASA achieves 29.9 vs. FKV's 29.1 (exceeding FKV by 0.8). The claim of "near-lossless" is most accurate in aggregate; individual tasks may show gaps of 1–2 points, particularly on the most challenging subsets. The comparison to baselines is stark — SnapKV's average gap of 4.38 points represents a fundamentally different regime of degradation — but "near-lossless" should be understood as "within statistical noise of full-KV performance on most tasks" rather than "literally zero degradation."

On long-CoT reasoning (Table 3), the claim needs more careful qualification. At the tightest budgets (300 tokens for MATH500, 500 tokens for AIME24), FASA shows substantial gaps from FKV: on Llama-8B MATH500, 62.2% vs. 72.4% (10.2-point gap); on Qwen-32B AIME24, 60.7% vs. 72.8% (12.1-point gap). These are not "near-lossless" — they represent meaningful accuracy degradation, even though they dramatically outperform baselines (SnapKV at 21.6% and 10.0% respectively). The claim holds as budgets increase: at 1000 tokens on MATH500, FASA is within 0.6–1.2 points of FKV across all three reasoning models, which is genuinely near-lossless. The qualification is that the budget required for near-lossless performance depends on task difficulty — easy tasks (or larger models) achieve it at lower budgets; hard tasks (or smaller models) require more tokens.

The perplexity results (Figure 4) most cleanly support the claim: FASA's curves visually overlap with FKV and Oracle across all sparsity levels, corpora, and models. The quantitative gap is consistently small (< 1 perplexity point) even at aggressive compression (token sparsity 0.1).

Does the Evidence Support the Claim of Universality and Task-Invariance of Dominant FCs?

The universality claim is supported by the visual evidence of heatmaps across model families, scales, and training paradigms (Figures 1, 10, 11), and the cross-task overlap analysis (Table 10, >70% overlap). However, the universality evidence has a significant gap: all tested models use RoPE. The extension to ALiBi and Partial-RoPE (Section 6) is preliminary — only one model per architecture (Baichuan-13B-Chat for ALiBi, DeepSeek-V2-Lite-Chat for Partial-RoPE) with limited task evaluation. The functional sparsity patterns in Figures 8–9 look qualitatively different from the RoPE heatmaps (more diffuse, less clearly concentrated in a few bright bands), and the performance gaps in Tables 7–8 (1–3 points below FKV) are larger than on RoPE models (<1 point). This suggests that while functional sparsity may be a general phenomenon, FASA's specific method for exploiting it (FC-level selection) may be optimal only for RoPE. The paper acknowledges this implicitly by treating the non-RoPE extension as a separate section rather than integrating it into the main results.

The task-invariance claim is well-supported by Table 6 (CV < 0.015 across calibration datasets) and Table 10 (>70% overlap). The single-sample calibration used in practice is a stronger test: if dominant FCs can be reliably identified from one example, they are genuinely task-invariant rather than merely correlated across tasks. However, a missing ablation is calibration on out-of-domain data: the calibration datasets (Qasper, GovReport, Musique, etc.) are all from LongBench, which shares a common distribution of English text. It is unclear whether calibration on, say, code or mathematical notation would produce the same dominant FC sets — the paper's claim of task-invariance is tested only within the LongBench task distribution, not across fundamentally different data modalities.

Does the Evidence Support the 2.56× Speedup Claim?

The speedup measurement (Figure 7) is at 64K sequence length with N_tip = 16 using FASA-C. The 2.56× figure is a single datapoint, and the paper does not report speedups across a range of batch sizes, GPU architectures, or precision formats. The speedup is measured against FKV with FlashAttention — a strong baseline already highly optimized for memory efficiency. The paper does not report whether the speedup is sustained under batching (where memory bandwidth saturation may reduce the relative benefit of accessing fewer dimensions) or whether it varies across GPU generations (A100 vs. H100, where memory bandwidth differs). The theoretical speedup of 8× (from the memory movement analysis) suggests the implementation has substantial overhead — the 2.56× empirical speedup implies that only about 32% of the theoretical bandwidth reduction translates to wall-clock improvement. A breakdown of where the remaining overhead goes (TIP computation, gather operations, kernel launch overhead, FlashAttention's existing efficiency) would strengthen the claim and guide optimization efforts.

Genuine Weaknesses in the Experimental Design

Single calibration sample, no cross-validation on FC selection stability. The paper calibrates on a single sample from Qasper. While Table 6 shows that calibration dataset choice matters little, it does not ablate calibration sample size — is one sample sufficient, or would 10 samples produce more stable dominant FC sets? The standard deviation of CA scores across samples within a dataset is not reported; without this, we cannot assess whether the dominant FC ranking from a single sample is reliable or subject to sampling noise.

No confidence intervals anywhere. Every number in Tables 2–8 is a point estimate. For a 500-question test set (MATH500) or a 500-example benchmark (LongBench test split), sampling error could be non-trivial. A 1-point difference on a 500-sample set with binary accuracy has a standard error of approximately sqrt(p(1-p)/500) ≈ 0.022 at p=0.5 — meaning 95% confidence intervals span roughly ±4.4 points. The claimed <0.7% average gap from FKV could be statistically indistinguishable from zero, but we cannot tell from the reported data. This is particularly important for the per-task breakdowns, where sample sizes are smaller (individual LongBench tasks may have 100–200 examples).

The Oracle baseline is computed with full-head scores, not with FASA's FC-based approximation. In Table 2, Oracle accuracy is computed by retaining the top-K tokens according to the full-head attention scores — this is the upper bound for any method that selects N_fac tokens, assuming perfect importance prediction. FASA's gap from Oracle (rather than from FKV) is the more honest measure of its prediction quality. On Llama-3.2-3B, Oracle averages 42.4 vs. FKV's 42.2; FASA averages 41.5 — a gap of 0.9 from Oracle. On Meta-Llama-3.1-8B, Oracle is 48.7 vs. FKV's 48.7; FASA is 48.2 — a 0.5-point gap. These Oracle-FASA gaps are small but non-zero, confirming that FASA's FC-based prediction is not perfect. The paper could strengthen its analysis by decomposing this gap: how much is due to FC selection (picking the wrong dominant FCs) vs. budget constraints (what if Oracle also used only N_tip FCs?) vs. irreducible approximation error (can any N_tip-dimensional subspace match full-head ranking?).

No comparison to learned token importance predictors. The paper asks "Can a token predictor achieve query-awareness without resorting to costly training?" but never compares against a trained predictor at the same computational budget. TokenButler (Akhauri et al., 2025) is cited but not evaluated. A head-to-head comparison — even on a single model — would contextualize FASA's performance relative to the alternative paradigm. The comparison to SparQ (Figure 17) is useful but SparQ is a heuristic method, not a learned predictor.

Memory and latency figures (Figure 7) are from a single sequence length sweep on unspecified hardware. The GPU model, batch size, precision, and FlashAttention version are not specified in the main text. The memory numbers are absolute (GB) but depend on model size; the paper uses Llama-3.1-8B for the efficiency analysis but does not state this in the figure caption. Without hardware specification, the latency numbers are not reproducible or comparable to other work.

The FLOPs-matched comparison in Section 7 uses the 14× figure. This figure appears in the Executive Summary material but is not grounded in the Experimental Analysis section — the paper does not report FLOPs-matched experiments comparing FASA-enhanced inference against a larger model with greedy decoding (as the reference example paper does). This is an omission: the Executive Summary claims "2.56× speedup using just 18.9% of the cache on AIME24," but this is a latency speedup on a single model, not a FLOPs-matched comparison against a larger pretrained model.

Missing Experiments That Would Strengthen the Paper

Ablation on N_tip across the full model, not just selected heads. The paper uses a uniform N_tip = 16 across all layers and heads, but the CA heatmaps (Figures 1, 13) show substantial layer-wise and head-wise variation in dominant FC patterns. Some heads have very clean dominant FC bands; others are more diffuse. An ablation varying N_tip per layer (e.g., fewer FCs in early layers, more in later layers) could improve the precision-efficiency trade-off. The compatibility with PyramidKV (Table 4) shows that layer-wise variation in N_fac helps; layer-wise variation in N_tip might help similarly.

Evaluation on retrieval-based long-context tasks. LongBench includes tasks like Passage Retrieval and Passage Count that explicitly require identifying specific information from long contexts. These are the hardest test of token eviction — if the critical token is a single number in a 10,000-token document, missing it means complete failure. The paper reports 2WikiMQA and HotpotQA (multi-doc QA) but does not isolate the retrieval-specific tasks for analysis.

Latency breakdown by stage. The paper reports total speedup but does not decompose it into TIP time, gather time, and FAC time. Knowing what fraction of the remaining overhead comes from each stage would guide optimization — if TIP dominates, reducing N_tip further would help; if gather/scatter dominates, memory coalescing optimizations would be the priority.

Ablation on the calibration budget K vs. downstream task performance. Table 5 shows CA scores are robust to K, but CA is an intermediate metric. Does downstream accuracy on, say, MATH500 change if dominant FCs are calibrated with K=128 vs. K=1024? This ablation would connect the CA metric more directly to task performance.

Evaluation on non-English text. All benchmarks are English. Since RoPE's frequency base and the resulting FC functional roles are language-agnostic (they depend on the mathematical structure, not training data language), FASA should theoretically work for any language. Demonstrating this — even on a small multilingual benchmark — would strengthen the universality claim.

In summary, the experimental evidence strongly supports FASA's effectiveness on the specific benchmarks and models tested, with the qualification that "near-lossless" performance requires adequate token budgets that grow with task difficulty. The universality and task-invariance claims are well-supported within the RoPE ecosystem but the extension to non-RoPE architectures is preliminary and shows larger performance gaps. The efficiency claims (2.56× speedup, 8× memory reduction) are directionally supported but lack the hardware-level detail and ablation necessary to assess generalizability across deployment scenarios. The absence of confidence intervals, the single calibration sample approach without stability analysis, and the lack of comparison to learned predictors are the most significant limitations of the experimental design.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Numbers

The compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The authors acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is a significant gap between the reported gains and what a practitioner would experience in deployment. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). For a single question, generating 2048 samples costs 8–16× more than the largest budget FASA is evaluated with, entirely before the strategy itself is deployed.

The consequence: The reported 4× efficiency gains over best-of-N (Figures 4, 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment where difficulty is unknown a priori, the total cost would be:

Total cost = cost(difficulty estimation) + cost(strategy execution)

If the difficulty estimation cost dominates, the actual end-to-end efficiency could be worse than the uniform best-of-N baseline the paper compares against. The paper briefly notes this is "a key avenue for future work," but does not explore any cheap difficulty estimation alternatives (e.g., using only 4–8 initial samples, or training a lightweight classifier from question text). Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency under the assumption of free difficulty estimation, not as a realized deployment gain.

What evidence exists: The difficulty estimation cost is explicitly quantified nowhere in the paper — we must infer it from the statement about 2048 samples and the budget ranges (1–512 generations) used in figures. Figure 4 shows that compute-optimal scaling with predicted difficulty bins (which still require 2048 samples + PRM scoring) tracks oracle difficulty closely, but neither curve accounts for the 2048-sample overhead. The paper never reports the total FLOPs cost of difficulty estimation + strategy execution relative to a fixed-budget baseline.

Mitigation status: The authors flag this explicitly as a limitation (Section 3.2, Section 8) and suggest training models to predict difficulty directly from question text as future work. However, no such model is developed or evaluated. The paper does not explore adaptive difficulty estimation (using early samples to inform later allocation), which could amortize estimation into the problem-solving process itself.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, and the Larger Model Gets No Test-Time Compute

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both data and parameters are scaled proportionally (Hoffmann et al., 2022). The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger baseline model is evaluated with greedy decoding only — no test-time compute augmentation of any kind (no best-of-N, no beam search, no revisions).

The consequence: Both choices make the pretraining baseline weaker than it needs to be, potentially inflating the apparent advantage of test-time compute. A compute-optimally trained larger model (Chinchilla-style, scaling parameters and data together) would likely outperform a parameter-only-scaled model at the same total FLOPs. Furthermore, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or best-of-16) would create a stronger baseline that could significantly narrow or reverse the reported advantages. The paper's own logic — that test-time compute amplifies existing capability — applies equally to the larger model. The current comparison essentially asks: "Is compute-optimal test-time scaling with a small model better than a naïvely-trained large model with no test-time compute?" The answer might be different for "compute-optimal test-time scaling with a small model vs. a well-trained large model with modest test-time compute."

What evidence exists: The paper reports the 14× comparison in Figure 9 and the bar charts in Figure 1. The authors note the parameter-only scaling caveat explicitly (Section 7). However, no ablation tests a stronger baseline (e.g., larger model with best-of-N, or a Chinchilla-trained larger model). The qualitative pattern — test-time compute wins on easy problems, loses on hard problems — is likely robust to baseline choice, but the magnitude of the advantage (e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions; Figure 1, top-right) is likely overstated.

Mitigation status: The paper is transparent about the parameter-only scaling choice and frames it as "representative of a canonical approach." However, the absence of any test-time compute for the larger model is not explicitly justified — it seems to be a simplification for experimental tractability rather than a principled choice. The authors flag compute-optimal pretraining as future work but do not discuss the missing test-time compute for the larger baseline. A fairer comparison would give both models a test-time compute budget and compare the slopes of their scaling curves.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability That Is Not Present

Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and budgets up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, while the 14× larger model's performance (stars) sits substantially higher.

The consequence: This establishes a hard boundary condition on the paper's central thesis that test-time compute can substitute for pretraining. The finding is clear: test-time compute amplifies existing capability but does not create it. If the base model's pass@1 rate on a problem class is near zero — meaning there are essentially no correct solutions in the proposal distribution to find or refine — then no amount of search or revision will help. The compute-optimal allocation policy, for all its sophistication, achieves nothing on bin 5 problems because there is nothing to find. This has direct implications for deployment: if a use case involves genuinely novel reasoning (problems outside the base model's training distribution), investing in test-time compute yields zero return — pretraining a larger or better model is the only viable path.

What evidence exists: The bin 5 results are consistent across all experiments. Figure 3 (right, bottommost line): PRM search achieves ~1–3% on bin 5 across all budgets. Figure 7 (right, bin 5): revisions achieve ~2–3% regardless of ratio. Figure 9 (bin 5): compute-optimal scaling flatlines near 0–5% even at the highest budgets, while the 14× larger model achieves substantially higher (exact number depends on R value). The paper is candid about this, noting in the Section 7 takeaway that "for the hardest problems (bin 5), no method makes meaningful progress" and that pretraining is "almost always more effective" on hard questions.

Mitigation status: The paper acknowledges this limitation clearly and frames it as a finding rather than a weakness — it is a characterization of when test-time compute helps versus when it does not. No mitigation is proposed because the limitation is fundamental: it reflects the base model's capability ceiling. The practical implication is that difficulty estimation is doubly important — it not only selects the optimal strategy but also identifies when no strategy will work, allowing the system to escalate to a human or a larger model.

Single Benchmark (MATH), Single Model Family (PaLM 2-S*), and a Small Test Set (500 Questions)

All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim cannot be verified without replication on other model families and benchmarks.

The consequence: Several aspects of the findings could be model-specific or benchmark-specific:

• PRM over-optimization behavior (Figure 3, right) depends on the verifier's calibration relative to the base model's output distribution. A model with different error patterns or a PRM trained differently might exhibit different over-optimization thresholds, changing which search algorithms are optimal at which budgets.
• Revision model training depends on the base model's ability to produce correct solutions at some non-trivial rate (to construct trajectories) and to learn from in-context incorrect examples. Models with different in-context learning capabilities or different pass@1 rates on MATH might show different revision scaling behavior.
• Difficulty bin boundaries — the five bins are defined by PaLM 2-S*'s pass@1 distribution on MATH. A different model or different benchmark would produce different bin assignments, and the optimal strategies might shift. A problem that is bin 3 (medium) for PaLM 2-S* might be bin 1 (easy) for a stronger model or bin 5 (hard) for a weaker one.
• Task domain: MATH consists of competition-level symbolic math problems. The difficulty-dependent patterns observed (beam search hurts easy problems, revisions help easy problems) might not generalize to, say, code generation (where executing generated code provides a natural verifier), factual QA (where correctness depends on knowledge, not reasoning), or open-ended generation (where there is no ground-truth correctness signal).

The test set size of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a very small sample for strategy selection — the optimal strategy identified on 50 questions might not be the true optimum, and the paper reports no confidence intervals to assess how much the selected strategies might vary with different random splits.

What evidence exists: The paper acknowledges the single-model limitation implicitly (by not claiming universality) and explicitly calls for "future work to replicate on other model families and benchmarks" (Section 8). However, the test set size and cross-validation implications are not discussed. The paper does report that oracle and predicted difficulty bins yield similar strategies (Figures 4, 8), which provides some robustness evidence, but the small sample size means the compute-optimal curves themselves have unquantified variance.

Mitigation status: The benchmark and model scope is stated transparently. No mitigation is provided beyond the authors' stated belief in PaLM 2-S*'s representativeness. A natural extension — evaluating on additional benchmarks (e.g., GSM8K for math, HumanEval for code) and model families — is flagged as future work.

Revisions and Search Are Studied Independently, Not Combined — The Full Potential of the Framework Is Not Explored

The paper studies two complementary mechanisms — PRM-guided search (modifying the verifier) and iterative revisions (modifying the proposal distribution) — but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence: This is a significant gap because the two mechanisms have complementary, difficulty-dependent strengths. Revisions (proposal modification) are most effective on easy problems where the base model's output is roughly correct and needs refinement — a local search in answer space. PRM search (verifier optimization) is most effective on medium-hard problems where the model needs broad exploration of qualitatively different strategies — a global search. The paper demonstrates that each mechanism individually recovers 4× efficiency gains over best-of-N, and the difficulty-bin analysis (Figures 3 right, 7 right) shows they are maximally effective in different difficulty regimes. This suggests that combining them — using the revision model as the proposal distribution within PRM-guided beam search, or using the PRM to guide which revision branches to pursue — could yield gains beyond either method alone.

The current results therefore represent a lower bound on what a fully integrated system could achieve. The paper's compute-optimal allocation (Figures 4, 8) selects the best single mechanism per difficulty bin, but does not explore whether a hybrid strategy (e.g., beam search over revision-generated candidates) would outperform either mechanism individually on the same problems.

What evidence exists: The complementary strengths are visible in the per-difficulty-bin analyses: revisions excel on bin 1–2 (Figure 7 right, easy bins show highest accuracy with fully sequential), while beam search excels on bins 3–4 (Figure 3 right, beam search outperforms best-of-N on medium bins). The paper never runs an experiment where both mechanisms are active simultaneously (e.g., the revision model generates candidates that are then scored and filtered by the PRM, or PRM search is applied at each revision step). Section 8 lists this as the first item in future work.

Mitigation status: The paper is transparent about this limitation and explicitly calls for combination experiments. The difficulty-dependent analysis (Section 5.3, 6.2) provides the intellectual justification for why combination should help, but no empirical evidence is offered. This is not a failure of the existing experiments — the paper's contribution is the systematic analysis of each mechanism independently — but it means a practitioner who implements both search and revisions and combines them may achieve results that this paper does not predict.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, and the Mitigation (Chain-Wide Selection) Is an Incomplete Patch

As noted in Section 6.1, approximately 38% of correct answers produced during a sequential revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction procedure: the model was trained only on sequences where all in-context answers are incorrect, followed by a correct target answer. During training, the model never sees a scenario where the current answer in context is already correct and should be preserved — it only learns to produce a correct answer given that all previous answers are wrong.

The consequence: At test time, the revision model has no learned behavior for what to do when it encounters a correct answer in its own revision chain. It tends to "fix" things that are not broken, undoing its own successful revisions. This creates a fundamental tension in the sequential revision approach: each additional revision step increases the chance of producing a correct answer (the model improves) but also increases the chance of overwriting a previously correct answer with an incorrect one. The 38% reversion rate means that a 4-step revision chain has roughly:

P(at least one correct answer in chain) ≈ 1 - (1 - p₀)⁴

but also:

P(final answer is correct) ≠ P(at least one correct in chain)

because later steps can corrupt earlier correct answers.

What evidence exists: The 38% figure is reported in Section 6.1, and the paper mitigates this by using majority voting or verifier-based selection across the entire chain — picking the best answer from any point in the chain rather than always taking the last revision. Figure 6 (right) shows that sequential + best-of-N weighted achieves ~41.5% vs. fully parallel at ~39% at 64 generations, confirming that chain-wide selection partially recovers the gains despite reversion. However, the paper does not report what accuracy would be if reversion did not occur (e.g., if a perfect selection mechanism always picked the best answer in the chain, or if the model never reverted correct answers). The 41.5% figure is therefore a mix of the revision model's genuine improvement capability and the selection mechanism's ability to catch and preserve correct answers before they are overwritten.

Mitigation status: The paper acknowledges the reversion problem and proposes chain-wide selection as a mitigation, but this is an incomplete solution. Chain-wide majority voting or verifier selection can miss correct answers (if the verifier is imperfect) or select answers that are correct but later in the chain (wasting the compute spent on subsequent revisions). A more principled solution — such as training the model to recognize when the current answer is already correct and output a "no revision needed" token, or using a separate stopping criterion based on verifier confidence — is not explored. The ReST^{EM} experiment (Appendix K, Figure 16) shows that attempts to further optimize the revision model with RL-style training actually worsen the reversion problem, making sequential revisions substantially degrade performance — suggesting the reversion behavior is not easily fixed by more training alone.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper instigates a conceptual shift in how the field approaches KV cache compression: from detecting sparsity post-hoc to exploiting sparsity that is architecturally guaranteed. Prior work on token eviction operated under the implicit assumption that attention sparsity is an emergent phenomenon — a byproduct of training dynamics and data distribution — that must be measured empirically at runtime through heuristics like accumulated attention scores (SnapKV), page-level similarity (Quest), or heavy-hitter tracking (H2O). FASA demonstrates that a substantial fraction of this sparsity is not emergent but encoded in the mathematical structure of RoPE itself: the functional heterogeneity of frequency chunks — where fewer than 1% of FCs carry contextual selection information while ~90% construct query-independent positional patterns (Table 9) — is a direct consequence of how RoPE parameterizes position through frequency-dependent rotations. This is not a behavioral observation; it is an architectural invariant.

The magnitude of this shift should be calibrated carefully. This is not a paradigm overthrow — it does not challenge the transformer architecture or propose an alternative to attention. Rather, it is a reframing of the token eviction problem from "how do we approximate attention scores cheaply?" to "which dimensions of the attention computation encode the query-dependent signal, and can we compute only those?" The paper provides both the diagnostic instrument to answer this question (the Contextual Agreement metric, Equation 4) and a complete method (FASA) that operationalizes the answer for RoPE-based models. The reframing matters because it converts a problem that previously seemed to require either expensive runtime computation or learned models into a structured subspace projection problem with a one-time, negligible-cost calibration.

This reframing reconciles a latent tension in the token eviction literature. Static methods (StreamLLM) succeed at preserving positional patterns (recency bias, attention sinks) because those patterns are robustly encoded by the non-dominant FCs — which is why retaining only initial and recent tokens works for tasks where positional heuristics suffice. But these same methods fail catastrophically on tasks requiring content-based retrieval from arbitrary positions (e.g., finding a specific fact buried in a long document) because the contextual FCs — the ones that would flag those intermediate tokens as important — are ignored by a purely position-based retention policy. FASA explains why StreamLLM's failure mode is systematic rather than incidental: the non-dominant FCs that StreamLLM implicitly relies on are architecturally incapable of encoding query-dependent relevance. Similarly, SnapKV's one-time prefill-stage importance estimation cannot adapt to the evolving relevance of tokens as decoding progresses because it uses aggregated attention scores that conflate positional and contextual contributions — again, an architectural limitation that FASA's decomposition makes explicit.

A significant secondary effect is that this work redirects research attention toward understanding the functional organization of position encodings rather than designing increasingly sophisticated heuristics to detect sparsity. The paper's finding that functional sparsity appears in ALiBi and Partial-RoPE (Section 6) — albeit at different granularities (head-level for ALiBi, dimensional partition-level for MLA) — suggests that the division of labor between positional and contextual information may be a general architectural principle of position encoding schemes, not a RoPE-specific curiosity. This opens a new research program: characterizing the functional decomposition induced by each position encoding mechanism, identifying which components carry semantic vs. positional information, and designing compression methods that exploit this decomposition directly. The paper provides both a template (the CA metric + offline calibration) and a cautionary tale (the FC is indivisible for RoPE; individual dimension selection fails catastrophically — Appendix D.2) for how to pursue this program.

Concretely, several research directions become more attractive:

• Architecture-aware compression methods that tailor their approach to the specific position encoding scheme rather than treating all transformers as interchangeable. FASA shows that RoPE's frequency-chunk structure can be exploited with FC-level granularity; ALiBi might require head-level or bias-term-level granularity; Partial-RoPE might require hybrid approaches.
• Verifier robustness and over-optimization (from the reference example paper's framework) becomes less central for FASA-style methods because the token importance prediction is not a learned verifier that can be over-optimized — it is a direct computation in a frequency subspace whose quality is determined by the one-time calibration, not by iterative search. This shifts the bottleneck from "how do we prevent the verifier from being exploited?" to "how do we identify the minimal sufficient subspace for contextual selection?"
• Learned token importance predictors (TokenButler, AttentionPredictor) become less attractive as a general-purpose solution because FASA demonstrates that for the dominant position encoding scheme in modern LLMs, a training-free approach achieves near-oracle accuracy. The burden of proof shifts to learned methods: they must demonstrate that their additional complexity and generalization risk are justified by gains over FASA's structurally-grounded approach, rather than over heuristic baselines.

Conversely, some directions become less promising:

• Magnitude-based dimension selection heuristics (as in SparQ) are shown to be a "poor substitute for true contextual awareness" (Figure 17), and the paper provides a mechanistic explanation (they sever RoPE's positional encoding by operating on individual dimensions rather than FC pairs). Future work on dimension selection for sparse attention should respect the FC as the minimum functional unit.
• Single-stage methods that attempt to use low-dimensional approximations as both token selectors and attention weight approximators are ruled out by the paper's negative result that FC-based scores cannot substitute for attention weights directly (Appendix D.2). This is not a calibration issue; it is a consequence of the fact that non-dominant FCs, while query-independent for ranking, contribute to the magnitude calibration needed for proper softmax normalization. Any method operating in a reduced frequency subspace must adopt a two-stage architecture — one stage for ranking, another for full-dimensional computation on the ranked subset — or accept degraded attention weight quality.

Follow-Up Research This Work Enables

Cheap, adaptive difficulty estimation for token budget allocation. The paper demonstrates that N_tip (TIP precision) and N_fac (FAC budget) trade off against each other (Figure 5): high-precision selection with small budget achieves similar accuracy as low-precision selection with large budget. This suggests an adaptive allocation policy: estimate the "difficulty" of the current decoding step (e.g., from the entropy of the TIP score distribution, or the variance of the dominant FC contributions), and dynamically adjust N_fac — using a small budget when attention is concentrated on few tokens, and a larger budget when attention is diffuse. A concrete experiment: on LongBench, measure the correlation between TIP score entropy and the optimal N_fac for each query, then implement a threshold-based policy and compare against fixed-budget FASA. If successful, this could recover additional efficiency on easy queries while preserving accuracy on hard ones, without the expensive difficulty estimation overhead that plagues the reference example paper's approach. The key advantage over that work is that FASA's difficulty signal (TIP score distribution) is a byproduct of the token selection computation itself, requiring zero additional FLOPs — unlike the 2048-sample estimation in the compute-optimal test-time scaling framework.

Layer-wise and head-wise N_tip optimization. The paper uses uniform N_tip = 16 across all layers and heads, but the CA heatmaps (Figures 1, 13) reveal substantial variation: some heads have sharp, concentrated dominant FC bands; others have more diffuse patterns. Similarly, Figure 13 shows that different layers exhibit qualitatively different dominant FC distributions. A natural extension is to calibrate N_tip per head (or per layer) based on the "sparsity" of the CA distribution — heads with a few very high-CA FCs might need only N_tip = 4, while heads with more distributed CA might need N_tip = 20. The calibration metric could be the Gini coefficient of the per-head CA distribution, or the number of FCs needed to reach a target compound CA (e.g., 90% of the maximum achievable with full FCs). A concrete experiment: on a single model (e.g., Llama-3.1-8B), sweep N_tip per layer from 4 to 16, measure the resulting compound CA and downstream task accuracy on LongBench, and compare total FLOPs (sum of N_tip across all heads) against the uniform-16 baseline. If successful, this could reduce TIP computation by 30–50% with no accuracy loss, or improve accuracy at the same compute budget by allocating more FCs to heads that need them.

FASA as a prefill-stage filter for other token eviction methods. The paper emphasizes FASA's decode-stage operation, but the dominant FCs could also be used during prefill to make an initial, cheap pass over the full context and identify a larger candidate set of potentially important tokens — which is then refined by a more expensive method (or by FASA itself during decoding). This is the inverse of FASA's two-stage design: instead of TIP (cheap) → FAC (expensive) on the same query, use TIP during prefill to pre-filter the context from, say, 100K tokens to 10K candidates, then during decoding apply FASA or SnapKV on the 10K subset. A concrete experiment: on LongBench's 128K-context tasks (Qwen2.5-14B-1M), compare accuracy and total FLOPs for (a) FASA alone with budget 256, (b) FASA prefill-filter → FASA decode with budget 256, and (c) FASA prefill-filter → SnapKV decode with budget 256. The hypothesis: prefill filtering removes obviously irrelevant tokens (the "positional baseline" tokens that non-dominant FCs would flag), allowing the decode-stage method to operate on a cleaner, more relevant subset and achieve higher effective accuracy at the same budget. This is a straightforward combination experiment that could push the accuracy-budget frontier further, particularly for extremely long contexts.

Failure mode characterization: when do dominant FCs miss critical tokens? The paper demonstrates that dominant FCs achieve 74–82% top-20 prediction accuracy (Table 11) and that FASA's performance closely tracks Oracle (Table 2). But the remaining gap — the tokens that dominant FCs miss — is uncharacterized. A systematic failure analysis would: (a) identify tokens in the LongBench test set where FASA's selected set diverges from Oracle's (full-head top-K), (b) categorize these failures by token type (e.g., named entities, numerical values, negations, discourse markers), position (early/middle/late in context), and task type (single-doc QA vs. multi-doc QA vs. summarization), and (c) analyze whether adding specific FCs (beyond the current dominant set) would recover these tokens. This analysis would reveal whether FASA's failures are systematic (e.g., it consistently misses negations because they are encoded in high-frequency FCs) or random (sampling noise in the CA metric). If systematic, it would suggest targeted FC selection strategies (e.g., explicitly including FCs known to encode negation, even if their individual CA is low, because their interaction with dominant FCs captures critical token types). This is a diagnostic experiment, not a method improvement, but it would substantially deepen our understanding of what information each FC carries and whether the current dominant FC selection criterion (individual CA maximization) is sufficient or whether a more nuanced criterion (e.g., marginal contribution to compound CA when added to an existing set) would identify different FCs.

Stress-testing the calibration procedure: adversarial and out-of-distribution contexts. The paper calibrates on standard English text (Qasper) and evaluates on standard English tasks (LongBench, WikiText, MATH). A stress-test would calibrate on (a) code (e.g., Python from The Stack), (b) mathematical notation (LaTeX), (c) non-English text (Chinese, Arabic), and (d) adversarial sequences designed to manipulate attention patterns (e.g., repeated token sequences known to trigger attention sinks, or documents with deliberately misleading positional cues). For each calibration source, measure the overlap of dominant FC sets with the standard English calibration, and evaluate downstream FASA performance on all four context types. This would test the limits of the task-invariance claim: is it invariance within a single language's distribution, or genuine cross-modal invariance grounded in RoPE's mathematical structure? If dominant FCs shift for code or non-English text, it would suggest that while the existence of functional sparsity is architecturally guaranteed, the specific FCs that are dominant may adapt to the statistics of the training data — a more nuanced picture than "universal and task-invariant." This experiment would also reveal whether FASA degrades gracefully or catastrophically when calibration and evaluation distributions differ, which is essential for practical deployment where the deployment context may be unknown at calibration time.

Generalization to non-transformer architectures with positional encodings. The paper's Section 6 demonstrates that functional sparsity exists in ALiBi and Partial-RoPE, but the evaluation is limited to one model per architecture and shows 1–3 point gaps from FKV (Tables 7–8) — larger than on full-RoPE models (<1 point). A systematic study would: (a) characterize the functional sparsity structure in a broader set of position encoding schemes (NoPE, learned absolute positions, T5-style relative bias, xPos, etc.), (b) determine the appropriate "indivisible functional unit" for each scheme (the equivalent of RoPE's FC), (c) measure whether the sparsity is task-invariant within each scheme, and (d) evaluate FASA-style two-stage token selection on each. For ALiBi, the indivisible unit appears to be the head (since biases are head-specific but dimension-agnostic within a head), suggesting a different FASA variant: select dominant heads rather than dominant FCs, compute TIP using only those heads, then perform FAC using all heads on the selected tokens. For Partial-RoPE, the RoPE-applied partition may exhibit FC-level sparsity while the non-RoPE partition may require dimension-level or head-level selection. This research program would establish the boundary conditions for frequency-aware sparse attention and provide a taxonomy of position encoding structures organized by their functional decomposition — a foundational resource for any future method that exploits architectural sparsity for efficiency.

Practical Applications and Downstream Use Cases

On-device deployment of long-context LLMs with constrained VRAM. FASA-M's 7.5× memory reduction at 64K context (Figure 7, from ~75 GB to ~10 GB) means that models like Llama-3.1-8B with 128K context windows — which currently require high-end server GPUs — could run on consumer-grade hardware with 12–16 GB VRAM. The key enabler is FASA-M's strategy of keeping only the dominant key dimensions on GPU (32 dimensions per head out of 128, for N_tip = 16) while offloading non-dominant keys and values to CPU memory, with just-in-time transfer of only the selected tokens for FAC. A concrete scenario: a developer running Qwen2.5-7B on an RTX 4070 (12 GB VRAM) for document Q&A over 50K-token legal contracts. Without compression, the KV cache alone would exceed available VRAM. With FASA-M at budget 256, the GPU-resident cache is under 2 GB, leaving ample room for model parameters and activations. The accuracy cost on LongBench (Table 2, Qwen2.5-7B: 47.9 vs. FKV 47.8) is a 0.1-point improvement, meaning the user gets full-model accuracy on hardware that would otherwise be incapable of running the model at all. The CPU-GPU transfer latency (~0.11 seconds per step at 64K, Figure 7) is noticeable but acceptable for interactive Q&A where users expect multi-second response times for long documents; prefetching techniques (mentioned but not evaluated) could reduce this further.

Cost-efficient batch inference for long-document processing. For organizations running large-scale batch inference — e.g., summarizing thousands of legal documents, extracting structured data from medical records, or processing code repositories — FASA-C's 2.56× decoding speedup at 64K (Figure 7) translates directly to cost savings. At current cloud GPU pricing (~$2–4/hour for an A100), a 2.56× speedup reduces per-document cost by roughly 60$110–220 at typical cloud rates. The accuracy cost is negligible (Table 2, Qwen2.5-14B-1M: FASA 49.2 vs. FKV 50.3, a 1.1-point gap on average). Critically, this cost saving does not require any calibration per batch — the dominant FC indices are identified once per model and reused indefinitely — and the method is orthogonal to other optimizations like quantization or batching, meaning savings stack multiplicatively.

Long chain-of-thought reasoning in resource-constrained settings. The DeepSeek-R1 style models generate thousands of reasoning tokens before producing a final answer, with the KV cache growing linearly with generation length. On MATH500 with R1-Distill-Llama-8B, FKV generates ~3,100 total tokens with 72.4% accuracy, while FASA at a 300-token budget achieves 62.2% accuracy and generates ~3,300 tokens — a 10-point accuracy gap at the tightest budget, closing to 71.8% (within 0.6 points of FKV) at 1000 tokens (Table 3). The practical implication: a practitioner deploying an R1-style reasoning model on a GPU with limited memory (e.g., 24 GB) can choose a token budget based on their accuracy tolerance and available VRAM. At budget 700, FASA uses ~18.9% of the full KV cache (700 / ~3700 at peak context) and achieves 69.4% accuracy on Llama-8B — a 3-point gap from FKV that may be acceptable for many applications (e.g., educational math tutoring where perfect accuracy is not required and cost is a primary concern). Critically, FASA also controls output verbosity — unlike H2O, which generates 2.7× more tokens than FKV on MATH500 (Table 3, 8370 vs. 3104 total tokens), FASA's output length (3298 tokens) is nearly identical to FKV's. This means the per-step savings from KV cache compression are not eroded by inflated generation length, a subtle but practically important advantage over heuristics that distort the model's generation behavior.

Integration into inference serving systems as a modular component. FASA's design — a single-line code insertion into the FlashAttention forward pass (Figure 14), with dominant FC indices stored in a global dictionary — makes it straightforward to integrate into existing inference frameworks (vLLM, TGI, TensorRT-LLM) as an optional plug-in. A serving system could expose FASA as a configuration option: users select a token budget and optionally provide a calibration sample (or use a pre-calibrated index file shipped with the model). Because FASA is orthogonal to other KV cache optimizations — demonstrated by the +0.4–1.1 point gain when combined with PyramidKV (Table 4) — it can be deployed alongside quantization (KV cache in INT4), layer-wise budget allocation, and prefix caching without conflicts. The combination FASA + PyramidKV + quantization could compound savings: FASA reduces memory I/O by 8× in the TIP stage, PyramidKV allocates fewer tokens to later layers, and quantization reduces per-token byte size by 4× (FP16 → INT4). A concrete scenario: serving Qwen2.5-32B with 32K context on an 8×A100 cluster. Full-KV inference might support 16 concurrent requests at 32K; with FASA-C + PyramidKV + INT4 KV cache, the per-request memory footprint is reduced by ~4–8×, potentially doubling or tripling throughput while maintaining accuracy within 1 point of FKV on most tasks. The paper does not evaluate this combination, but the modular design and demonstrated compatibility make it a natural engineering extension.