TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

🎯 Pitch

TriAttention reveals that pre-RoPE query and key vectors naturally cluster around fixed centers, creating predictable, distance-based attention patterns that can replace fragile importance estimation from rotating queries. This simple trigonometric insight enables matching full-attention reasoning accuracy on AIME25 while slashing KV cache memory by over 10×, where leading baselines lose half their accuracy at the same compression rate.

1. Executive Summary

This paper introduces TriAttention, a KV cache compression method that estimates key importance from pre-RoPE Q/K vectors rather than post-RoPE attention scores, avoiding the instability that plagues observation-window-based methods like SnapKV and R-KV. The core insight is Q/K concentration — the observation that pre-RoPE query and key vectors cluster tightly around fixed non-zero centers across positions and contexts — which causes attention to follow predictable distance preferences describable by a trigonometric series (a function of Q-K distance whose coefficients are determined by the learned centers). Evaluated on AIME24, AIME25, and MATH 500 with models including Qwen3-8B, DeepSeek-R1-Distill variants, and GPT-OSS-20B, TriAttention matches Full Attention reasoning accuracy while achieving 2.5× higher throughput or 10.7× KV memory reduction on AIME25, and nearly doubles R-KV's accuracy at the same memory budget (32.9% vs. 17.5% on AIME25), establishing that stable pre-RoPE statistics enable reliable key importance estimation without relying on the narrow observation windows that limit post-RoPE methods.

2. Context and Motivation

The Core Problem: KV Cache Growth Makes Long Reasoning Infeasible

The fundamental problem this paper addresses is that extended chain-of-thought reasoning in LLMs creates a KV cache memory bottleneck that makes deployment impractical, particularly on consumer hardware. Modern reasoning models like DeepSeek-R1 and Qwen3 produce chain-of-thought sequences spanning tens of thousands of tokens — the paper cites 32K-token generation as a typical scenario (§1, §5.1). Since the KV cache stores one key-value pair per token per layer, it grows linearly with sequence length. For a model like Qwen3-8B with 36 layers and 32 attention heads, a 32K-token context produces a KV cache requiring tens of gigabytes of GPU memory — often exceeding what is available on a single consumer GPU like an RTX 4090 (24GB), as the paper demonstrates with its OpenClaw deployment scenario (Appendix J, Figure C).

This is not merely a memory issue — it is a throughput issue as well. Larger KV caches mean slower attention computation because the quadratic attention mechanism must process more tokens per step. The paper shows that on a single A100 80GB GPU, Full Attention with Qwen3-8B achieves only ~223 tokens/second on MATH 500 (§5.2.4, Table 4). For applications requiring real-time interaction or batch processing of many queries simultaneously, this throughput is a significant bottleneck.

The problem is particularly acute for long reasoning chains, which have become the dominant paradigm for solving complex mathematical, coding, and scientific problems. The key tension is: reasoning quality improves with longer chains-of-thought (more tokens = more thorough reasoning), but longer chains make the KV cache memory problem worse. Without compression, deploying these models on affordable hardware means either truncating reasoning chains (hurting accuracy) or running out of memory entirely.

Where Prior Approaches Fall Short

KV cache compression has been studied extensively (§2.2), but all prior methods share a common limitation: they operate in the post-RoPE space, estimating key importance from attention scores computed after positional encoding has been applied. The paper categorizes these methods into three families and identifies specific failure modes for each.

Heuristic methods like StreamingLLM retain a fixed pattern of tokens — initial "sink" tokens plus a sliding window of recent tokens — based on the empirical observation that early positions receive disproportionate attention regardless of content. While simple and computationally cheap, these methods cannot adapt to content-dependent importance. A token in the middle of a reasoning chain might be critical for later backtracking, but if it falls outside the sliding window, it is permanently discarded. This is particularly damaging for reasoning tasks where intermediate states must be retained for arbitrary durations.

Attention-based methods like H2O, SnapKV, R-KV, and LazyEviction improve on heuristics by using actual attention scores from recent queries to identify important tokens. The core idea is: watch which tokens receive high attention during a short observation window, then retain those tokens for future use. This approach has an inherent instability problem that the paper identifies and makes precise (§2.3):

"queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable reasoning"

Let me unpack this carefully, because it is the central technical motivation for the entire paper. RoPE encodes position by rotating Q and K vectors — each frequency band rotates at a rate $\omega_f$ per position. This means a query at position $p_q$ and a query at position $p_q + 1$ have different orientations in the vector space. When SnapKV or R-KV computes attention scores using recent queries, these scores are computed with queries that have specific positional rotations. But future queries will have different rotations because they will be at different positions.

The consequence is profound: only the most recent queries have orientations similar to future queries. As the paper puts it, this creates a "tiny observation window." A token that receives low attention during this window — perhaps because its content is not currently needed — may be evicted, even if it becomes critical later. This is especially problematic for retrieval heads, which the paper cites from Wu et al. (2025) and Xiao et al. (2025): these heads can remain dormant for long periods, then suddenly need to attend to a specific past token to retrieve information. If that token was evicted during dormancy, the retrieval fails, and the reasoning chain breaks.

The paper cites corroborating evidence from prior work: Zhang et al. (2025) found that increasing the observation window does not help — performance peaks at around 25 queries and declines thereafter. This is a striking empirical finding that the paper uses to argue the observation-window approach is fundamentally limited, not just insufficiently tuned. The decline after 25 queries suggests that older queries, with their older positional rotations, actually mislead importance estimation rather than improving it.

Norm-based methods like VATP attempt a different approach: rather than relying solely on attention scores (which can be misleading — attention sinks receive high scores but have near-zero value norms), they incorporate value vector norms as a complementary importance signal. However, the paper identifies a distinct limitation for norm-based methods in post-RoPE space (§2.3):

"they leverage only vector magnitudes while ignoring directional information. Ideally, incorporating Q/K directions would improve importance estimation—attention depends on both norms and the angle between Q and K. However, in post-RoPE space, directions are entangled with positional rotations"

This is a crucial insight. The dot product $\langle q, k \rangle$ that determines attention has two components: the magnitudes $\|q\| \|k\|$ and the cosine of the angle between $q$ and $k$ . Norm-based methods only capture the magnitude component. In post-RoPE space, the angle component is a function of both content and position — disentangling them is extremely difficult because the RoPE rotation mixes them together. If you observe that two queries attend similarly to a key, you cannot easily tell whether it is because their content is similar (the pre-RoPE vectors are similar) or because their positions are similar (the RoPE rotations are similar).

The Unifying Limitation: Post-RoPE Instability

The paper's key move is to identify a unifying limitation across all three method families: they all operate in post-RoPE space, where positional rotations have already been applied. This shared foundation creates distinct but related problems:

Attention-based methods suffer from the narrow observation window because queries rotate and only the most recent queries provide representative attention patterns.
Norm-based methods cannot exploit directional information because directions are entanglement with position after RoPE is applied.

This framing is what motivates the paper's turn to the pre-RoPE space. Before RoPE rotation is applied, Q and K vectors are purely content-dependent — position has not yet been encoded. If important patterns exist in this pre-RoPE space that are stable and interpretable, they could provide a foundation for importance estimation that avoids the instability of post-RoPE methods entirely.

Prior Work on Pre-RoPE Structure

The paper does not claim to be the first to notice structure in pre-RoPE vectors. The observation that Q/K vectors exhibit non-random distributions is implicit in prior work on attention mechanisms. However, the paper's contribution is to characterize this structure systematically — as a concentration phenomenon quantified by Mean Resultant Length — and, more importantly, to connect it directly to attention behavior via the trigonometric series.

Prior work had separately observed:

That certain attention heads exhibit distance-dependent attention patterns (attention sinks, local attention windows) — but without explaining WHY these patterns emerge from learned parameters.
That Q/K vectors can be decomposed into frequency bands where different heads specialize (Zhou et al., 2025) — but without connecting this to a global concentration property.
That value norms provide useful importance signals (Guo et al., 2024; Kobayashi et al., 2020) — but without integrating this with directional information.

The paper unifies these scattered observations under a single framework: Q/K concentration in pre-RoPE space is a model-intrinsic property (not task-dependent) that causes attention to follow predictable distance preferences via a trigonometric series, and both the distance preference (from Q/K centers) and the norm information (from Q/K magnitudes) can be combined for reliable key importance estimation.

How This Paper Positions Itself

The paper's positioning is clear and bold: post-RoPE KV compression methods are fundamentally limited by observation-window instability, and the pre-RoPE space offers a more stable foundation. This is not an incremental improvement over SnapKV or R-KV — it is presented as a different category of method altogether.

The positioning has three key pillars:

Pillar 1: Stability over recency. Instead of relying on recent queries (which have limited representativeness due to rotation) to estimate future attention, TriAttention uses the Q/K centers computed from calibration data. These centers are stable because they are computed in pre-RoPE space, before positional rotation is applied. The paper explicitly claims (§4) that this avoids the "tiny observation window" problem entirely.

Pillar 2: Mathematical grounding. Rather than treating attention patterns as empirical phenomena to be observed (as attention-based methods do), the paper derives them from first principles: when Q/K are concentrated, the RoPE attention formula reduces to a trigonometric series in Q-K distance. This makes importance estimation predictable — you can compute which distances are preferred from the Q/K centers alone, without ever observing actual attention scores. This is a qualitatively different approach from methods that must observe attention to learn which tokens are important.

Pillar 3: Complementary signals with principled weighting. TriAttention combines two score components: $S_{trig}$ (distance preference from the trigonometric series) and $S_{norm}$ (norm-based importance). Rather than treating these as ad-hoc features, the paper derives a principled weighting scheme based on Q/K concentration: when concentration is high (high Mean Resultant Length $R$ ), $S_{trig}$ dominates because the trigonometric series is accurate; when concentration is low, $S_{norm}$ becomes more important because there is more variation around the centers that the series cannot capture. This adaptive weighting (§4.2) is directly motivated by the concentration analysis in §3.

The Practical Stakes

The paper motivates its work with a concrete deployment scenario that illustrates the practical stakes (§5.2, Appendix J). The OpenClaw deployment on a single RTX 4090 with Qwen3-32B (INT4 quantized) shows that without KV compression, multi-turn interaction causes an out-of-memory error before the agent can complete its task. The first request alone exceeds 15K tokens, and each subsequent round expands the context further. With Full Attention, the GPU runs out of memory. With TriAttention, the agent completes the entire session within the 24GB budget.

This is not an artificial benchmark — it is a realistic deployment of a reasoning agent on consumer hardware. The paper is essentially arguing: if you want to run state-of-the-art reasoning models on affordable GPUs (as opposed to datacenter A100/H100 clusters), you must compress the KV cache, and existing compression methods degrade accuracy too severely to be usable. TriAttention is positioned as the method that makes this deployment scenario viable.

The efficiency numbers reinforce this: on AIME25 at equivalent accuracy to Full Attention (40.8%), TriAttention achieves 2.5× higher throughput (564 vs. 223 tokens/s) and 10.7× KV memory reduction (§5.2.4, Figure 1). These are not marginal improvements — they are the difference between a model being usable interactively versus being too slow, or fitting on a GPU versus running out of memory.

Addressing Potential Skepticism

The paper implicitly addresses several reasons why one might be skeptical of its approach:

"Why would Q/K vectors be concentrated?" The paper does not provide a theoretical explanation — it presents concentration as an empirical observation, validated across three model architectures (Qwen3, Qwen2.5, Llama3) and two attention mechanisms (GQA and MLA, Appendix I). The finding that ~85-97% of heads have Mean Resultant Length $R > 0.95$ (Figure 2C, Table G) is presented as a strong empirical regularity, not a theoretically predicted property. The cross-architecture and cross-domain stability (Appendix H: calibration on coding vs. reasoning yields comparable results) further supports the claim that this is model-intrinsic rather than task-specific.

"Doesn't calibrating on a small dataset overfit?" The calibration sensitivity analysis (Appendix H, Table F) addresses this directly: performance is stable from 50K to 960K calibration tokens, and even using Google homepage HTML as calibration data achieves accuracy comparable to using high-quality chat data. This is a strong robustness result — it suggests the Q/K statistics are so stable that calibration data quality barely matters. The paper frames this as evidence that Q/K concentration captures a property of the model architecture and pretraining, not a property of any specific task domain.

"How can you predict attention without observing any actual attention?" This is addressed by the reconstruction correlation analysis in §3.3 and Figure 3. The paper shows that the trigonometric series computed from Q/K centers achieves a Pearson correlation of 0.5-0.9 with actual attention logits across most heads, with mean values above 0.5 across all tested architectures. This is not perfect prediction (the correlation is not 1.0), but it is sufficient for importance estimation — the paper shows that the remaining variation is captured by the norm-based complement. The ablation in Table 3A confirms that removing $S_{trig}$ causes a catastrophic accuracy drop (from 42.1% to 18.8% on AIME24), validating that the distance preference signal is essential even if imperfect.

Summary of the Gap and Positioning

The paper addresses a gap that can be summarized as: post-RoPE KV cache compression methods have an inherent stability problem caused by positional rotation of queries, which limits their observation window to ~25 tokens and causes important keys to be lost during long reasoning chains. The paper's response is not to improve the observation window (which prior work showed is futile beyond a certain point) but to abandon the post-RoPE space entirely and instead exploit the stable structure of pre-RoPE Q/K vectors — specifically their concentration around fixed centers — to estimate importance through a mathematically derived trigonometric series. This is positioned as a fundamentally different category of method, with stability guarantees that observation-based methods cannot provide.

3. Technical Approach

3.1 Reader Orientation

TriAttention is a KV cache compression system that decides which key-value pairs to keep in memory during long text generation. It solves the problem of KV cache memory bottlenecks by scoring each cached key with a fixed mathematical function — derived from pre-RoPE Q/K statistics — rather than relying on recent attention scores that become unstable as the sequence grows. The "shape" of the solution is: measure Q/K centers once during calibration, then at inference time plug these centers into a trigonometric series that predicts which key positions matter, combine with norm information weighted by concentration, and retain only the top-scoring keys.

3.2 Big-Picture Architecture (Diagram in Words)

The TriAttention system has five major components connected in a pipeline:

Offline Calibration — runs once before deployment. Feeds a calibration dataset through the model, collects all pre-RoPE Q and K vectors across all heads and positions, and computes per-head statistics: the Q center $E[q_f]$ and expected norm $E[\|q_f\|]$ for each frequency band $f$ , and the Mean Resultant Length $R_f$ quantifying how concentrated the vectors are.
Trigonometric Series Scorer ( $S_{trig}$ ) — at inference time, for each cached key, computes a score based on the distance $\Delta$ between the key and potential future query positions. Uses the pre-computed Q centers and the actual key vector plugged into the RoPE attention formula approximated as a trigonometric series. This captures distance preference: heads that favor local attention, attention sinks, or intermediate distances get a scoring curve that peaks at those preferred distances.
Norm-Based Scorer ( $S_{norm}$ ) — computes a complementary score from Q/K magnitudes alone: $E[\|q_f\|] \cdot \|k_f\|$ , weighted by $(1 - R_f)$ so it contributes more when Q/K concentration is weaker (the trigonometric series approximation is less reliable). This captures token salience independent of position: low-norm keys contribute little regardless of distance.
Adaptive Combiner — adds the two scores together: $S(k, \Delta) = S_{trig}(k, \Delta) + S_{norm}(k)$ . The weighting is automatic because $R_f$ is baked into $S_{norm}$ — when concentration is high, $R_f \to 1$ makes the norm term vanish; when concentration is low, $(1 - R_f)$ preserves the full norm contribution.
KV Cache Pruner — triggered every 128 tokens (a "window"). Scores all currently cached keys using the combined scorer evaluated at multiple future offsets $\delta \in \{1, 2, 4, 8, ...\}$ , averages the scores, normalizes within each query head (for GQA models), takes the maximum across shared heads, and retains only the top- $B$ keys by final score. Everything else is evicted.

Information flows as follows during inference: a new token is generated → its KV pair is appended to the cache → every 128 tokens, if the cache exceeds budget $B$ , the pruning procedure is triggered → all cached keys are scored by the Trig + Norm combiner using the pre-computed Q centers → keys are ranked and the bottom ones are evicted → generation continues with the reduced cache.

3.3 Roadmap for the Deep Dive

First, the mathematical foundation: how RoPE attention is expressed in complex form and why Q/K concentration causes the logit to collapse to a trigonometric series in $\Delta$ . This is the core theoretical insight.
Second, the Q/K concentration phenomenon itself — how it is measured (Mean Resultant Length), how prevalent it is, and the reconstruction correlation validating that the series predicts real attention. This establishes the empirical basis for the method.
Third, the TriAttention scoring function: the trigonometric series term $S_{trig}$ and the norm term $S_{norm}$ in detail, including why each term has the form it does.
Fourth, the adaptive weighting mechanism: how $R_f$ controls the balance automatically.
Fifth, the KV cache pruning procedure: window-based scheduling, averaging over future offsets, GQA aggregation via z-score + max, and how the budget $B$ is enforced.
Sixth, the calibration procedure: offline computation of Q centers, expected norms, and Mean Resultant Lengths.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper proposing a new inference-time method whose core idea is that pre-RoPE Q/K concentration causes attention to follow predictable distance preferences via a trigonometric series, and that this predictability enables stable key importance estimation that avoids the observation-window bottleneck of post-RoPE methods.

Mathematical Foundation: RoPE Attention as a Trigonometric Series

The entire method rests on a specific mathematical relationship between RoPE attention and trigonometric series that emerges when Q/K vectors are concentrated. I will walk through this derivation carefully, because understanding it is essential to understanding why TriAttention works.

RoPE in complex form. RoPE divides a $d$ -dimensional vector into $d/2$ two-dimensional subspaces. For each subspace (called a frequency band), RoPE applies a 2D rotation by angle $\omega_f p$ at position $p$ , where $\omega_f = \theta^{-2f/d}$ with $\theta = 10000$ . This rotation can be expressed elegantly in complex numbers. If we represent the two components $(x_{2f}, x_{2f+1})$ as a complex number $x_f = x_{2f} + i \cdot x_{2f+1}$ , then rotating by angle $\omega_f p$ is simply multiplication by $e^{i \omega_f p}$ :

$\tilde{q}_f(p_q) = q_f \cdot e^{i \omega_f p_q}$

$\tilde{k}_f(p_k) = k_f \cdot e^{i \omega_f p_k}$

where $q_f, k_f \in \mathbb{C}$ are the pre-RoPE (content-only, no position) complex representations of the Query and Key in frequency band $f$ , and $p_q, p_k$ are their token positions. The tilde notation $\tilde{q}_f, \tilde{k}_f$ denotes the post-RoPE vectors after rotation.

The attention dot product in complex form. The attention logit between a query at position $p_q$ and key at position $p_k$ is computed as the real part of the complex inner product, summed over all frequency bands. For a single band $f$ , the contribution is:

$\text{Re}(\tilde{q}_f(p_q) \cdot \overline{\tilde{k}_f(p_k)}) = \text{Re}\left(q_f \cdot e^{i \omega_f p_q} \cdot \overline{k_f} \cdot e^{-i \omega_f p_k}\right)$

The exponentials combine through $p_q - p_k$ , and the pre-RoPE vectors combine as $q_f \overline{k_f} = \|q_f\| \|k_f\| e^{i(\arg(q_f) - \arg(k_f))}$ . Pulling everything together:

$\text{Re}(\tilde{q}_f(p_q) \cdot \overline{\tilde{k}_f(p_k)}) = \|q_f\| \|k_f\| \cos(\omega_f (p_q - p_k) + (\arg(q_f) - \arg(k_f)))$

Let $\Delta = p_q - p_k$ be the Q-K distance (positive when the query comes after the key, which is always true during autoregressive generation). Let $\phi_f = \arg(q_f) - \arg(k_f)$ be the phase difference between the pre-RoPE Q and K vectors in band $f$ . Summing over all $f$ gives the full pre-softmax attention logit at the head level:

$\text{logit}(q, k) = \sum_{f=0}^{d/2 - 1} \|q_f\| \|k_f\| \cos(\omega_f \Delta + \phi_f)$

where $\Delta = p_q - p_k$ is the signed Q-K distance, $\omega_f = \theta^{-2f/d}$ is the RoPE angular frequency for band $f$ , $\|q_f\|$ and $\|k_f\|$ are the magnitudes (norms) of the pre-RoPE Q and K vectors in band $f$ , and $\phi_f = \arg(q_f) - \arg(k_f)$ is their phase difference.

What this equation computes: for a specific query-key pair at specific positions, it computes the attention logit (the raw score before softmax) by summing over all frequency bands, where each band contributes a cosine term whose amplitude is the product of Q and K magnitudes in that band, whose frequency is the RoPE frequency, whose spatial argument is the Q-K distance shifted by the phase difference. The result is a scalar — higher values mean the query attends more strongly to that key.

Why this form matters: the logit depends on $q$ and $k$ in two ways: through their magnitudes $\|q_f\| \|k_f\|$ and through their phase alignment $\cos(\omega_f \Delta + \phi_f)$ . The phase alignment term is where position enters — as $\Delta$ changes, the cosine oscillates. Different heads can learn different $\phi_f$ values (via different Q/K directions) to make the cosine peak at different $\Delta$ , effectively implementing different distance preferences.

The critical simplification under concentration. The key observation is that when Q and K vectors are highly concentrated — meaning nearly all queries and keys in a given head point in approximately the same direction — we can replace individual $q_f$ and $k_f$ with their centers:

$q_f \approx \mathbb{E}[q_f], \quad k_f \approx \mathbb{E}[k_f]$

where the expectations are taken over tokens in a calibration dataset. Under this approximation, $\|q_f\| \|k_f\|$ and $\phi_f$ become constants (they no longer depend on which specific tokens $q$ and $k$ are). The logit then becomes a function of $\Delta$ alone:

$\text{logit}(\Delta) \approx \sum_{f=0}^{d/2 - 1} \|\mathbb{E}[q_f]\| \|\mathbb{E}[k_f]\| \cos(\omega_f \Delta + \bar{\phi}_f)$

where $\bar{\phi}_f = \arg(\mathbb{E}[q_f]) - \arg(\mathbb{E}[k_f])$ is the phase difference between the Q and K centers. Expanding the cosine using the angle addition formula:

$\text{logit}(\Delta) \approx \sum_{f} \underbrace{\|\mathbb{E}[q_f]\| \|\mathbb{E}[k_f]\| \cos(\bar{\phi}_f)}_{a_f} \cos(\omega_f \Delta) + \underbrace{(-\|\mathbb{E}[q_f]\| \|\mathbb{E}[k_f]\| \sin(\bar{\phi}_f))}_{b_f} \sin(\omega_f \Delta)$

The coefficients $a_f$ and $b_f$ are constants determined entirely by the Q and K centers. This is a trigonometric series in the distance $\Delta$ — essentially, a function that can produce an arbitrary periodic pattern over distance by combining sinusoids at the fixed RoPE frequencies.

Why a trigonometric series, not a Fourier series: the frequencies $\omega_f$ follow a geometric progression ( $\omega_0 = 1$ , $\omega_1 = \theta^{-2/d}$ , $\omega_2 = \theta^{-4/d}$ , ...) rather than the harmonic progression ( $\omega_0, 2\omega_0, 3\omega_0, ...$ ) of a classical Fourier series. The term "trigonometric series" is the general mathematical category that includes both — any sum of sines and cosines with arbitrary frequencies.

What this means operationally: given only the Q and K centers (computed once from calibration data), you can predict for a head whether it will attend more to nearby keys, distant keys, or keys at some intermediate distance — without ever observing a single actual attention score. The curve $\text{logit}(\Delta)$ tells you the expected logit as a function of distance. If the curve peaks at small $\Delta$ , the head is local; if it peaks at large $\Delta$ , the head implements attention sinks; if it has multiple peaks, the head may implement more complex distance-dependent patterns. This predictability is what TriAttention exploits.

The Q/K Concentration Phenomenon: Empirical Characterization

The mathematical simplification above is only useful if Q/K vectors are actually concentrated. The paper devotes substantial analysis to establishing this as a robust empirical fact.

Measurement via Mean Resultant Length. Concentration is quantified using a standard directional statistics metric called the Mean Resultant Length. For a set of vectors, $R$ is defined per frequency band as:

$R_f = \frac{\|\mathbb{E}[q_f]\|}{\mathbb{E}[\|q_f\|\]}$

where $\mathbb{E}[q_f]$ is the mean vector (center) of the Q vectors in frequency band $f$ and $\mathbb{E}[\|q_f\|]$ is the mean magnitude.

What it computes: the ratio of the length of the mean vector to the average length of the individual vectors. If vectors all point in the same direction, their mean points strongly in that direction and $\|\mathbb{E}[q_f]\|$ is large, so $R_f \to 1$ . If vectors point in random directions, the mean averages toward zero and $R_f \to 0$ . The denominator $\mathbb{E}[\|q_f\|]$ normalizes by the typical vector length, making $R_f$ a pure measure of directional concentration independent of magnitude scale.

Why this metric: Mean Resultant Length directly measures how well the "replace with center" approximation works. When $R_f$ is high, nearly all tokens' Q vectors in band $f$ are well-approximated by the center $\mathbb{E}[q_f]$ . When $R_f$ is low, the approximation breaks down and individual deviations matter. This makes $R_f$ the natural quantity to use for adaptive weighting between the trigonometric series (accurate when $R_f$ is high) and norm-based scoring (needed when $R_f$ is low).

Prevalence across heads. Figure 2(C) in the paper shows that across all heads in Qwen3-8B (36 layers × 32 heads = 1152 heads), the vast majority have $R$ values approaching 1.0. The paper states that "the vast majority exhibit R values approaching 1.0" and quantifies in Appendix I that 84.7% of Qwen3-8B heads have $R > 0.95$ , with 90.8% having $R > 0.90$ . For another architecture (GLM-4.7-Flash using MLA), the numbers are even higher: 96.6% with $R > 0.95$ and 99.8% with $R > 0.90$ .

Cross-domain stability. The paper tests whether concentration depends on the input domain by measuring MRL on Math, Coding, and Chat data for Qwen3-8B. The values are "nearly identical (0.977–0.980)," with ~90% of heads exhibiting $R > 0.95$ regardless of domain. This is a critical validation: if concentration were domain-dependent, calibration data quality would matter, and TriAttention might fail when deployed on data different from the calibration distribution. The cross-domain stability suggests concentration is a model-intrinsic property — a consequence of how the model was pretrained, not what it is processing.

Visual evidence. Figure 2(A) overlays pre-RoPE Q/K vectors from three different input sequences in the dominant frequency band. Rather than scattered points, the vectors form tight clusters. Figure 2(B) shows the same vectors after RoPE rotation — they spread out into arc patterns as position varies, visually demonstrating why post-RoPE analysis is difficult: the same content vector disperses across an arc depending on position.

Dominant frequency band selection. Not all frequency bands contribute equally to attention. The paper defines the contribution of band $f$ as:

$C_f = \mathbb{E}[\|q_f\|] \cdot \mathbb{E}[\|k_f\|\]$

where expectations are over the calibration dataset. Bands are ranked by $C_f$ , and typically the top-2 bands are selected for visualization. These dominant bands account for the majority of the attention logit magnitude, so concentration in these bands is what primarily determines attention patterns.

TriAttention Key Importance Scoring: The Trigonometric Series Term

Having established that Q/K concentration causes predictable distance preferences, the paper designs a scoring function that quantifies how important each cached key will be for future queries. The scoring has two additive terms, of which the first and most novel is the trigonometric series term.

Scoring philosophy. The goal is to estimate, for each key $k$ at position $p_k$ currently in the cache, how much attention it will receive from a future query at some position $p_q = p_k + \Delta$ where $\Delta > 0$ . The key insight: instead of trying to predict the specific content of future queries (which is impossible), use the Q center $\mathbb{E}[q_f]$ as a proxy — justified because queries are highly concentrated, so any future query will be close to the center.

$S_{trig}(k, \Delta) = \sum_{f} \|\mathbb{E}[q_f]\| \cdot \|k_f\| \cdot \cos(\omega_f \Delta + \phi_f)$

where $\|\mathbb{E}[q_f]\|$ is the magnitude of the Q center in frequency band $f$ (a pre-computed constant from calibration), $\|k_f\|$ is the magnitude of the actual cached key $k$ in band $f$ (known at inference time), $\omega_f$ is the RoPE frequency (fixed by model architecture), $\Delta = p_q - p_k$ is the assumed future Q-K distance, and $\phi_f = \arg(\mathbb{E}[q_f]) - \arg(k_f)$ is the phase difference between the Q center and the actual key.

What it computes: for a given key and an assumed future distance $\Delta$ , it computes the expected attention logit that a typical future query (represented by the Q center) would assign to that key. The summation is over all frequency bands, with each band contributing the product of Q center magnitude, key magnitude, and a cosine term encoding how well the key's phase aligns with the Q center at distance $\Delta$ . Higher scores mean the key is at a distance from the future query that the head's distance preference curve treats as important.

Why the Q center, not actual queries: using actual future queries would require waiting until those queries are generated, which defeats the purpose of proactive key eviction. Using the Q center makes the score computable at eviction time from quantities that are either pre-computed (the center) or known (the cached key). This is the fundamental difference from post-RoPE methods, which must observe actual attention scores from recent queries — TriAttention predicts importance from stable statistics.

Why the key magnitude $\|k_f\|$ is kept, not replaced by $\|\mathbb{E}[k_f]\|$ : if both Q and K were replaced by their centers, the score would be identical for every key at the same distance $\Delta$ , which would provide no discrimination among keys. By keeping the actual key magnitude, the score differentiates keys by their content-dependent salience — keys with larger magnitudes in frequency bands where the Q center is large will score higher. This is a first-order correction to the pure-center approximation, capturing the fact that even within a concentrated distribution, some keys are "louder" than others.

Why this form (trigonometric series with actual key norms): the distance preference is encoded in the cosine term — as $\Delta$ varies, $\cos(\omega_f \Delta + \phi_f)$ oscillates, and the sum over $f$ creates a curve that can peak at specific distances. The key magnitude scaling $\|k_f\|$ modulates this curve key-by-key, so two keys at similar distances can receive different scores if their content profiles differ. This combines the two things that matter for attention: being at the right distance (positional preference) and having the right content (magnitude alignment).

TriAttention Key Importance Scoring: The Norm-Based Term

The trigonometric series term assumes Q and K are exactly at their centers. In practice, there is variation around the centers, and some of this variation affects attention independently of distance. The norm-based term accounts for this residual variation.

Basic form. The simplest norm-based score would be the product of expected Q and actual K magnitudes, summed over bands:

$S_{norm}^{(0)}(k) = \sum_f \mathbb{E}[\|q_f\|\] \cdot \|k_f\|$

where $\mathbb{E}[\|q_f\|\]$ is the expected magnitude of queries in band $f$ and $\|k_f\|$ is the actual magnitude of the cached key in band $f$ .

What it computes: the expected contribution of the magnitude component of the attention dot product, ignoring the cosine (directional) component entirely. It assumes that a key with large norm will tend to receive high attention regardless of position, because the magnitude product $\|q\| \|k\|$ is a multiplicative factor on the cosine term. This is a reasonable assumption because the cosine term is bounded in $[-1, 1]$ — it can only modulate the magnitude product, not outweigh it by orders of magnitude.

Why this is necessary: consider two keys at the same preferred distance. The trigonometric series term will give them similar scores (same cosine value, differing only by $\|k_f\|$ ). But if one key has extremely low norm across all bands — say, it's a punctuation token whose representation is near-zero — it will contribute little to attention regardless of its distance. The norm-based term explicitly penalizes such keys.

Why key norms rather than attention scores: prior norm-based methods like VATP use value norms, not key norms. The paper's choice to use key norms is directly motivated by the RoPE attention formula (Equation 2): the magnitude product $\|q_f\| \|k_f\|$ appears as a multiplicative factor. Key norm is therefore the natural complement to the cosine-based distance preference — together they cover both multiplicative components of the dot product.

Adaptive Weighting via Q/K Concentration

The two scoring components are not equally reliable across all heads. The paper introduces an adaptive weighting scheme that automatically determines how much each component contributes, per frequency band, based on how concentrated the Q/K vectors are.

The weighting factor. For each frequency band $f$ , the Mean Resultant Length of Q vectors is:

$R_f = \frac{\|\mathbb{E}[q_f]\|}{\mathbb{E}[\|q_f\|\]}$

When $R_f$ is high (near 1), Q vectors are tightly clustered around their center, so approximating all queries by the center is accurate — the trigonometric series term is reliable. When $R_f$ is low (near 0), there is substantial variation around the center, so the trigonometric series approximation is noisy — the norm term becomes more important to capture the residual variation.

The paper implements this by scaling the norm-based term by $(1 - R_f)$ :

$S_{norm}(k) = \sum_f (1 - R_f) \cdot \mathbb{E}[\|q_f\|\] \cdot \|k_f\|$

where $R_f$ is the Q Mean Resultant Length in band $f$ , $\mathbb{E}[\|q_f\|\]$ is the expected Q magnitude, and $\|k_f\|$ is the actual key magnitude.

What it computes: a norm-based score where each band's contribution is suppressed if Q vectors in that band are highly concentrated. When $R_f \to 1$ , $(1 - R_f) \to 0$ and the contribution vanishes — the trigonometric series already captures everything. When $R_f \to 0$ , $(1 - R_f) \to 1$ and the full norm contribution is preserved — the trigonometric series is unreliable, so fall back to pure norm-based scoring.

Equivalent reformulation. By expanding the definition of $R_f$ , the weighted norm term can be rewritten:

$S_{norm}(k) = \sum_f \left(\mathbb{E}[\|q_f\|\] - \|\mathbb{E}[q_f]\|\right) \cdot \|k_f\|$

This shows that $S_{norm}$ is proportional to the dispersion of Q vectors (expected magnitude minus center magnitude). When dispersion is zero (perfect concentration), $S_{norm} = 0$ and $S_{trig}$ fully determines the score. When dispersion is large, $S_{norm}$ contributes significantly.

Why this adaptation is not obvious: one might be tempted to treat concentration as a binary property (concentrated or not) or to use a hard threshold. The paper's continuous weighting via $(1 - R_f)$ is more nuanced — it smoothly interpolates between the two regimes, with the interpolation factor being the natural measure of how good the approximation is. This means heads that are "moderately concentrated" get a mix of both signals, which is presumably optimal since the trigonometric series is partially informative in this regime.

The combined score. The final score for a key at assumed future distance $\Delta$ is simply the sum:

$S(k, \Delta) = S_{trig}(k, \Delta) + S_{norm}(k)$

where $S_{trig}(k, \Delta)$ captures positional preference (is this key at the right distance?) and $S_{norm}(k)$ captures content salience (is this key "loud" enough to matter?), with the balance automatically controlled by the Q concentration in each frequency band.

Handling Multiple Future Positions

A key may be queried from any future position, not just one specific distance. A key that is at a "bad" distance for one future query may be at a "good" distance for a later query. To handle this, the paper evaluates the scoring function at multiple future offsets and averages.

Offset set. The set of future offsets is:

$D = \{1, 2, 4, 8, 16, \ldots, 2^{16}\}$

which spans from immediate adjacency ( $\Delta = 1$ ) to very long-range ( $\Delta = 65536$ ) with logarithmic spacing. For the main experiments, $|D| = 17$ offsets are used (Section 5.2, Table E in Appendix).

Why logarithmic spacing: the trigonometric series contains sinusoidal components at geometrically-spaced RoPE frequencies. To adequately sample the oscillatory behavior, more samples are needed at small $\Delta$ (where the cosine terms change rapidly) than at large $\Delta$ (where the terms change slowly or have decayed). Linear spacing would severely undersample the near-distance regime while oversampling the far-distance regime where little changes. The ablation in Appendix G (Table E) confirms this dramatically: geometric spacing achieves 45.8% accuracy on AIME24 vs. 28.7% for linear spacing with the same number of offsets — a 17.1 percentage point gap.

Why a range of $2^{16}$ : the maximum context length for modern LLMs is typically 32K to 128K tokens. An offset of $2^{16} = 65536$ covers distances out to 64K, ensuring that the scoring considers even very long-range attention patterns. The ablation in Appendix G (Table E) shows that increasing the maximum distance from 128 to 4096 improves AIME24 accuracy from 41.7% to 48.8% (+7.1%), confirming that long-range offsets matter. Further increases to 65536 yield a small decrease (to 45.8%), suggesting that beyond a certain range, the benefit from additional offsets is offset by dilution of the average with less informative far-distance scores.

Averaging. For each key $k$ at current position $p_k$ , the current query position is approximately $p_q \approx p_k + \Delta$ where $\Delta$ is the number of tokens generated so far (the key's "age"). Future queries will be at positions $p_k + \Delta + \delta$ for $\delta \in D$ . The averaged score is:

$\tilde{S}(k) = \frac{1}{|D|} \sum_{\delta \in D} S(k, \Delta + \delta)$

where $\Delta$ is the current age of key $k$ , $\delta$ iterates over future offsets, and $|D|$ is the number of offsets (typically 17).

What it computes: for each cached key, it computes the expected attention score averaged over a wide range of future query positions, from immediately upcoming to very distant. This is analogous to taking a moving average of the distance-preference curve over $\Delta$ — keys whose distance puts them near a peak of the curve for some reasonable fraction of future positions will score highly; keys in "troughs" will score poorly.

Why averaging, not max: one alternative would be to take the maximum score over offsets (a key is important if it will be important at any future position). The paper chooses averaging instead, which is more conservative — it requires keys to be consistently useful rather than briefly peaking. This makes sense because if a key is only important at one specific future distance, it may already be evicted before that distance arrives (since pruning happens periodically, not at every step).

Why treat all offsets equally: the paper does not weight nearer offsets more heavily. This implicitly assumes that a query at distance $\delta$ is equally likely regardless of $\delta$ , which is approximately true for long-range attention where the model can attend anywhere in the past. For local attention patterns (which dominate many heads), nearer offsets would be more important, and weighting accordingly might improve performance. The paper does not explore offset weighting, possibly because the dominant-head analysis suggests that different heads have different preferred distances, making any uniform weighting scheme suboptimal.

KV Cache Pruning Procedure

The scoring function computes importance for each key, but the practical implementation must address two engineering considerations: minimizing computational overhead from frequent scoring, and handling Grouped-Query Attention (GQA) where multiple query heads share each KV head.

Window-based pruning. Computing $\tilde{S}(k)$ for every key at every decoding step would be expensive — the summation over frequency bands and the averaging over offsets involve significant computation per key. The paper follows R-KV's approach: pruning is triggered only once every $\beta = 128$ generated tokens. When the 128th token of each window is generated, if the current cache size exceeds the budget $B$ , all cached keys are scored, the top- $B$ are retained, and the rest are evicted. Within a window, the cache grows unboundedly from its size at the start of the window.

Rationale for $\beta = 128$ : this balances two competing factors. Smaller $\beta$ means more frequent pruning, which keeps memory usage tighter but increases scoring overhead. Larger $\beta$ reduces overhead but allows the cache to temporarily grow, potentially exceeding memory capacity. The choice of 128 is inherited from R-KV and is not ablated in the paper; it represents a pragmatic balance that keeps scoring overhead negligible (the paper does not report a separate overhead analysis for the scoring computation itself, but the throughput numbers in Table 4 show substantial speedups, suggesting the overhead is small amortized over 128 tokens).

Handling Grouped-Query Attention (GQA). In GQA, $G$ query heads share one KV head. Each query head $g \in \{0, ..., G-1\}$ has its own Q center statistics $\mathbb{E}[q_f]^{(g)}$ and $\mathbb{E}[\|q_f\|]^{(g)}$ , which produce $G$ different scores for each cached key. These scores operate at different numerical scales across heads (one head might produce scores in the range $[-10, 10]$ while another produces $[-1, 1]$ ), making direct comparison or simple averaging unreliable.

The paper solves this with normalize-then-aggregate:

Z-score normalization per head. For each query head $g$ , compute the sample mean $\mu_g$ and standard deviation $\sigma_g$ of its scores over all currently cached keys. Then normalize:

$\hat{S}^{(g)}(k) = \frac{\tilde{S}^{(g)}(k) - \mu_g}{\sigma_g}$

where $\tilde{S}^{(g)}(k)$ is the raw score from query head $g$ for key $k$ , $\mu_g$ is the mean score for head $g$ across all keys, $\sigma_g$ is the standard deviation for head $g$ across all keys, and $\hat{S}^{(g)}(k)$ is the normalized score.

What this computes: converts each head's scores to a common scale where zero means average, positive means above average, and the unit is standard deviations. This makes scores comparable across heads regardless of their original scaling.

Maximum aggregation. The final score for a key is the maximum normalized score across the $G$ query heads sharing this KV head:

$S_{final}(k) = \max_{g \in \{0, ..., G-1\}} \hat{S}^{(g)}(k)$

Why maximum, not mean: a key is retained if any query head deems it important. This is a conservative (retention-favoring) strategy — if one head needs the key for retrieval and another doesn't, eviction based on the mean score might discard it, breaking the retrieval head. The maximum ensures that the key survives if at least one head strongly needs it.

Why not a learned aggregation: one could train a small network to combine the $G$ scores, but this would require training data (attention patterns from full-cache runs) and would be model-specific. The normalize-max approach is simple, requires no training, and has a clear operational interpretation: retain keys that any head considers important.

Retention and eviction. After computing $S_{final}(k)$ for all cached keys, the keys are sorted by this score, and only the top- $B$ are retained. When $B$ is smaller than the current cache size $C$ , the $C - B$ lowest-scoring keys are permanently evicted — they are removed from the KV cache and will never be available to future attention computations. This is an irreversible decision, which is why the scoring function must be as accurate as possible: once a key is evicted, if a future query needs it, there is no recovery mechanism.

Default budget. The paper uses a default KV budget of $B = 2048$ tokens for most experiments on AIME benchmarks with 32K-token generation. For DS-Llama and MATH 500, a smaller budget of $B = 512$ is used "due to shorter response lengths, ensuring that KV cache compression is actually exercised during generation" — if the generated sequence is shorter than $B$ , no pruning ever occurs and the method reduces to Full Attention.

Offline Calibration Procedure

TriAttention requires per-head statistics that are computed once offline before deployment. The calibration procedure is straightforward but has important design choices.

What is computed. For each attention head in the model, the calibration produces:

$\mathbb{E}[q_f]$ : the mean (center) of pre-RoPE Q vectors in frequency band $f$ , computed as the sample mean over all calibration tokens. This is a complex number (or 2D vector) per band per head.
$\mathbb{E}[\|q_f\|\]$ : the expected magnitude of pre-RoPE Q vectors in band $f$ , computed as the sample mean of the norms.
$R_f = \|\mathbb{E}[q_f]\| / \mathbb{E}[\|q_f\|\]$ : the Mean Resultant Length, derived from the above two quantities.

Note that K statistics ( $\mathbb{E}[k_f]$ , etc.) are not stored for inference — they are only used during the analysis phase (for computing reconstruction correlation, visualizations, and the theoretical derivation). At inference time, the scoring function uses the Q centers (pre-computed) and the actual key vectors $k_f$ (from the cache), so only Q statistics need to be stored.

Why Q centers, not K centers: the scoring function $S_{trig}$ uses $\mathbb{E}[q_f]$ as a proxy for future queries but uses the actual $k_f$ for the cached keys. The rationale is asymmetry: future queries are unknown and must be approximated (by their center), while cached keys are known exactly and their individual characteristics (especially norm) provide useful discrimination. If the K center were used instead, all keys at the same distance would receive identical trigonometric series scores, making the score useless for distinguishing among them.

Calibration data. The paper uses ShareGPT chat data for the main experiments (calibration data quality ablation in Appendix H, Table F). However, the calibration sensitivity experiments show that performance is "stable across calibration sizes from 50k to 960k tokens" (AIME24 accuracy ranges 45.4–45.8%), and that using Google homepage HTML (low quality calibration data) achieves 46.2%, comparable to ShareGPT chat data (46.7%).

Why calibration data barely matters: this is a striking finding that the paper interprets as evidence that Q/K concentration is a model-intrinsic property — it emerges from the model's pretrained weights, not from the specific distribution of calibration tokens. The Q centers capture a structural property of how the model's attention heads are configured, which is essentially constant regardless of what text is fed through the model. This is both a robustness guarantee (TriAttention won't break on out-of-distribution inputs) and a practical convenience (no need to carefully curate calibration data).

Storage overhead. The stored statistics are: per head, per frequency band $f$ , two scalars (real and imaginary parts of $\mathbb{E}[q_f]$ or equivalently the 2D vector) plus one scalar for $\mathbb{E}[\|q_f\|\]$ , plus the derived $R_f$ . With $d/2$ frequency bands (e.g., $d = 128$ for typical heads means 64 bands), this is a few hundred floats per head, negligible compared to the model parameters. The total storage is $O(\text{num\_heads} \times d)$ , which for Qwen3-8B (1152 heads × ~128 dimensions) is on the order of hundreds of kilobytes — essentially zero overhead.

Summary of Design Choices and Their Justifications

Pre-RoPE over post-RoPE: avoids the observation-window instability caused by RoPE rotation dispersing queries into position-dependent orientations. Pre-RoPE vectors are purely content-dependent and stable.
Trigonometric series from Q/K centers over learned importance predictors: the series is derived from the RoPE formula itself — it is not a learned model but a mathematical consequence of concentration. This means no training is needed for the importance estimator, and the predictions have a clear mechanistic interpretation.
Combined trig + norm scoring over either alone: the trigonometric series captures positional preference (which distances matter) but assumes perfect concentration; the norm term captures token salience (which keys are "loud") and is weighted by concentration to automatically handle heads with varying degrees of concentration. Neither alone is sufficient — the ablation in Table 3A shows removing $S_{trig}$ drops AIME24 accuracy from 42.1% to 18.8%; the paper also reports that removing $S_{norm}$ drops accuracy by 5.4 percentage points.
Adaptive weighting via $(1 - R_f)$ over fixed weighting or hard threshold: automatically balances the two components per frequency band based on the natural measure of how valid the concentration approximation is. No hyperparameter tuning needed.
Averaging over logarithmically-spaced future offsets over using a single offset or linearly-spaced offsets: ensures adequate coverage of both near and far distances; logarithmic spacing is critical because sinusoidal components vary rapidly at small $\Delta$ (geometric spacing achieves +17.1% over linear spacing, Appendix G).
Window-based pruning ( $\beta = 128$ ) over per-step pruning: amortizes scoring cost over 128 tokens, keeping overhead negligible while still bounding memory growth.
Z-score + max aggregation for GQA over mean or learned aggregation: simple, interpretable, and conservative — retains keys that any query head considers important.

4. Key Insights and Innovations

Innovation 1: Pre-RoPE Space as the Foundation for Stable KV Importance Estimation

The paper's most fundamental move is not a particular scoring formula but a diagnostic reframing: the instability that plagues KV cache compression is not a failure of specific algorithms but a consequence of operating in the wrong vector space. This shifts the problem from "how do we build a better importance estimator from attention scores?" to "which representation space makes importance estimation stable in the first place?"

What the field did before. Every prior KV cache compression method — heuristic (StreamingLLM), attention-based (H2O, SnapKV, R-KV, LazyEviction), and norm-based (VATP) — computed importance from post-RoPE representations (§2.2). That is, they worked with vectors after positional encoding had been applied. The justification was practical: attention scores, the most natural importance signal, are computed in post-RoPE space. If a token receives high attention, it must be important; if it receives low attention, it can be evicted. This logic is so natural that prior work focused on improving the aggregation of these attention scores (H2O accumulates across steps; SnapKV uses a local window and aggregates; R-KV uses most-recent queries with redundancy detection) without questioning whether the underlying signal was reliable.

The paper's diagnostic. The instability is not an engineering limitation — it is a mathematical inevitability of working in post-RoPE space. RoPE rotates Q and K vectors by position-dependent angles $\omega_f p$ , which means the same content vector points in different directions depending on where it appears in the sequence. A query at position $p$ and a query at position $p + 100$ have the same pre-RoPE content representation but different orientations. When attention-based methods use recent queries to estimate future importance, they are using queries whose orientations are similar to nearby future queries but increasingly dissimilar to distant future queries. This creates the "narrow observation window" problem — only ~25 queries are representative, as Zhang et al. (2025) empirically confirmed — and it is not fixable by using more queries, better aggregation, or more sophisticated decay schemes. It is a geometric constraint.

Why pre-RoPE space is different. Before RoPE rotation, Q and K vectors are purely content-dependent. Position has not been injected. If these vectors exhibit stable structure — if they cluster around fixed centers regardless of where the token appears — then that structure is position-independent and can be exploited for importance estimation without the observation-window limitation. The paper shows precisely this: Q/K vectors in pre-RoPE space are highly concentrated around fixed non-zero centers (Figure 2C, ~85-97% of heads with Mean Resultant Length $R > 0.95$ ), and this concentration is stable across positions, input domains, and model architectures (Appendices H, I). This is not a claim that pre-RoPE vectors encode all attention-relevant information — they don't encode position, which matters — but rather that the content-dependent component of attention (which determines which tokens are salient) lives in a stable subspace that can be exploited without the instability of positional rotation.

Significance. This is a fundamental shift in how to think about KV importance, not an incremental improvement. It opens an entire category of methods that derive importance from pre-RoPE structure rather than from observed attention scores. The post-RoPE observation window was a conceptual local maximum — the field was optimizing within its constraints rather than questioning whether a different representation space would be more stable. The paper demonstrates that abandoning post-RoPE space entirely yields 2× accuracy at the same memory budget (32.9% vs. 17.5% on AIME25, Table 1) — a gap large enough to suggest the post-RoPE approach has a hard ceiling that TriAttention's pre-RoPE approach transcends, not just incrementally improves upon.

Innovation 2: Q/K Concentration as a Causal Mechanism for Predictable Attention, Not Just a Correlate

Prior work had observed that attention heads exhibit distance-dependent patterns — attention sinks attend to the first few tokens regardless of content, local heads attend to nearby tokens, etc. But these observations were descriptive: they characterized what attention does without explaining why those patterns emerge from the model's learned parameters. The paper's second innovation is to identify Q/K concentration as the causal mechanism that produces these patterns, and to derive the quantitative relationship — the trigonometric series — that connects the learned Q/K centers to the resulting attention-vs-distance curve.

What was missing before. The transformer literature has extensively documented attention patterns: StreamingLLM exploited attention sinks (Xiao et al., 2024), DuoAttention classified heads as retrieval or streaming (Xiao et al., 2025), and various works observed that different heads specialize in different distance ranges. But these were treated as empirical regularities to be measured and exploited, not as consequences of a deeper property of the learned representations. The connection between the learned parameters (the Q/K weight matrices) and the observed behavior (the distance preference) was a black box — you could observe that head 5 in layer 12 attends locally, but you couldn't look at its weight matrix and predict that it would.

The paper's causal claim. The paper traces a clear chain: RoPE attention depends on $\|q_f\| \|k_f\| \cos(\omega_f \Delta + \phi_f)$ → when Q/K are concentrated, $\|q_f\| \|k_f\|$ and $\phi_f$ are approximately constant across tokens → the logit reduces to a function of $\Delta$ alone → the specific curve (peaking at small $\Delta$ , large $\Delta$ , or intermediate) is determined by the Q/K centers. This is not a correlation; it is a derivation. The coefficients $a_f$ and $b_f$ in the trigonometric series are computed directly from $\mathbb{E}[q_f]$ and $\mathbb{E}[k_f]$ , and the resulting curve predicts where the attention head will focus.

Evidence for causality. The reconstruction correlation analysis (Figure 3, Appendix B.6) is the key test: if the causal claim is correct, the trigonometric series computed from Q/K centers should predict actual attention patterns without any fitting or observation of attention scores. The paper finds mean Pearson correlations above 0.5 across all tested architectures (Qwen3, Qwen2.5, Llama3), with distributions peaking in the 0.6–0.9 range. This is not a perfect prediction — the residual variation is handled by the norm-based complement — but it is strong enough to demonstrate that Q/K centers are the primary determinant of distance preference. The correlation is predictive, not just descriptive: you can compute the centers from calibration data and forecast which distances each head will prefer, without ever running the model to observe its attention.

Why this matters beyond TriAttention. This insight transforms how we understand what attention heads learn during pretraining. Instead of thinking about heads as implementing "retrieval," "local attention," or "sink" as fixed functional categories, the paper suggests these are emergent consequences of a simpler learning process: the model learns Q/K centers (and variance around them) that, when plugged into the RoPE formula, produce useful distance preferences for the task. This is a more parsimonious and mechanistically grounded account that could inform future work on attention interpretability, architecture design, and training.

Distinction from incremental advance. This is not a small refinement. The field went from "we observe distance-dependent attention patterns and exploit them heuristically" to "we can predict distance-dependent attention patterns from learned parameters using a closed-form mathematical expression." This is a qualitative change in explanatory depth — it's the difference between cataloging animal behaviors and understanding the genetic code that produces them.

Innovation 3: Trigonometric Series as a Scoring Mechanism — Derived, Not Learned

TriAttention's scoring function is mathematically derived from the RoPE formula under the concentration approximation, not learned from data. This is a conceptual departure from the dominant paradigm in KV cache compression, where importance scoring is either heuristic (StreamingLLM's fixed retention pattern) or learned implicitly through attention observation (H2O's accumulated scores, SnapKV's voting, R-KV's query-based scoring). The paper argues that for this particular problem, a derived scoring function outperforms learned/observed ones because it avoids the instability that makes learning difficult.

The alternative the field was pursuing. Attention-based methods all follow the same template: observe which tokens receive high attention during a window, then retain those tokens. This is essentially learning importance from data — the data being the model's own attention scores on the current sequence. The learning is non-parametric (no model is trained) but it is learning nonetheless: the system must infer a general importance rule from a limited sample of observations. The instability problem shows that this sample is too small and biased to produce reliable inference — you're trying to learn a function (importance-vs-token) from ~25 noisy observations in a space where the underlying signal (attention) depends on position in complex ways.

The paper's alternative. Rather than observing attention to learn importance, derive importance from the mathematical structure of attention itself. The derivation is: (1) RoPE attention has the form $\sum_f \|q_f\| \|k_f\| \cos(\omega_f \Delta + \phi_f)$ ; (2) under Q/K concentration, $\|q_f\| \|k_f\|$ and $\phi_f$ are well-approximated by their centers; (3) therefore the expected attention at distance $\Delta$ is a trigonometric series with coefficients determined by the Q/K centers. The scoring function $S_{trig}$ is the realization of this derivation for the specific case where the key is known and the query is approximated by its center.

This is a fundamentally different philosophy: the scoring function is not adapting to the current sequence or learning from observations; it is applying a fixed, pre-computed formula that captures a structural property of the model's attention heads. The "learning" happens once during calibration (computing Q/K centers), and after that, the scoring is deterministic and data-independent.

Why this works better. The paper identifies two reasons. First, the calibration data is large and diverse (50K–960K tokens from ShareGPT), providing stable estimates of the Q/K centers. Compare this to attention-based methods that must infer importance from ~25 recent tokens on a single sequence — the statistical efficiency difference is enormous. Second, the Q/K centers are model-intrinsic — they barely change across calibration domains (Appendix H: coding vs. reasoning vs. HTML calibration yields near-identical performance), so the calibration generalizes robustly. Attention-based methods, by contrast, must adapt to each new sequence's specific content and structure, which makes them vulnerable to distribution shift within a single sequence (e.g., when the reasoning style changes mid-generation).

Significance. This is not just "we found a better way to score keys." It is a demonstration that for mechanistic problems (where the computation is well-understood), deriving the scoring function from first principles can outperform learning it from data, because the derivation captures invariances that learned methods must re-discover from limited samples. This has implications beyond KV compression — it suggests that for other inference-time optimization problems with well-understood computational structure (speculative decoding, early exiting, mixture-of-experts routing), analytical approaches based on model internals may outperform purely empirical ones.

Innovation 4: Concentration-Weighted Hybrid Scoring — The First Principled Integration of Directional and Magnitude Signals

Norm-based importance (VATP, Kobayashi et al., 2020) and attention-based importance (H2O, SnapKV, R-KV) have been treated as competing paradigms — you either score keys by their value norms or by their attention scores. The dominant assumption was that attention scores, when properly aggregated, capture all relevant importance information, and norm-based methods are an alternative for when attention scores are unavailable or unreliable. The paper's fourth innovation is to show that these are complementary signals that capture fundamentally different aspects of attention, and to provide a principled, parameter-free method for integrating them based on the same concentration metric that motivates the trigonometric series.

The decomposition. The attention logit has two multiplicative components per frequency band: a magnitude component $\|q_f\| \|k_f\|$ and a directional component $\cos(\omega_f \Delta + \phi_f)$ . The trigonometric series term $S_{trig}$ captures the directional component under the approximation that Q/K are at their centers — it answers "is this key at the right distance?" The norm-based term $S_{norm}$ captures the magnitude component — it answers "is this key 'loud' enough to matter?" These are not redundant: a key can be at exactly the right distance (high $S_{trig}$ ) but have tiny norm (low $S_{norm}$ ), meaning it contributes little; or a key can have huge norm (high $S_{norm}$ ) but be at a distance the head ignores (low $S_{trig}$ ).

Why prior work didn't integrate them. Attention-based methods implicitly capture both components — an attention score is the product of magnitude and directional alignment — but they confound them and are limited by the observation window. Norm-based methods capture only the magnitude component but avoid the observation window problem by computing directly from cached values. The paper's insight is that in pre-RoPE space, you can separate the two components, compute each from its most reliable source, and recombine. The directional component (distance preference) comes from the Q/K centers via the trigonometric series, which is stable because centers are computed from large calibration data. The magnitude component comes from the actual cached key norms, which are directly observable without any position dependence.

The weighting mechanism. The weighting factor $(1 - R_f)$ on the norm term is the key conceptual move. Rather than treating concentration as a binary property (this head is concentrated vs. not), the paper uses the Mean Resultant Length $R_f$ as a continuous measure of how good the center approximation is. When $R_f \to 1$ , the center approximation is nearly perfect, so $S_{trig}$ is reliable and $S_{norm}$ is redundant (it would just add noise). When $R_f$ is lower, the center approximation degrades, and $S_{norm}$ provides useful complementary information about the variation around the center. The weighting is adaptive per frequency band — a head could be highly concentrated in its dominant bands (so $S_{trig}$ dominates there) but less concentrated in other bands (so $S_{norm}$ contributes there).

This is an elegant solution to an otherwise thorny hyperparameter problem: how much should the trigonometric series score be trusted versus the norm score? The paper shows that removing the concentration-weighting — using the unweighted norm term $S_{norm}^{(0)}$ — degrades AIME25 accuracy from 32.9% to 28.7% (Table 3B), a 4.2 percentage point drop. This confirms that the weighting matters and that the concentration metric correctly identifies when the trigonometric series is reliable.

Significance. This is an architectural insight for KV cache compression: importance has two orthogonal dimensions (positional preference and token salience) that can be estimated from different sources and combined with a principled weighting. Prior work treated these as alternatives; the paper shows they are complements that together outperform either alone. The fact that the weighting is derived from the same concentration metric that motivates the trigonometric series — rather than being a separately tuned hyperparameter — makes this a coherent theoretical framework rather than an ad-hoc combination of tricks.

Innovation 5: The Over-Optimization Problem Is Architectural, Not Algorithmic — and Pre-RoPE Structure Avoids It

The paper's negative result — that post-RoPE observation-based methods have a hard accuracy ceiling that TriAttention transcends — is itself an insight. Prior work on KV cache compression attributed performance gaps to algorithmic details: SnapKV's voting mechanism, R-KV's redundancy detection, LazyEviction's delayed eviction. The implicit assumption was that better algorithms within the post-RoPE paradigm could close the gap to Full Attention. The paper's results challenge this assumption.

The evidence for a fundamental ceiling. On AIME25 with Qwen3-8B at a KV budget of 2048 (Table 1), TriAttention achieves 32.9% accuracy versus R-KV's 17.5% — a gap of 15.4 percentage points. SnapKV achieves 20.0%. Full Attention achieves 40.8%. This means R-KV and SnapKV are less than halfway from 0 to Full Attention accuracy, despite being sophisticated attention-based methods. Across all four models and two AIME benchmarks, TriAttention consistently matches or exceeds Full Attention at the same budget, while post-RoPE methods consistently fall short by large margins.

Why this suggests a ceiling. If the gap were due to algorithmic details (e.g., R-KV's redundancy detection is imperfect), one would expect that improvements to those details would incrementally close the gap. But the gap is so large (2× accuracy difference) that it suggests a qualitative limitation: post-RoPE methods are systematically missing important tokens that TriAttention preserves. The paper's analysis suggests this is because post-RoPE methods operate with a narrow observation window, and within that window, certain important tokens (those needed for long-range retrieval or backtracking) receive low attention and are evicted. No amount of algorithmic refinement can fix this, because the missing signal — the attention that would have identified those tokens as important — is outside the observation window and therefore permanently unavailable.

The memory retention benchmark confirms this interpretation. On the Recursive State Query benchmark (Figure 5D), R-KV exhibits "catastrophic accuracy degradation starting at depth 16, dropping from approximately 61% at depth 14 to 31% at depth 16." This is exactly the failure mode predicted by the narrow-window hypothesis: at shallow depth, intermediate states are recent enough to fall within the observation window; at deeper depth, they age beyond the window and are evicted, causing the model to lose critical backtracking information. TriAttention shows no such degradation — it matches Full Attention through depth 16 and only begins to lag at depth 18+. This demonstrates that the failure is not about the complexity of the reasoning but about the age of the information, which is precisely what the observation-window limitation would predict.

The meta-insight. This paper effectively shows that the post-RoPE paradigm for KV cache compression has a fundamental accuracy ceiling determined by the size of the observation window — which is itself limited by the positional rotation of queries. This ceiling is architectural, not algorithmic, meaning it cannot be overcome by better algorithms within the paradigm. TriAttention bypasses the ceiling by operating in pre-RoPE space, where importance can be estimated from position-independent statistics without any observation window at all. This is not a claim that TriAttention is the final answer; rather, it demonstrates that escaping the post-RoPE paradigm is necessary for substantial improvements in KV cache compression accuracy, and that the pre-RoPE concentration structure provides a viable escape route.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three mathematical reasoning benchmarks: AIME 2024 (30 problems), AIME 2025 (30 problems), and MATH 500 (Hendrycks et al., 2021) (500 problems spanning diverse mathematical tasks). MATH 500 uses the standard test split. Additionally, LongBench (Bai et al., 2024) with 16 subtasks and RULER (Hsieh et al., 2024) retrieval tasks at 4K context evaluate general task generalization (Appendix F). A custom Recursive State Query benchmark (Appendix C) evaluates memory retention under DFS simulation.
Base model(s). Four reasoning models spanning architectures and scales: Qwen3-8B (Qwen Team, 2025), DeepSeek-R1-Distill-Llama-8B (DeepSeek-AI, 2025), DeepSeek-R1-Distill-Qwen-7B (DeepSeek-AI, 2025), and GPT-OSS-20B (OpenAI, 2025). These are selected for strong chain-of-thought reasoning and diverse architectures (GQA for most, MLA validated in Appendix I on GLM-4.7-Flash). Qwen3-8B serves as the primary testbed for most ablations and scaling analyses.
Metrics. Primary metric is accuracy (pass rate): for AIME, each problem is sampled 8 times and average pass rate is reported; for MATH 500, each problem is sampled once. Pass rate uses the standard grading function from Lightman et al. (2022) (Appendix G). For the Recursive State Query benchmark, stack exact match is the metric — the complete path must be correct in order, making it maximally sensitive to KV cache information loss. For LongBench and RULER, standard task-specific metrics are used. Throughput is measured as average tokens generated per second over 16K decoding length at maximum batch size on a single A100 80GB GPU.
Baselines. Full Attention (no pruning, performance upper bound) and three compression methods: SnapKV (Li et al., 2024b) — selects tokens based on historical attention scores from a local observation window; R-KV (Cai et al., 2025) — attention-based importance scoring combined with redundancy detection for reasoning models; StreamingLLM (Xiao et al., 2024) — retains initial sink tokens plus sliding window; PyramidKV (Cai et al., 2024); KnormPress (Devoto et al., 2025); Ada-KV+SnapKV (Feng et al., 2025); and H2O (Zhang et al., 2023) when memory permits (H2O requires O(n²) memory and cannot use FlashAttention). Additional comparisons with LazyEviction (Zhang et al., 2025), TOVA (Oren et al., 2024), and RaaS (Hu et al., 2025) appear in Appendix E.
Generation budget / compute accounting. Compute is measured by KV cache budget — the maximum number of tokens retained in the KV cache. Default budget is 2048 tokens for AIME benchmarks with 32K-token maximum generation length. For DS-Llama and MATH 500, budget is 512 tokens "due to shorter response lengths, ensuring that KV cache compression is actually exercised during generation" (§5.1). For throughput comparisons, the metric is tokens/second on a single A100 80GB at maximum batch size, following R-KV's evaluation protocol (§5.2.4). Pruning is triggered once every β = 128 generated tokens; within each 128-token window, the cache grows unrestricted until the next pruning point. All experiments use bfloat16 precision with FlashAttention-2 (A100) or FlashAttention-3 (H100 for GPT-OSS). Generation uses temperature 0.6 and top-p 0.95.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. AIME benchmarks with 30 problems each are small test sets, and results are reported as single numbers without confidence intervals. The 8-sample averaging per AIME problem provides some reduction in sampling variance but no formal uncertainty quantification. For LongBench and RULER, standard single-pass evaluation is used.

Main Quantitative Results

Reasoning Task Performance Across Models and Benchmarks

AIME24 and AIME25 (Table 1). The headline result across four models is that TriAttention consistently achieves the best performance among compression methods, often approaching or matching Full Attention while post-RoPE baselines fall substantially short. On Qwen3-8B at KV budget 2048:

AIME24: TriAttention achieves 42.1% vs. SnapKV 34.6%, R-KV 25.4%, Full Attention 57.1%. TriAttention is 16.7 percentage points ahead of R-KV and within 15 points of Full Attention.
AIME25: TriAttention achieves 32.9% vs. SnapKV 20.0%, R-KV 17.5%, Full Attention 40.8%. The gap to R-KV is 15.4 points — nearly double R-KV's accuracy.

This pattern holds across all four models. On DeepSeek-R1-Distill-Llama-8B (DS-Llama), TriAttention achieves 33.8% on AIME24 vs. R-KV's 25.8% (a smaller but consistent 8-point gap). On DeepSeek-R1-Distill-Qwen-7B (DS-Qwen), TriAttention achieves 42.5% vs. R-KV's 34.6%. On GPT-OSS-20B (the largest model), TriAttention achieves 59.2% on AIME24 vs. R-KV's 49.6%, and 49.2% on AIME25 vs. R-KV's 39.2%. In every single model × benchmark combination, TriAttention outperforms all baselines. The second-best method varies: SnapKV occasionally edges out R-KV (DS-Llama AIME24: 5.0% vs. 25.8% — anomalously low SnapKV performance suggesting sensitivity to model architecture), but neither consistently approaches TriAttention.

MATH 500 (Table 2). At a tighter KV budget of 512 tokens, TriAttention achieves 56.0% on Qwen3-8B vs. R-KV's 46.4% and SnapKV's 49.2%, with Full Attention at 69.6%. On DS-Llama, TriAttention reaches 80.6% vs. R-KV 76.9%, closely approaching Full Attention's 82.4%. On DS-Qwen: 79.6% vs. R-KV 71.6% (Full: 87.0%). On GPT-OSS: 81.2% vs. R-KV 77.4% (Full: 91.4%). The gains are smaller than on AIME — the gap to R-KV is typically 4–8 points rather than 15 — likely because MATH 500 problems are easier and produce shorter reasoning chains where the observation window limitation is less severe. However, TriAttention still leads across all four models.

Performance Scaling with KV Budget (Figure 5A-C)

Evaluating Qwen3-8B across KV budgets from 512 to 4096 on three benchmarks reveals how both methods scale:

MATH 500 (Figure 5A): TriAttention matches Full Attention at budget 1024 (68.4% vs. 69.6%) and slightly exceeds it at higher budgets. R-KV plateaus around 60–62% even at 4096 budget, never reaching Full Attention accuracy. The gap between TriAttention and R-KV widens from about 10 points at budget 512 to about 12 points at budget 4096.
AIME24 (Figure 5B): TriAttention reaches 45.8% at budget 2048 (Table 3, ablation baseline) and climbs to ~48–50% at 4096. R-KV reaches only ~25% at 2048 and plateaus near 30% at higher budgets. The curves suggest TriAttention continues to improve with budget while R-KV saturates.
AIME25 (Figure 5C): TriAttention achieves 32.9% at 2048 and 43.3% at 4096, surpassing Full Attention's 40.8% at the highest budget. R-KV achieves 17.5% at 2048 and reaches approximately 25% at 4096. TriAttention nearly doubles R-KV at the lowest budget (512) and maintains a 15+ point lead throughout.

The persistent gap across all budgets — even at 4096 where both methods retain a substantial fraction of the full context — strongly supports the paper's claim that post-RoPE methods have a fundamental accuracy ceiling that TriAttention transcends, not just a disadvantage at aggressive compression ratios.

Memory Retention on Recursive State Query (Figure 5D)

On Qwen3-8B with KV budget 2048, evaluating stack exact match accuracy at DFS recursion depths from 6 to 20:

Depth 6–14: TriAttention performs comparably to Full Attention, even slightly outperforming it at depths 8 and 12. Both maintain accuracy above ~60% (exact numbers from Figure 5D: approximately 88% at depth 6, declining to ~61% at depth 14 for TriAttention).
Depth 16: Full Attention and TriAttention remain roughly at parity (~55–60%). R-KV drops to 31%, exactly the "catastrophic accuracy degradation" the paper describes.
Depth 18–20: TriAttention begins to lag behind Full Attention (approximately 42% vs. 55% at depth 18; 25% vs. 45% at depth 20). R-KV collapses further to near 20% at depth 18 and below 15% at depth 20.

The critical transition at depth 16 — where R-KV drops from ~61% to 31% while TriAttention stays near 58% — is strong evidence for the observation-window interpretation. At depth 16, intermediate states from early in the DFS traversal have aged beyond R-KV's effective observation window (~25 tokens), causing their eviction and the subsequent error cascade. TriAttention's pre-RoPE scoring correctly identifies these states as important regardless of their age, preserving them through depth 16. Only at depth 18+ does the budget constraint (2048 tokens potentially insufficient to store all intermediate states for very deep recursion) begin to affect TriAttention, and even then its degradation is gradual rather than catastrophic.

Throughput and Efficiency (Tables 4 and 5, Figure 1)

Comparison with Full Attention (Table 4). At equivalent accuracy to Full Attention, TriAttention shows speedups ranging from 1.9× to 6.3×:

MATH 500: TriAttention with budget 1024 achieves 68.4% accuracy (vs. Full 69.6%) at 1,405 tokens/s (vs. Full 223 tokens/s) — 6.3× speedup.
AIME24: TriAttention with budget 4096 achieves 54.6% (vs. Full 57.1%) at 414 tokens/s (vs. Full 223 tokens/s) — 1.9× speedup.
AIME25: TriAttention with budget 3072 achieves 40.8% (matching Full exactly) at 564 tokens/s (vs. Full 223 tokens/s) — 2.5× speedup.

The speedup varies with benchmark difficulty: easier benchmarks (MATH 500) allow more aggressive compression, yielding larger speedups, while harder benchmarks (AIME24) require larger budgets to match Full Attention accuracy, yielding smaller speedups.

Comparison with R-KV (Table 5). At comparable accuracy, TriAttention needs only half the KV budget: on MATH 500, TriAttention at budget 1024 (68.4%) matches R-KV at budget 2048 (68.2%) while achieving 85% higher throughput (1,405 vs. 760 tokens/s). At the same memory budget of 1024, TriAttention achieves substantially higher accuracy: 68.4% vs. 60.4% on MATH 500 (+8.0 points) and 25.8% vs. 10.4% on AIME24 (+15.4 points). Throughput at equal budget is comparable (1,345 vs. 1,405 tokens/s for R-KV and TriAttention respectively), meaning TriAttention provides strictly better accuracy at no throughput cost.

KV memory reduction (Figure 1). On AIME25 at 40.8% accuracy (matching Full Attention), TriAttention reduces KV cache memory by 10.7× compared to Full Attention — claiming the KV cache stores only about 1/11th as many tokens while maintaining equal reasoning performance.

General Task Generalization: LongBench and RULER (Appendix F, Tables B–D)

LongBench (Table B): On Qwen3-8B at 50% KV budget across 16 subtasks (QA, summarization, few-shot, retrieval, counting, code), TriAttention achieves the highest average of 48.1, winning 11 out of 16 subtasks. Full Attention achieves 47.2 (TriAttention slightly exceeds Full on average, a counterintuitive result the paper does not explain — possibly noise or TriAttention pruning removing distracting tokens). Ada-KV+SnapKV achieves 45.6, SnapKV 45.2, PyramidKV 42.7, StreamingLLM 39.4, KnormPress 35.1. The gap of +2.5 over Ada-KV+SnapKV is modest but consistent. On specific subtasks, TriAttention's largest wins are on TREC (69.0 vs. 46.0 for Ada-KV, a 23-point gap) and Qasp (43.0 vs. 36.8). On some tasks it loses: PaRe (7.0 vs. 8.0 for Ada-KV and PyramidKV) and LCC (65.0 vs. 66.4 for SnapKV), suggesting domain-specific variation.

RULER (Table C): At 50% KV budget and 4K context, TriAttention achieves 66.1 vs. StreamingLLM 61.1, SnapKV 55.6, PyramidKV 40.7 — a +5.0 lead over the next-best compression method. The RULER benchmark specifically tests retrieval capabilities, making it a strong test of whether TriAttention preserves tokens needed for long-range attention.

Comparison with H2O (Table D): H2O cannot use FlashAttention and requires O(n²) memory, so it can only run on 12 of 16 LongBench subtasks within 48GB GPU memory. On those 12, TriAttention achieves 45.4 average vs. H2O's 41.4, winning 10 of 12 subtasks. This comparison is somewhat unfair to H2O (it's severely memory-constrained), but it demonstrates that TriAttention outperforms even methods with access to full attention matrices.

Comparison with LazyEviction and Additional Baselines (Appendix E, Table A)

On AIME24 with DeepSeek-R1-Distill-Qwen-7B at varying KV budgets (10%, 20%, 30% of full context), TriAttention outperforms all methods at every budget level:

At 10% budget: TriAttention 40.0%, LazyEviction 33.3%.
At 20% budget: TriAttention 43.3%, LazyEviction 40.0%.
At 30% budget: TriAttention 46.7% (matching Full Attention), LazyEviction 43.3%, H2O 33.3%, TOVA 36.7%, RaaS 36.7%, R-KV 43.3%.

Notably, TriAttention matches Full Attention at 30% KV budget, while R-KV and LazyEviction — the strongest post-RoPE baselines — fall short by 3.4 points.

Ablation Studies and Robustness Checks

Trigonometric series score removal (Table 3A): Removing $S_{trig}$ and relying solely on $S_{norm}$ causes AIME24 accuracy to drop from 42.1% to 18.8% (−23.3 points) and AIME25 from 32.9% to 21.2% (−11.7 points). This is the single largest ablation gap, confirming that the distance preference signal from the trigonometric series is the dominant component. The fact that $S_{norm}$ alone achieves only 18.8% on AIME24 (vs. 25.4% for R-KV in Table 1) suggests that norm-based scoring without positional information is insufficient for complex reasoning.

Norm-based score removal: The paper reports that "removing $S_{norm}$ and relying solely on $S_{trig}$ drops AIME24 accuracy from 45.8% to 40.4% (−5.4%)" (§5.2.3). This confirms the norm term provides complementary information, though the trigonometric series is the primary driver of performance.

Concentration-based weighting removal (Table 3B): Replacing the weighted $S_{norm}$ (with $(1 - R_f)$ scaling) with the unweighted $S_{norm}^{(0)}$ (full norm contribution in all bands) drops AIME24 from 42.1% to 41.3% (−0.8 points) and AIME25 from 32.9% to 28.7% (−4.2 points). The small gap on AIME24 but larger gap on AIME25 is not discussed by the paper but may reflect different head concentration profiles on harder problems. The 4.2-point drop on AIME25 confirms that adaptive weighting matters, particularly for more challenging reasoning.

Cross-domain calibration (Table 3C): Calibrating on coding data (Jain et al., 2025) instead of reasoning data and testing on AIME yields 44.2% vs. 42.1% on AIME24 (coding calibration slightly better) and 29.2% vs. 32.9% on AIME25 (reasoning calibration better). The differences are within 3.7 points and go in opposite directions on the two benchmarks, suggesting the calibration domain choice has no systematic effect. The paper interprets this as "Q/K statistics are model-intrinsic properties, robust to the choice of calibration data" — a claim supported by the lack of consistent direction in the differences.

Future offset range ablation (Appendix G, Table E): Increasing maximum offset distance from 128 to 4096 improves AIME24 accuracy from 41.7% to 48.8% (+7.1%), demonstrating that long-range offsets are important. Further increases to 8192 (46.2%) and 65536 (45.8%) show diminishing returns, with accuracy declining slightly — possibly because very distant offsets contribute noisy scores that dilute the average.

Future offset spacing strategy (Appendix G, Table E): With 17 offsets covering range [1, 65536], geometric spacing $\{1, 2, 4, 8, \ldots\}$ achieves 45.8% vs. linear spacing's 28.7% — a 17.1 percentage point gap. This is a substantial effect size, confirming that logarithmic sampling of offsets is crucial: the trigonometric series terms oscillate rapidly at small $\Delta$ , requiring dense sampling in the near-distance regime, while far-distance terms vary slowly and need only sparse coverage.

Calibration data quantity sensitivity (Appendix H, Table F): Performance on AIME24 is stable across calibration sizes: 50K tokens → 45.4%, 200K tokens → 45.8%, 960K tokens → 45.8%. The maximum variation is 0.4 percentage points, well within noise given the 30-problem AIME24 test set. This suggests calibration stabilizes quickly and additional data provides no benefit.

Calibration data quality sensitivity (Appendix H, Table F): Using Google homepage HTML (low quality), code data (medium quality), and ShareGPT chat data (high quality) yields AIME24 accuracies of 46.2%, 43.3%, and 46.7% respectively. The low-quality HTML calibration marginally outperforms high-quality chat data (46.2% vs. 46.7%), a counterintuitive result that strongly supports the model-intrinsic property claim. The code calibration performs slightly worse (43.3%, −3.4 points vs. chat), but this is within the range of variation across data sources and may reflect random noise on the small test set.

MLA architecture validation (Appendix I, Table G): On GLM-4.7-Flash using Multi-head Latent Attention (MLA) with 940 heads, reconstruction quality (Pearson r) and Q/K concentration (MRL) are comparable to or stronger than GQA architectures. 96.6% of MLA heads have $R > 0.95$ (vs. 84.7% for Qwen3-8B GQA), and 23.1% have $r > 0.70$ (vs. 13.0% for GQA). This suggests the concentration phenomenon is architecture-general and possibly even more pronounced in MLA, which is significant because MLA is increasingly common in frontier models (DeepSeek-V2/V3, Qwen3). The paper does not report end-to-end TriAttention accuracy on GLM-4.7-Flash, only the reconstruction quality and concentration metrics — no downstream task performance is shown for MLA.

Comparison across additional baselines (Appendix E): The paper includes comparisons with LazyEviction, H2O, TOVA, RaaS, StreamingLLM, PyramidKV, KnormPress, and Ada-KV+SnapKV across multiple benchmarks, consistently showing TriAttention outperforming all baselines. The breadth of baseline coverage is strong — 8 compression methods compared — though all baselines are post-RoPE methods, meaning the comparison is fundamentally "pre-RoPE approach vs. post-RoPE approaches" rather than "TriAttention vs. other pre-RoPE methods" (there are no other pre-RoPE compression methods to compare against).

Critical Assessment

On the Central Claim: Pre-RoPE Scoring Outperforms Post-RoPE Methods

The paper's core empirical claim is that TriAttention achieves substantially higher accuracy than post-RoPE KV cache compression methods — nearly doubling R-KV's accuracy on AIME25 (32.9% vs. 17.5%) and consistently outperforming all baselines across models and benchmarks. The evidence for this claim is strong and consistent, with every single model × benchmark combination in Tables 1–2 showing TriAttention in first place among compression methods. The magnitude of the gaps (15+ points on AIME, 4–8 points on MATH 500) is large enough that even accounting for the small test sets (30 AIME problems), the ranking is unlikely to be reversed by sampling noise alone.

The budget scaling analysis (Figure 5) provides additional support: the gap persists across all budget levels, not just at aggressive compression ratios. R-KV fails to match Full Attention even at budget 4096 on MATH 500, while TriAttention matches it at budget 1024. This suggests the improvement is not about TriAttention being more memory-efficient at a given budget — it actually achieves higher accuracy than R-KV at the same budget — but about its importance estimation being fundamentally more accurate.

However, there are several qualifications to consider:

Small test sets for AIME. AIME24 and AIME25 each contain only 30 problems. An 8-sample average per problem provides some reduction in variance, but the effective sample size is still 30 — a single difficult problem on which TriAttention happens to succeed and R-KV fails can swing the reported accuracy by ~3.3 percentage points. The paper reports no confidence intervals, standard errors, or statistical tests. On AIME25 with Qwen3-8B, the gap is 15.4 points (32.9% vs. 17.5%), which even with n=30 is likely significant, but smaller gaps (e.g., 3.4 points on DS-Qwen AIME24: 42.5% vs. 39.2% for R-KV? — actually 42.5% vs. 34.6% for R-KV, a 7.9 point gap) would benefit from uncertainty quantification that is absent.

Single model family for most ablations. Qwen3-8B is the workhorse for nearly all ablations, scaling analyses, and robustness checks. While results are reported on four models in Tables 1–2, the deeper analyses (budget scaling, memory retention, calibration sensitivity, offset ablation, weighting ablation) are all on Qwen3-8B alone. We cannot assess whether geometric spacing of offsets matters equally for other architectures, or whether cross-domain calibration is robust beyond Qwen3, without those experiments.

Most baselines are tuned for post-RoPE space. R-KV, SnapKV, and LazyEviction all have hyperparameters (observation window size, score decay, redundancy thresholds) tuned for their post-RoPE operation. The paper uses these methods "as-is" without re-tuning for the specific models or benchmarks — this is standard practice, but it's possible that better-tuned baselines would narrow the gap. The R-KV authors designed their method for reasoning models specifically, and the paper's R-KV results match the typical accuracy levels reported in the R-KV paper, suggesting the baselines are fairly represented.

On the Claim: TriAttention Matches Full Attention While Reducing Memory 10.7×

The throughput table (Table 4) shows TriAttention matching Full Attention accuracy at 40.8% on AIME25 while achieving 2.5× throughput and 10.7× memory reduction. This claim is supported by the numbers as reported. However:

Accuracy matching is not exact matching. On AIME25 at budget 3072, TriAttention achieves exactly 40.8% — identical to Full Attention's 40.8%. But at other budget levels, the relationship varies: at budget 4096, TriAttention achieves 43.3%, exceeding Full; at budget 2048, it achieves 32.9%, falling short. The "matched accuracy" point at 3072 is a single budget level selected precisely where the curves cross. The more general statement is that TriAttention's accuracy-vs-budget curve intersects Full Attention's accuracy at some achievable budget, not that it universally matches Full Attention.
10.7× KV memory reduction is computed at the specific budget (3072 on AIME25) where accuracy matches Full. At this budget, Full Attention stores all generated tokens (~32K), while TriAttention stores only 3072 — roughly 10.4× reduction (the paper reports 10.7× — this slight discrepancy may reflect that the exact full-cache size varies per sequence). The 10.7× figure is fair for this specific operating point but should not be misinterpreted as a universal reduction factor.

On the Claim: Q/K Concentration Is Model-Intrinsic and Domain-Independent

The calibration sensitivity experiments (Appendix H) are among the paper's most important robustness checks because the entire method depends on calibration statistics being stable. The results are striking: calibration size from 50K to 960K tokens barely matters (45.4%–45.8%), and low-quality HTML data achieves accuracy comparable to chat data (46.2% vs. 46.7%). The cross-domain test (Table 3C) shows calibration on coding vs. reasoning yields non-systematic differences.

Strengths: These are genuinely interesting negative results — the null finding that calibration data doesn't matter is itself an important insight that supports the model-intrinsic claim.

Weaknesses: All calibration sensitivity tests are on Qwen3-8B only. We don't know if this robustness holds for DS-Llama, DS-Qwen, or GPT-OSS. The MLA validation (Appendix I) tests only reconstruction correlation and concentration, not downstream task performance after calibration with different data. It's possible that MLA models, despite showing even stronger concentration, are more sensitive to calibration data distribution — the paper provides no evidence either way.

Additionally, "model-intrinsic property" is demonstrated across three GQA architectures (Qwen3, Qwen2.5, Llama3) and one MLA architecture (GLM-4.7), all from similar model families (dense transformers with RoPE). Whether this property extends to models with fundamentally different attention mechanisms (linear attention, sliding window attention, hybrid architectures) is untested.

On the Memory Retention Claims

The Recursive State Query benchmark (Figure 5D) is a well-designed stress test that directly probes the paper's core mechanistic claim: that post-RoPE methods lose important tokens due to the observation window limitation. The catastrophic degradation of R-KV at depth 16 while TriAttention maintains comparable accuracy to Full Attention through depth 16 is compelling evidence.

However, the benchmark is custom and unvalidated. The paper provides no evidence that DFS simulation is representative of real reasoning chain memory demands. Real chain-of-thought reasoning does not follow deterministic DFS patterns — tokens may be revisited in more complex ways, and forgetting may manifest differently. The benchmark convincingly demonstrates that TriAttention preserves long-range dependencies better than R-KV in a controlled setting, but the mapping from this result to "TriAttention enables better reasoning" is inferential, not directly demonstrated. The AIME and MATH 500 results provide the direct evidence for reasoning; the DFS benchmark provides mechanistic explanation.

Additionally, the x-axis labels in Figure 5D are "depth" numbers (6, 8, 10, ..., 20), but the paper doesn't report the corresponding sequence lengths. For budget 2048, at what depth does the sequence exceed 2048 tokens? The DFS simulation probably generates tokens at a roughly constant rate per depth step, so the depth at which TriAttention begins to degrade (18+) may simply reflect the point where even TriAttention's superior scoring cannot fit all intermediate states within 2048 tokens. Reporting sequence lengths would clarify whether this is a budget saturation effect or a failure of the scoring function.

Missing Experiments and Ablations

Several experiments would have significantly strengthened the paper:

TriAttention on the base models of DeepSeek-R1-Distill variants. The distilled models are fine-tuned from base models for reasoning. Testing on the non-fine-tuned base models would reveal whether TriAttention's advantages are specific to reasoning-tuned models or are general properties of the architecture. The paper doesn't do this.
Observation window sensitivity for post-RoPE baselines. The paper claims post-RoPE methods are limited by observation window size, but it never varies the window size for SnapKV or R-KV to demonstrate that larger windows don't help (or hurt). Zhang et al. (2025) is cited for this claim, but the paper should replicate this finding on its own models to verify that TriAttention's advantage is specifically due to avoiding the observation window, not due to better hyperparameter tuning.
Pre-RoPE vs. post-RoPE versions of the same scoring function. What would happen if you computed the trigonometric series scoring using post-RoPE Q centers instead of pre-RoPE? The centers would be position-dependent, but one could attempt to average over positions. This ablation would directly test whether the pre-RoPE space specifically matters, or whether any Q-center-based scoring would work. The paper doesn't run this.
Head-specific budgets. All experiments use a uniform KV budget shared across all heads. The paper briefly mentions "head-specific budgets" as future work (Appendix A) but doesn't evaluate it. Different heads have different distance preferences — a local head may need only a few nearby keys, while a retrieval head needs to cover the full context. Assigning budgets proportional to each head's effective attention span might yield further improvements.
Latency breakdown. The throughput numbers (Table 4) show end-to-end speedups, but there's no breakdown of where the time is spent: the trigonometric series computation, the norm scoring, the top-k selection, the cache management. This makes it hard to assess whether further optimization of the scoring function would yield meaningful throughput gains or whether the bottleneck is elsewhere (e.g., the top-k selection itself).
Comparison with token dropping during prefill. Some KV cache methods drop tokens during the prefill phase rather than during generation. TriAttention is generation-phase only — it prunes the cache periodically during decoding. Comparing against prefill-phase compression methods (which start with a compressed cache) would test whether TriAttention's importance scoring complements or overlaps with prefill compression.

Summary Assessment

The experiments strongly support the paper's central claim that pre-RoPE-based key importance estimation outperforms post-RoPE approaches on reasoning benchmarks, with large and consistent margins. The ablation studies effectively validate the individual contributions of the trigonometric series term (+23.3 points), the norm-based term (+5.4 points), and the concentration-based weighting (+4.2 points on AIME25), and the offset design choices (geometric spacing adds +17.1 points; long-range offsets add +7.1 points). The calibration sensitivity experiments provide robust evidence for the model-intrinsic property claim, albeit on a single model family. The memory retention benchmark provides mechanistic evidence that TriAttention preserves long-range dependencies better than observation-window methods.

The primary experimental limitations are: small AIME test sets (30 problems each) without confidence intervals; heavy reliance on Qwen3-8B for all detailed analyses with only summary results on other models; a custom, unvalidated memory retention benchmark; and missing ablations that would further clarify the pre-RoPE-vs-post-RoPE distinction, observation window effects, and computational bottlenecks. The breadth of baseline coverage (8 comparison methods) is a strength, as is the cross-architecture validation showing the concentration phenomenon in both GQA and MLA models.

The results are consistent enough across four models and multiple benchmarks to establish TriAttention as superior to existing compression methods for the tested models and tasks. The claim that observation-window methods have a fundamental ceiling is supported by the persistent gap at even the highest KV budgets (4096 tokens), where both methods have ample memory and the performance difference must come from importance estimation quality rather than memory efficiency. Whether the pre-RoPE approach generalizes to non-reasoning tasks, non-transformer architectures, or production deployment at scale is not addressed by the current experiments and remains open.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Included in the Efficiency Numbers

The assumption or constraint. TriAttention requires offline calibration to compute per-head Q statistics — the Q centers E[q_f], expected norms E[∥q_f∥], and Mean Resultant Lengths R_f. The paper frames this as a one-time pre-deployment cost and excludes it from all efficiency calculations. It explicitly states in Appendix H that calibration stabilizes with as little as 50K tokens, and argues the statistics are robust across domains. However, the cost of this calibration — running the full model over the calibration dataset, extracting pre-RoPE Q/K vectors from every head at every position, and computing the statistics — is neither measured nor reported anywhere in the paper. For a model like Qwen3-8B with 1152 attention heads and 64 frequency bands per head, this means storing and averaging approximately 150K scalar statistics, each computed from potentially millions of Q/K vector samples.

The consequence. The 2.5× throughput and 10.7× memory reduction figures (§5.2.4, Table 4, Figure 1) represent inference-time gains only, without amortizing the calibration cost. For a deployment that processes millions of queries (where calibration cost amortizes to near zero), this is a reasonable accounting. But for deployments that process only a small number of queries after calibration — or that need to re-calibrate frequently due to model updates or distribution shift — the calibration cost could dominate. If calibration requires processing 200K tokens (the minimal size showing stable performance in Appendix H, Table F), and inference involves processing 32K tokens per query, the calibration cost equals roughly 6 inference sessions. For a one-off evaluation or a low-volume deployment, the amortized efficiency gain is substantially smaller than the headline numbers suggest.

What evidence exists in the paper. The calibration sensitivity experiments (Appendix H, Table F) show performance is stable from 50K to 960K tokens, and is robust to calibration data quality (HTML vs. chat data). But these experiments measure downstream accuracy, not calibration cost. The paper provides no wall-clock time, FLOP count, or memory consumption for the calibration step itself. The throughput measurements in Table 4 are for inference only.

Mitigation status. The paper does not acknowledge this as a limitation. The calibration is treated as a negligible one-time cost, and the paper does not discuss scenarios where re-calibration is needed or the cost might matter. The claim that calibration data barely matters (Appendix H) suggests that a single calibration suffices for all deployments of a given model, which mitigates the concern for high-volume production settings. But the actual cost of calibration is never quantified, leaving practitioners without the information needed to assess whether the amortized efficiency gains are positive for their specific deployment volume.

TriAttention Is Not Evaluated on Non-Reasoning Tasks Beyond Limited LongBench/RULER Tests

The assumption or constraint. The paper's main evaluation — AIME24, AIME25, MATH 500 — is exclusively on mathematical reasoning benchmarks requiring multi-step chain-of-thought generation. The method is motivated by the observation that long reasoning chains create severe KV cache bottlenecks, and the claimed advantages (2× accuracy over R-KV, matching Full Attention) are demonstrated primarily in this domain. The paper extends evaluation to LongBench (16 subtasks) and RULER (retrieval tasks) in Appendix F, showing TriAttention leading all baselines, but these results are reported as aggregate averages with limited per-task analysis. The paper does not evaluate on other long-context scenarios where KV cache compression matters: long-document summarization, multi-turn dialogue, in-context learning with many examples, or retrieval-augmented generation with long retrieved contexts.

The consequence. We cannot assess whether TriAttention's advantages are specific to reasoning tasks or generalize to all long-context scenarios. The mathematical reasoning benchmarks share structural properties — they require preserving intermediate states for backtracking, the reasoning follows a relatively linear chain with occasional revisiting of prior steps, and the important tokens are often those containing intermediate conclusions rather than substantive content. In contrast, a long-document summarization task might have importance concentrated in key sentences scattered throughout the document, with no backtracking requirement. A multi-turn dialogue might have importance concentrated in recent turns plus occasional references to earlier context. Without evaluation on these domains, we do not know whether the trigonometric series distance preferences — which TriAttention relies on — are equally informative when the attention patterns differ fundamentally from those in mathematical reasoning.

The LongBench and RULER results (Appendix F, Tables B–D) provide some evidence of generalization — TriAttention wins 11 of 16 LongBench subtasks and leads RULER by +5 points over the next-best method. But the average gap on LongBench is modest (+2.5 over Ada-KV+SnapKV), and on individual subtasks TriAttention sometimes loses (e.g., PaRe at 7.0 vs. 8.0 for Ada-KV and PyramidKV). This suggests that while TriAttention is generally strong, its advantage varies substantially by task type, and the paper does not analyze which task characteristics favor or disfavor the pre-RoPE approach.

What evidence exists in the paper. Appendix F (Tables B–D) provides LongBench and RULER results. Table B shows per-subtask breakdown, confirming task-level variation. Table C shows RULER with 4K context and 50% KV budget. But these are presented as appendices without the same depth of analysis as the main reasoning results — no budget scaling curves, no ablation of scoring components per task type, no analysis of which heads are most important for which tasks.

Mitigation status. The paper acknowledges the scope limitation implicitly but does not address it as a limitation. The future work section (Appendix A) mentions extending "evaluation to broader domains such as coding and agentic tasks," which suggests the authors recognize the current evaluation is domain-narrow. The OpenClaw deployment demonstration (Appendix J) provides a qualitative example of TriAttention working in an agentic, multi-turn setting with document reading, but this is a single demonstration rather than a systematic evaluation.

The Method's Effectiveness Depends on Q/K Concentration, Which Is Not Guaranteed to Hold Across All Future Models and Architectures

The assumption or constraint. The entire TriAttention method rests on Q/K concentration — the empirical observation that pre-RoPE Q and K vectors cluster tightly around non-zero centers. The paper quantifies this via Mean Resultant Length and shows that 84.7% of Qwen3-8B heads (GQA) and 96.6% of GLM-4.7-Flash heads (MLA) have R > 0.95 (Appendix I, Table G). The trigonometric series approximation, and hence the reliability of S_trig as an importance score, depends directly on this concentration. When R_f is low — meaning Q vectors in frequency band f are dispersed rather than concentrated — the center approximation degrades, and the paper's response is to increase the norm-based term's contribution via the (1 − R_f) weighting. However, this only partially addresses the problem: the norm-based term captures token salience (magnitude) but provides no positional information. If a head has low concentration across all bands, neither S_trig (unreliable due to poor center approximation) nor S_norm (no positional signal) can accurately estimate which distances the head prefers — the scoring reduces to a token-salience heuristic that ignores distance entirely.

The consequence. TriAttention's performance may degrade substantially on models where Q/K concentration is lower than in the tested architectures. The paper tests four GQA models (Qwen3-8B, DS-Llama-8B, DS-Qwen-7B, GPT-OSS-20B) and reports concentration statistics for one MLA model (GLM-4.7-Flash, Appendix I), all of which show high concentration. But these models represent a specific slice of architectural design space — all use RoPE positional encoding, all are dense transformers, and all are from a similar generation of models (2024–2025). Future models with different design choices — alternative positional encodings, sparse attention patterns, different training objectives, or architectures that diverge from the standard transformer — may not exhibit the same concentration property. If a model's Q/K vectors are less concentrated, TriAttention would lose its primary advantage (the trigonometric series) and fall back to what is essentially norm-based scoring with distance-unaware positional heuristics. The paper provides no analysis of how low concentration can go before TriAttention's accuracy drops below that of post-RoPE methods — we do not know whether the concentration threshold at which the method breaks is 84%, 50%, or 20% of heads having R > 0.95.

What evidence exists in the paper. Figure 2(C) shows the distribution of R values for Qwen3-8B (GQA), confirming high concentration. Appendix I Table G compares concentration between Qwen3-8B (GQA) and GLM-4.7-Flash (MLA), showing MLA has even higher concentration. But these are only two architecture families out of the many in deployment. The paper does not evaluate on architectures where concentration might be lower — no experiments on models with different positional encodings (ALiBi, learned positional embeddings, no positional encoding), sparse mixture-of-experts architectures, or models trained with different objectives that might affect attention head specialization.

Mitigation status. The paper does not discuss this as a limitation. It presents Q/K concentration as a prevalent, stable, and model-intrinsic property, supported by cross-architecture evidence (three GQA families, one MLA family). The implication is that concentration is a universal property of RoPE-based transformers. But this is an extrapolation from a limited sample — the paper provides no theoretical argument for why concentration must occur, only empirical evidence that it does occur in the tested models. A practitioner considering TriAttention for a model not in the tested set would need to measure concentration themselves before assuming the method will work. The paper provides the methodology for this measurement (calibration procedure, MRL computation) but does not provide guidance on what R values constitute "sufficient" concentration for reliable performance.

The Memory Retention Benchmark Is Custom and the Mapping to Real Reasoning Tasks Is Unvalidated

The assumption or constraint. To demonstrate that TriAttention preserves long-range dependencies better than observation-window methods, the paper introduces a Recursive State Query benchmark (§5.2.2, Appendix C) based on depth-first search simulation. The benchmark requires the model to simulate DFS on a graph, track the current node, stack state, and visited nodes, and report these after a specified number of steps. The paper argues that DFS naturally tests memory retention because "the model must descend through nested calls, retain intermediate states, then backtrack to produce results" (Appendix C.2). The key result (Figure 5D) shows R-KV suffering catastrophic degradation at depth 16 while TriAttention matches Full Attention through depth 16 and degrades only gradually thereafter. This is interpreted as evidence that TriAttention preserves long-range dependencies that R-KV loses due to observation-window limitations.

The consequence. DFS simulation is a highly structured, deterministic algorithmic task. Real mathematical reasoning on AIME or MATH 500, while requiring backtracking and intermediate state retention, is far less rigid. In DFS, the model must recall specific nodes and stack states in exact order — a narrow, well-defined memory requirement. In mathematical reasoning, the model may need to recall a previously derived lemma, an intermediate numerical result, or a partially simplified expression, but these have richer semantic content and may be reconstructible from context even if the exact token is lost. The catastrophic failure of R-KV on DFS at depth 16 may overstate its failure on real reasoning tasks (where the model can sometimes recover from eviction by re-deriving or approximating lost information), and TriAttention's strong DFS performance may not fully translate to superior reasoning on every real task.

More fundamentally, the benchmark is unvalidated — the paper provides no evidence that DFS simulation performance correlates with AIME or MATH 500 reasoning performance across compression methods or budget levels. It is possible that TriAttention excels at DFS because its trigonometric series captures the regular, distance-dependent attention patterns that DFS simulation induces (regular revisiting of states at predictable intervals), while real reasoning produces more irregular attention patterns where the advantage is smaller. The paper presents the DFS results as mechanistic evidence for the observation-window hypothesis, which is valuable, but the leap from "TriAttention preserves states in DFS simulation" to "TriAttention enables better mathematical reasoning" is an inference, not a demonstrated causal relationship.

What evidence exists in the paper. Figure 5(D) shows the DFS results, and Appendix C describes the benchmark design. The paper does not report quantitative correlation between DFS performance and AIME/MATH 500 performance, nor does it analyze whether the specific heads that are important for DFS are the same heads that are important for mathematical reasoning. There is no budget scaling on the DFS benchmark — only a single budget of 2048 is tested, so we cannot assess whether TriAttention's advantage grows or shrinks with memory pressure in this setting.

Mitigation status. The paper does not acknowledge the DFS benchmark's limitations as a proxy for real reasoning memory demands. The benchmark is presented as a direct test of memory retention without qualification about its representativeness. While the design is clever and the results are consistent with the paper's mechanistic claims, the lack of validation against real reasoning tasks means the DFS results should be interpreted as suggestive mechanistic evidence rather than conclusive proof that TriAttention's superiority on AIME is specifically due to better memory retention (as opposed to other aspects of its scoring function).

Headline Efficiency Gains Come with Unquantified Latency and Implementation Overhead

The assumption or constraint. The paper reports throughput (tokens/second) and KV memory reduction as the primary efficiency metrics. TriAttention achieves 2.5× throughput on AIME25 and 6.3× on MATH 500 compared to Full Attention (Table 4). These numbers measure tokens generated per second under batch processing — total generation time divided by total tokens. However, TriAttention adds per-token computation that is not present in Full Attention: at each pruning step (every 128 tokens), it must compute S_trig(k, δ) for every cached key at every future offset δ ∈ D, compute S_norm(k) for every cached key, normalize and aggregate across query heads, sort by score, and evict low-scoring entries. The trigonometric series computation involves, for each of up to B cached keys and each of |D| = 17 offsets, a sum over d/2 frequency bands (typically 64) — roughly B × 17 × 64 cosine evaluations per pruning step. For B = 2048, this is approximately 2.2 million cosine evaluations every 128 tokens, plus the norm computation, per-head normalization, and sorting.

The consequence. The throughput numbers in Table 4 amortize this overhead over all tokens generated, but the overhead is not analyzed separately. We do not know what fraction of the per-token computation time is spent on TriAttention's scoring versus the actual attention computation. This matters for two reasons. First, if the scoring overhead is significant, the throughput gains from KV compression are partially offset by the cost of deciding what to compress — the net gain is smaller than the compression ratio alone would suggest. Second, and more importantly for latency-sensitive applications, the scoring computation is bursty: most tokens incur no overhead (they are generated during a window without pruning), but every 128th token triggers a large batch of scoring computation. This creates latency spikes at pruning boundaries — the time to generate the 128th token in a window is substantially longer than the time to generate the 127th. For interactive applications where consistent per-token latency matters (e.g., streaming responses to users), these spikes may be unacceptable even if average throughput is high.

The paper also does not report time to first token with TriAttention. During the prefill phase (processing the input prompt), the KV cache fills with prompt tokens. The first pruning event occurs after 128 generated tokens, so the initial 128-token generation phase runs with a full (or near-full) cache containing all prompt tokens plus the first 128 generated tokens — potentially much larger than the budget B. This means the early generation phase may be slower than the steady-state phase after pruning begins, affecting user-perceived latency.

What evidence exists in the paper. Table 4 reports end-to-end throughput, and the throughput comparison with R-KV (Table 5) shows TriAttention at comparable or slightly higher throughput than R-KV at the same budget (1,345 vs. 1,405 tokens/s), suggesting the overhead is not dramatically larger than R-KV's. But the paper provides no latency breakdown, no per-step timing analysis, no measurement of latency variance, and no comparison of prefill latency. The future work section (Appendix A) mentions "development of a dedicated, high-performance inference kernel" for further acceleration, which implicitly acknowledges that the current implementation is not optimized, but does not quantify the current overhead.

Mitigation status. Not addressed. The paper's efficiency claims are based on throughput, which is the appropriate metric for batch processing but masks latency spikes. A practitioner considering TriAttention for an interactive deployment would need to measure latency variance independently — the paper provides no data to assess whether TriAttention is suitable for latency-sensitive applications. The window-based pruning design (β = 128) is a tradeoff between scoring overhead and memory growth that the paper inherits from R-KV without analyzing the latency implications of this specific choice. Smaller window sizes would reduce burstiness but increase overhead; larger windows would reduce overhead but increase memory peaks and latency spikes. The paper explores neither the overhead budget nor alternative windowing strategies.

The 14× Larger Model Baseline in FLOPs-Matched Comparisons Is Absent

The assumption or constraint. The paper makes strong claims about TriAttention enabling deployment on consumer hardware that Full Attention cannot support: the OpenClaw deployment (Appendix J) runs Qwen3-32B on a single RTX 4090, and Full Attention causes out-of-memory while TriAttention completes the task. The paper also emphasizes throughput gains of 1.9× to 6.3× over Full Attention on A100 GPUs. However, the paper does not perform a FLOPs-matched or parameter-matched comparison against a larger model with no compression. That is, it does not ask: if I have a fixed inference budget (memory + FLOPs), is it better to run a smaller model with TriAttention or a larger model with Full Attention? This is the inference-time analog of the pretraining-inference tradeoff studied in the reference example paper (Section 7 of that paper), and it is a natural question for practitioners making deployment decisions.

The consequence. The paper's results show that TriAttention enables Qwen3-8B to achieve accuracy close to Full Attention Qwen3-8B while using 10.7× less KV memory. But what if the freed memory were used to load a larger model — say, a Qwen3-14B or Qwen3-32B — with Full Attention (or with a simpler compression method)? The paper does not provide this comparison. On AIME24, Full Attention Qwen3-8B achieves 57.1% (Table 1). TriAttention at budget 4096 achieves 54.6% (Table 4). A hypothetical Qwen3-14B with Full Attention might achieve 65%+ — substantially higher than either — and might fit in the same memory as Qwen3-8B + TriAttention KV cache. Without this comparison, the paper cannot claim that TriAttention represents an optimal allocation of the total memory budget; it only demonstrates that TriAttention is better than Full Attention for the same model and better than other compression methods. The OpenClaw deployment shows TriAttention enables a model to fit where it otherwise wouldn't, but it does not ask whether a smaller model without compression would have sufficed for the task.

This is particularly relevant because KV cache compression reduces memory for the cache but does not reduce the memory for model weights. For large models, model weights dominate total memory. Qwen3-8B (8B parameters, ~16GB in bfloat16) leaves ~64GB for KV cache on an 80GB A100 — the KV cache compression primarily helps with batch size and throughput, not with whether the model fits. On consumer GPUs like the RTX 4090 with 24GB, Qwen3-32B at INT4 (~16GB for weights) leaves only ~8GB for KV cache, making compression essential — but the paper tests this only qualitatively with OpenClaw, not with systematic accuracy benchmarking on AIME or MATH 500.

What evidence exists in the paper. None. The paper does not perform any FLOPs-matched, parameter-matched, or memory-matched comparison against larger uncompressed models. The models tested (8B and 20B) are evaluated only against Full Attention at the same model size and against compression baselines at the same model size. The OpenClaw demonstration (Appendix J, Figure C) shows feasibility but is not an accuracy benchmark.

Mitigation status. Not addressed. The paper frames its contribution as enabling long reasoning on hardware-constrained deployments, but it never directly answers the question: for a fixed GPU memory budget, what model size + compression strategy maximizes accuracy? This is a significant gap for practitioners making deployment decisions, because the choice is rarely "use compression or not" for a fixed model — it is "which model size and compression strategy gives the best accuracy within my hardware constraints?" The paper's results are consistent with the hypothesis that TriAttention enables a smaller model + compression to outperform a larger model without compression in some regimes, but this hypothesis is never tested.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a paradigm-level reframing of KV cache compression, not an incremental improvement within the existing paradigm. The dominant approach since H2O (2023) has been to estimate key importance from post-RoPE attention scores observed during a recent window of decoding. The field's research has focused on improving the aggregation of these observations — H2O's accumulated attention, SnapKV's voting within a local window, R-KV's redundancy detection — without questioning whether the underlying signal was reliable. TriAttention's core move is to reject the post-RoPE observation paradigm entirely and instead exploit a structural property of pre-RoPE representations that makes importance estimation stable without any observation window.

This is not a "better SnapKV." It is a different category of method. The implications cascade through how the field should think about the problem:

The observation-window limitation is now a diagnosed ceiling, not an open optimization problem. Zhang et al. (2025) empirically observed that increasing the observation window beyond ~25 queries degrades performance, but the reason for this counterintuitive result was unclear. TriAttention provides the mechanistic explanation: queries rotate with position under RoPE, so older queries have orientations misaligned with future queries, making their attention scores actively misleading for importance estimation. This transforms the observation-window problem from an empirical curiosity into a fundamental constraint — it is a geometric consequence of RoPE, not an implementation detail. Future research on post-RoPE compression methods now faces a known and quantified ceiling: if the effective observation window is capped at ~25 tokens, no amount of algorithmic sophistication within that window can capture tokens that become important more than ~25 steps after they were last attended to. This makes further incremental work on attention-score aggregation within the post-RoPE paradigm significantly less attractive.

The pre-RoPE space becomes the natural locus for structural analysis of attention mechanisms. Prior interpretability work on attention heads — classifying them as retrieval, streaming, local, sink — analyzed the outputs of attention (the attention patterns themselves) rather than the parameters that produce them. TriAttention demonstrates that these attention patterns are causally determined by the Q/K centers (and their concentration) in pre-RoPE space, and that the relationship is mathematically derivable via the trigonometric series. This transforms the study of attention heads from a descriptive enterprise ("heads of type X attend at distance Y") to a mechanistic one ("the Q/K centers [a_f, b_f] cause distance preference Y via the trigonometric series"). This opens interpretability research directions that were previously inaccessible: one could now ask why pretraining produces certain Q/K centers, whether specific training objectives or data distributions predictably shift these centers, and whether different center configurations correspond to functionally distinct computational roles in the model.

Q/K concentration emerges as a model-intrinsic property with practical and theoretical significance. The paper's evidence that concentration is nearly identical across Math, Coding, and Chat domains (MRL 0.977–0.980), and that calibration on low-quality HTML data performs comparably to high-quality chat data (46.2% vs. 46.7% on AIME24, Appendix H), suggests this is not a learned adaptation to any specific task distribution — it is a structural consequence of how RoPE-based transformers converge during pretraining. This is a newly identified empirical regularity with no current theoretical explanation. Why would gradient descent on next-token prediction produce Q/K vectors that cluster tightly around fixed centers? Is it a consequence of the RoPE geometry itself — perhaps the sinusoidal positional encoding creates a loss landscape where constant Q/K vectors represent a local optimum — or does it emerge from the interaction of attention with the training data distribution? Answering this could deepen our understanding of transformer training dynamics and potentially inform architecture design (e.g., should we explicitly encourage or exploit concentration rather than treating it as an accidental property?).

Verifier over-optimization is not the bottleneck for KV compression — observation-window instability is. The reference example paper on test-time compute scaling identified verifier over-optimization as the central bottleneck preventing unbounded improvements from additional inference compute. TriAttention identifies a different bottleneck for a different problem: KV cache compression accuracy is limited not by how well importance scores are aggregated (the analog of verifier quality) but by whether representative importance signals can be observed at all. Post-RoPE methods suffer from a fundamental sample inefficiency — the effective sample size for learning token importance is capped at ~25 representative queries, regardless of how long the sequence is. TriAttention bypasses this bottleneck by using Q/K centers computed from calibration data, which effectively provides an unlimited sample size (every token in the calibration dataset contributes to the center estimate). This reframes the competition: it is no longer about better algorithms for processing observed attention scores; it is about finding representation spaces where importance can be estimated from structural properties rather than runtime observations.

The field now has permission to look beyond attention scores for KV importance. The dominance of attention-based compression methods created an implicit assumption that attention scores are the most principled signal for importance — after all, attention scores define which tokens influence the output. TriAttention demonstrates that a mathematically derived score from pre-RoPE structure can substantially outperform methods that use actual attention scores. This is counterintuitive — how can a predicted score based on centers outperform the actual attention scores that determine the model's computation? The answer is that attention scores from recent queries are a biased and incomplete sample of future attention scores due to positional rotation, while the trigonometric series score, though approximate, is an unbiased and complete estimate of the expected attention over all future positions. The lesson generalizes: for problems where the quantity to be predicted (future attention) is observed through a structurally limited window (recent queries), a structural model of the underlying process may outperform direct observation, even when the model is approximate. This has implications for other inference-time optimization problems where observation windows are limited (speculative decoding, early exiting, dynamic compute allocation) — structural models of the computation may be more reliable than runtime heuristics.

Follow-Up Research This Work Enables

Quantifying the observation-window ceiling for post-RoPE methods as a function of RoPE frequency and model architecture. The paper argues that post-RoPE methods are limited because queries rotate with position, but it never directly measures how the effective window size depends on RoPE hyperparameters (base frequency θ, dimension d, frequency progression). A systematic study would: vary θ across values (10000, 500000, 1000000), measure the correlation between a query at position p and a query at position p + Δ as a function of Δ, and determine the Δ at which this correlation drops below some threshold (e.g., r < 0.5). This would produce a curve: effective window size vs. θ. If the effective window size increases with larger θ (which slows the rotation rate), then one could predict a priori which base frequencies make post-RoPE methods more viable. Conversely, if the effective window is always ~25 regardless of θ — perhaps because the limiting factor is not rotation rate but the dimensionality of the frequency bands — that would strengthen the paper's claim that the limitation is fundamental rather than a tunable parameter. This experiment would also test whether TriAttention's advantage shrinks for models with very large θ, where post-RoPE methods might have observation windows large enough to cover the full reasoning chain.

Characterizing the relationship between Q/K concentration and pretraining: does concentration emerge or is it learned early? The paper shows that concentration is model-intrinsic and stable across domains, but does not investigate when during pretraining it emerges. A training-dynamics study would: take intermediate checkpoints from a model's pretraining run (e.g., Qwen3-8B at 10B, 100B, 500B, 1T tokens), measure the Mean Resultant Length distribution across heads at each checkpoint, and track whether concentration increases gradually, emerges suddenly at some phase transition, or is present from very early in training. If concentration emerges early and stabilizes, it suggests it is a consequence of the architecture rather than the training data, and TriAttention could be applied to partially-trained models. If it develops gradually and correlates with downstream performance, it becomes a potential diagnostic for training quality. A strong follow-up would also track the reconstruction correlation r̄ over training — does the trigonometric series become more predictive of attention as concentration increases? This would directly validate the causal claim that concentration drives predictability. The calibration dataset would be fixed (say, the Pile validation split), and the same 10K-token test sequence would be used for reconstruction correlation at each checkpoint.

TriAttention with head-specific KV budgets: exploiting the heterogeneity of distance preferences. The paper treats all heads identically — same budget B, same future offset set D, same scoring formula. But Figure 2(C) shows substantial variation in concentration across heads, and the reconstruction correlation analysis (Figure 3) shows that not all heads are equally well-predicted by the trigonometric series. This suggests that the optimal strategy may be head-specific: heads with very high concentration and high reconstruction correlation — where the trigonometric series is near-perfect at predicting attention — could be allocated smaller budgets because TriAttention reliably identifies their important keys. Heads with lower concentration — where the norm-based term dominates — might need larger budgets because importance estimation is less precise. A concrete experiment: (1) For each head, compute its concentration (MRL) and reconstruction correlation r̄ from calibration data. (2) Allocate the total KV budget across heads proportional to (1 − r̄ × R̄), so that heads with both low concentration and low reconstuction get more memory. (3) Evaluate on AIME24/25 compared to uniform budget allocation. The hypothesis is that head-specific allocation would improve accuracy at the same total budget, because memory is shifted from heads where TriAttention already identifies important keys perfectly (and can therefore be aggressive) to heads where its estimates are noisy (and therefore need a buffer). This experiment does not require changing TriAttention's scoring function — only the budget allocation — making it a low-implementation-overhead extension.

Can the trigonometric series be used for head-level importance, enabling structured pruning of entire attention heads during inference? The paper focuses on pruning individual keys within each head's cache. But the reconstruction correlation r̄ provides a per-head quality metric for how well the trigonometric series predicts attention. Heads with very low r̄ — where attention does not follow predictable distance preferences — may be heads whose attention patterns are highly content-dependent and irregular. If these heads also contribute little to downstream task performance, one could potentially drop these heads entirely during inference, not just prune their caches. A concrete experiment: (1) For each head, compute r̄ and also measure the head's contribution to output by ablating it (zeroing its attention output) and measuring the drop in next-token prediction loss or task accuracy. (2) Test whether r̄ correlates with head importance — if low-r̄ heads are also low-importance heads, then r̄ serves as a proxy for head importance without requiring expensive ablation experiments. (3) During inference, drop the KV cache entirely for heads with r̄ below some threshold, not just prune within them. This would provide additional memory savings beyond per-token pruning. This experiment connects TriAttention's analysis to the broader literature on structured pruning and attention head importance (Michel et al., 2019; Voita et al., 2019).

TriAttention for cross-sequence KV cache sharing in batched inference. In batched inference serving multiple independent sequences simultaneously, the KV cache memory is the dominant bottleneck for batch size. Currently, each sequence has its own KV cache, despite the fact that different sequences may attend to tokens with similar content — just at different positions. Since TriAttention decomposes importance into a position-dependent component (the trigonometric series) and a content-dependent component (the norm-based score), it suggests a natural extension: share the norm-based importance across sequences by using a content-addressable memory structure. Sequences that contain tokens with similar key representations (similar norm profiles) could share the physical storage for those tokens, while each sequence maintains its own positional mapping. A concrete experiment: (1) For a batch of B sequences, compute the norm-based importance S_norm(k) for each token in each sequence. (2) Cluster tokens across sequences by their key representations (e.g., using k-means on the key vectors). (3) For each cluster, retain only the top-M highest-scoring tokens across all sequences, shared in a cross-sequence cache. (4) Each sequence's attention computation retrieves from both its private cache (for position-specific tokens) and the shared cache (for content-similar tokens). (5) Measure accuracy vs. memory on AIME with batch size > 1. This would test whether TriAttention's decomposition enables a new category of shared KV cache that is not possible with pure attention-based importance scoring (since attention scores are sequence-specific).

Is concentration a property of RoPE, or of the transformer architecture more broadly? The paper tests GQA and MLA architectures, all with RoPE. Would the same concentration phenomenon appear in models with alternative positional encodings — ALiBi, learned absolute positional embeddings, T5-style relative position biases, or no positional encoding at all? This is a critical stress test because if concentration is specifically a consequence of RoPE's rotational structure — perhaps the rotation forces Q/K vectors into a compact representation to maintain stable attention patterns across positions — then TriAttention's approach would not transfer to non-RoPE architectures, limiting its generality. A concrete experiment: (1) Take a non-RoPE model (e.g., a model with ALiBi or learned positional embeddings) that still uses standard multi-head attention. (2) Measure pre-RoPE concentration — or pre-positional-encoding concentration — using the same MRL metric. (3) Attempt to fit a trigonometric-like series, replacing the RoPE cosine term with whatever positional interaction the architecture uses. (4) Measure reconstruction correlation r̄. If concentration is absent or reconstruction correlation is near zero in non-RoPE architectures, it localizes the phenomenon to RoPE specifically. If concentration persists, it suggests a deeper property of how transformers learn to allocate attention — perhaps attention heads naturally converge to low-rank representations (near-constant Q/K) as a consequence of the softmax bottleneck or gradient dynamics, independent of positional encoding.

Online adaptation of Q centers during inference. The paper computes Q centers once from offline calibration data and treats them as fixed for all inference. But the calibration data (ShareGPT chat) may not perfectly match the model's distribution during long reasoning chains — the model's internal representations might shift subtly as it generates chain-of-thought, particularly for models fine-tuned on reasoning data (like the DeepSeek-R1 distills). An online adaptation scheme would: (1) During the first few windows of generation (before severe pruning begins), collect actual Q vectors from the decoding process. (2) Compute a running estimate of the Q center, blending the offline calibration center with the online estimate using an exponential moving average. (3) Use the adapted center for subsequent pruning decisions. This would test whether deployment-time distribution shift affects TriAttention's accuracy and whether simple adaptation can recover it. The experiment would compare offline-only calibration vs. offline+online adaptation on AIME24/25, with the adaptation rate (EMA decay) as a hyperparameter. A null result — no improvement from adaptation — would strengthen the paper's claim that Q/K concentration is genuinely model-intrinsic and invariant even to the distribution shift between chat data and mathematical reasoning. A positive result would reveal that the "model-intrinsic property" claim has boundaries and that adaptation improves robustness at the margin.

Practical Applications and Downstream Use Cases

Deploying frontier reasoning models on consumer GPUs for interactive applications. The OpenClaw demonstration (Appendix J, Figure C) is the paper's most concrete practical scenario: Qwen3-32B (INT4 quantized, ~16GB for weights) on a single RTX 4090 (24GB total) runs out of memory with Full Attention during multi-turn document processing because the KV cache grows beyond the ~8GB remaining after model weights. TriAttention keeps the KV cache within budget, enabling the agent to complete the task. This is not a contrived benchmark — it is a realistic deployment of a capable reasoning agent on hardware that costs under $2000, compared to the$ 10,000+ A100 GPUs typically required for 32B-class models at full context. The practical value is that individuals, small labs, and startups can run state-of-the-art reasoning models locally for complex multi-turn tasks (document analysis, code review, research assistance) without renting cloud GPUs. The specific numbers: at 32K context, Full Attention's KV cache for Qwen3-32B would require roughly 10–12GB (depending on precision), exceeding the available ~8GB on the RTX 4090 after model weights. TriAttention at budget 2048 reduces KV memory by ~10×, fitting comfortably within the remaining memory. The throughput numbers (2.5× speedup at equivalent accuracy on A100, Table 4) suggest the consumer GPU deployment would also be faster than trying to run Full Attention with aggressive memory swapping.

Cost-efficient batch evaluation of reasoning benchmarks at scale. Organizations evaluating LLMs on large reasoning benchmarks (thousands of AIME-style problems, MATH 500, GSM8K) for model selection or monitoring typically run each problem independently with maximum context to avoid truncating reasoning chains. With 32K-token generation, a single A100 can process only a few sequences in parallel before KV cache memory is exhausted. TriAttention's 10.7× KV memory reduction at equivalent accuracy (Figure 1, AIME25) means that approximately 10× more sequences can be batched simultaneously on the same GPU, directly translating batch size improvements into total evaluation throughput. For an organization evaluating 10 candidate models on 10,000 problems each at 32K context, reducing evaluation time from 100 GPU-hours to 10 GPU-hours represents substantial cost savings. The practical implementation would use TriAttention with a conservative budget (e.g., 4096 tokens, which matches or exceeds Full Attention accuracy on AIME25 per Figure 5C) to ensure evaluation fidelity while still achieving ~8× memory reduction. The calibration cost (processing 200K tokens, ~5 minutes on a single A100) amortizes to zero over 100K evaluation queries.

Enabling long-horizon reasoning in memory-constrained edge deployments. Beyond consumer GPUs, there are deployment scenarios where KV cache memory is the binding constraint for model capability: on-device inference on phones (where memory is shared with the OS and other apps), in-browser inference via WebGPU (where GPU memory is severely limited), and embedded systems for robotics or scientific instruments. In these settings, the choice is often not "how much compression?" but "can we run the model at all?" TriAttention's ability to maintain reasoning accuracy at extreme compression ratios — matching Full Attention on AIME25 at budget 4096, and on MATH 500 at budget 1024 (Figure 5A,C) — means that reasoning-capable models previously restricted to cloud deployment can potentially run on-device. A concrete scenario: a phone-based math tutoring app that uses Qwen3-8B (or a distilled 1.5B variant) to provide step-by-step reasoning feedback. Without KV compression, the app would need to either truncate reasoning chains (harming quality) or stream to the cloud (requiring connectivity and incurring latency). With TriAttention at budget 1024, the entire reasoning chain up to several thousand tokens fits in the phone's available memory, enabling offline, low-latency reasoning. The calibration is done once server-side and the statistics (a few hundred KB) are bundled with the app.

When to Prefer This Method

The paper does not explicitly position TriAttention against specific alternatives with a decision rubric, but the experimental results imply clear boundary conditions for when TriAttention is the appropriate choice:

TriAttention is the right choice when:

The model uses RoPE-based positional encoding (nearly all modern open-weight LLMs: Qwen, Llama, DeepSeek, Gemma, Mistral). The Q/K concentration phenomenon has been validated across GQA and MLA architectures in these families (§3.3, Appendix I).
The inference workload involves long generated sequences (≥8K tokens) where the observation-window limitation of post-RoPE methods becomes severe. The paper's results show the largest accuracy gaps vs. R-KV/SnapKV on AIME (15+ points) compared to MATH 500 (4–8 points), reflecting that shorter sequences reduce but do not eliminate the observation-window problem. At very short generation lengths (<2K tokens), the advantage narrows and the calibration cost may not amortize.
The deployment hardware is memory-constrained (consumer GPUs, edge devices) where KV cache memory is the binding constraint rather than model weight memory. If model weights already consume most of the available memory, KV cache compression is essential for any long generation. If the hardware has abundant memory (e.g., 8×A100 for a 7B model), the throughput gains from KV compression are less critical.
Offline calibration is feasible (the model can be run over a small calibration dataset before deployment). This is essentially always true for server-side deployment; it may be a constraint for fully on-device deployment where no pre-processing step can run server-side.
The task benefits from retaining long-range dependencies — reasoning with backtracking (math, code, planning), multi-turn dialogue with distant references, document-grounded QA where evidence is scattered. Tasks where importance is concentrated entirely in recent tokens (e.g., next-sentence prediction, streaming transcription) may not benefit from TriAttention's distance-preference mechanism.

Post-RoPE observation methods (SnapKV, R-KV) may remain preferable when:

The model architecture does not use RoPE, or uses a non-standard positional encoding where Q/K concentration has not been validated. TriAttention's trigonometric series derivation depends on the RoPE rotation formula; for other positional encodings, a different mathematical relationship would need to be derived, and the concentration property may not hold.
Calibration is genuinely impossible — e.g., a user downloads a model and needs to run it immediately without any preprocessing step. In this case, post-RoPE methods that work "out of the box" without calibration may be preferable despite lower accuracy, assuming the accuracy loss is acceptable.
Latency at pruning boundaries is a hard constraint and the bursty computation pattern of window-based pruning (every 128th token triggers scoring of all cached keys) causes unacceptable jitter. Post-RoPE methods also use window-based pruning and have similar burstiness, but the specific computational cost of the trigonometric series (cosine evaluations over all frequency bands and all future offsets) may be higher per key than attention-based scoring — the paper does not provide this breakdown, so this is speculative.

Full Attention (no compression) remains necessary when:

Performance cannot degrade at all from the uncompressed baseline. While TriAttention matches Full Attention at some budgets (e.g., AIME25 at budget 3072 or 4096, Figure 5C), it does not universally match it — there is always a small accuracy gap at most budgets. For applications where every percentage point of accuracy matters (e.g., medical diagnosis, legal reasoning where errors carry high cost), even a 1–2% degradation may be unacceptable.
The KV cache is not memory-constrained — e.g., running a small model (1–3B parameters) on a large GPU (80GB A100) with relatively short generation lengths (<4K tokens). In this regime, the memory is never full, so compression provides no throughput benefit and only adds complexity.