Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction

ArXiv: 2601.17668

🎯 Pitch

Future KV-cache utility is an intrinsic property decodable from input hidden states—not from attention features—but only if you use sink-attention gates trained on reconstruction losses. Fast KVzip proves this, matching KVzip’s accuracy while evicting 70% of cache with zero runtime reconstruction, making it 2× faster in prefill.

---

1. Executive Summary

This paper introduces Fast KVzip, a gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational overhead by integrating lightweight sink-attention gating modules directly into both the prefill and decoding stages. The method is trained using a task-agnostic reconstruction objective — distilling KV importance scores pre-computed from the computationally expensive KVzip reconstruction process — on only 1M tokens of the FineWeb-Edu corpus, requiring less than one H100 GPU hour for 14B-scale models. Across the Qwen2.5-1M, Qwen3, and Gemma3 families on benchmarks spanning prefill-intensive long-context tasks (SCBench, MRCR, up to 170K tokens) and decoding-intensive reasoning tasks (AIME24, MATH), Fast KVzip maintains near-lossless performance while evicting up to 70% of the KV cache, substantially outperforming SnapKV, Expected Attention, DuoAttention, R-KV, and TrimKV while matching the accuracy of KVzip without its runtime reconstruction overhead. The method establishes that future KV utilization is largely an intrinsic property decodable from input hidden states alone — without reconstructing entire contexts — but that this decodability is effective only when the gating architecture uses sink-attention mechanisms rather than linear models or standard MLPs, and only when trained on reconstruction-based rather than next-token-prediction or instruction-QA target signals.

2. Context and Motivation

The Core Problem: The KV Cache Is a Bottleneck That Existing Solutions Can't Handle Efficiently

The fundamental tension this paper addresses is straightforward to state but remarkably difficult to resolve: the KV cache is essential for efficient LLM inference, but its memory footprint scales linearly with both sequence length and batch size, making it the dominant bottleneck for long-context deployments. For every token the model processes during prefill, it must store key and value tensors for every attention head in every layer. For every new token generated during decoding, it must store one more set of KV pairs. This means a model with $L$ layers, $H$ attention heads, and hidden dimension $D$, processing a sequence of length $T$, requires storage for $2 \times L \times H \times T \times D$ floating-point values — a number that grows without bound as context windows expand into the hundreds of thousands of tokens.

The KV cache exists specifically because re-computing attention features from scratch at every autoregressive step would be computationally prohibitive (Dai et al., 2019). Without caching, each decoding step would require a full attention computation over all previous tokens, turning the quadratic cost of attention into a cubic one across the full generation. The KV cache solves this by trading memory for compute: store the intermediate key and value projections once, then retrieve them for subsequent attention computations. This is a standard engineering solution, but it creates a new problem — one that has become increasingly acute as models have grown to support context windows of 128K, 256K, or even 1M tokens.

The paper contextualizes this tension concretely. For a model like Qwen2.5-14B-1M processing a 170K-token context (the scale shown in Figure 1b), the raw KV cache alone can consume tens of gigabytes of GPU memory, rivaling or exceeding the memory required for the model weights themselves. In batched serving scenarios — where multiple sequences are processed simultaneously to maximize throughput — the memory pressure multiplies, constraining either the maximum batch size or the maximum context length the system can support. This is not a theoretical concern: it directly limits what models can be deployed on what hardware, at what cost, and with what latency.

Why This Matters: The Practical and Conceptual Stakes

The practical importance of this problem is immediate and multi-dimensional:

Memory constrains deployment feasibility. GPU memory (HBM) is the scarcest and most expensive resource in LLM serving. A model that can only fit a single sequence in memory because the KV cache consumes the remaining headroom cannot be served efficiently to multiple users. Conversely, a model whose KV cache can be compressed allows larger batch sizes, higher throughput, and lower per-query costs. The economic implications are substantial: if you can serve 4× more queries per GPU by compressing the KV cache to 30%, your infrastructure costs drop proportionally.

Prefill latency is dominated by memory I/O, not compute, at long contexts. The paper highlights this in Figure 10a, which shows that Fast KVzip reduces prefill time substantially compared to the no-compression baseline. The reason is architectural: attention computation is memory-bandwidth-bound during prefill, especially with FlashAttention-2's optimized kernel design (Dao, 2024). Storing fewer KV features means less memory to write, less memory to read during subsequent attention operations, and less data movement overall — all of which translate to wall-clock speed improvements that are not captured by FLOPs counting alone.

Long-context tasks are increasingly central to LLM applications — document understanding, repository-level code comprehension, multi-turn conversational agents, and retrieval-augmented generation all require models to maintain and reason over very long contexts. If the KV cache bottleneck prevents these applications from running efficiently, the gap between model capability and practical deployment widens.

But beyond the practical engineering concerns, there is a deeper conceptual question at stake: is all of the KV cache actually necessary? The existence of successful compression methods — both heuristic ones like H₂O and SnapKV, and reconstruction-based ones like KVzip — demonstrates empirically that large fractions of KV features can be discarded with minimal accuracy loss. This suggests that attention patterns in pretrained LLMs exhibit substantial redundancy, and that not all tokens contribute equally to future predictions. The question then becomes: can we identify which KV pairs matter before we need them, and do so efficiently enough that the identification cost doesn't eat up the compression savings?

Prior Approaches and Their Limitations

The paper organizes existing KV cache compression methods into a taxonomy that reveals a systematic trade-off that no prior method has successfully resolved:

Heuristic sparsity-based methods (H₂O, SnapKV, Finch) are fast but fragile. These approaches observe attention patterns during prefill or decoding and use simple heuristics to identify "important" KV pairs — typically by accumulating attention scores and retaining the top-scoring tokens. H₂O (Zhang et al., 2023) tracks cumulative attention weights and keeps the "heavy hitters" while discarding the rest. SnapKV (Li et al., 2024) analyzes attention patterns from the last few tokens of the prompt to determine which earlier tokens are likely to be attended to. The advantages are clear: these methods add minimal computational overhead (essentially just tracking and sorting attention scores) and require no training. The disadvantage, as the paper argues, is equally clear: "sparsity patterns inferred at inference time often overfit the current input and fail to generalize across future queries" (Section 2). In multi-query settings — where a single long context is queried multiple times with different questions — a heuristic that evicts based on a single query's attention pattern will discard information needed for subsequent queries. This is the fundamental fragility: without knowing what future queries will ask, heuristic methods are making irreversible decisions based on partial information.

Query-agnostic methods (KVzip, Expected Attention, EpiCache) are robust but slow. Recognizing the multi-query fragility, these approaches estimate KV importance independently of any specific query. KVzip (Kim et al., 2025a) takes the most direct approach: during prefill, it temporarily stores all KV features, then reconstructs the entire context by feeding each sentence back through the model and recording which KV pairs receive high attention scores during this reconstruction. These scores serve as a query-agnostic importance metric — tokens that are consistently attended to when reconstructing the context are likely to be important for any query about that context. Expected Attention (Devoto et al., 2025) takes a different approach, modeling the expected attention that a KV pair would receive under a distribution of future queries (approximated using mean query statistics).

The problem is cost. KVzip's reconstruction process requires, during prefill, re-feeding the entire context through the model — essentially doubling the prefill computation. Figure 1b quantifies this concretely: at 170K tokens, KVzip's prefill time is substantially higher than both the no-compression baseline and Fast KVzip. This makes KVzip impractical for latency-sensitive deployments: the compression overhead cancels out, or worse, the memory savings it provides. The paper characterizes this as the fundamental dilemma: "methods with negligible compression overhead result in significant performance degradation, while methods that preserve accuracy tend to incur prohibitive compression overhead" (Section 1).

Architecture-modification methods (DuoAttention, sliding-window) require retraining. DuoAttention (Xiao et al., 2025) predefines attention patterns at the head level — some heads are designated for streaming attention (local window only), others for full-context retrieval — based on calibration data analysis. While effective, this approach requires modifying the model architecture and retraining or extensive fine-tuning, making it incompatible with the frozen-weight deployment scenario that dominates practical LLM serving. Most practitioners do not have the resources to retrain a 14B-parameter model to add compression capabilities; they need methods that work with existing model weights.

Gating-based methods (Locret, TrimKV, DMS) exist but are limited or fragile. The paper acknowledges that using learned gates to predict KV importance is not a new idea. Locret (Huang et al., 2024) trains gating modules on instruction-based QA data but is confined to prefill-stage compression only — it does not address the decoding phase where KV caches continue to grow. TrimKV (Bui et al., 2025) trains gates using next-token prediction as the training objective and demonstrates results on mathematical reasoning, but "suffers from performance degradation in prefill-intensive scenarios, specializing in narrow domains of mathematical reasoning" (Section 2). The paper provides evidence of this fragility in Figure 5 (discussed in technical detail in Section 3.3 of the paper): gates trained on next-token prediction or instruction-QA targets fail to generalize to retrieval tasks because their target signals encode task-specific attention patterns that don't transfer. Figure 6 visualizes why: reconstruction attention patterns are "uniform and structured," next-token prediction patterns are "denser," and QA patterns are "sparse" — each reflecting fundamentally different information needs. A gate trained on one pattern evaporates on tasks requiring another.

No existing method achieves the accuracy-efficiency frontier. The state of the field, as the paper sees it, has a clear gap: you can be fast (heuristic methods, ~1% overhead, but accuracy drops at high compression ratios), or you can be accurate (KVzip, maintains near-lossless performance at 30% budget, but doubles prefill time), but you cannot be both. Fast KVzip explicitly targets this gap: match KVzip's compression quality while eliminating its runtime reconstruction overhead.

How This Paper Positions Itself

The paper's positioning is built on a single key insight articulated in the introduction:

"the future utilization of KV pairs is largely an intrinsic property that can be directly decoded from the input hidden states, without reconstructing the entire context as in KVzip"

This claim is both conceptual and practical. Conceptually, it asserts that whether a particular key-value pair will be useful for future attention is not something that requires looking at the entire context and all possible queries — it is latent in the hidden representations themselves, at the moment they are computed. This is a strong claim about the nature of attention in pretrained LLMs: it suggests that the model's own internal representations contain enough information to predict which of its own outputs (the KV features) will matter later, without needing to simulate future attention explicitly.

Practically, this insight enables a clean architectural solution: add a lightweight gating mechanism to each attention layer that takes the current hidden state as input and outputs importance scores for the KV features being produced. If the gating mechanism can accurately predict KV importance from local information alone, then compression becomes essentially free — it piggybacks on the forward pass that was already happening, rather than requiring a separate reconstruction pass.

But the paper is careful not to present this as trivial. The extensive empirical analysis around gate training targets (Figure 5), gate inputs (Figure 7), and gate architectures (Figure 9) reveals that realizing this insight requires getting several non-obvious design choices right:

• Training target matters fundamentally. Distilling KVzip's reconstruction scores works; training on next-token prediction or QA attention patterns doesn't generalize. This validates the insight that reconstruction captures a task-agnostic property of KV utility, while also showing that not all "importance signals" are created equal.

• Hidden states outperform key states as gate inputs. Figure 7 shows that using the raw hidden states — before the key projection — as gate inputs outperforms using the projected key features or pre-RoPE key states. This is non-obvious: one might expect the key features to be more informative about KV importance since they are what attention operates on. The paper suggests that hidden states "contain richer information for predicting the future utility of KV features" (Section 3.3), and that position-encoded features actually degrade performance, motivating a separation between positional information (handled by a local window) and content-based importance (handled by the gate).

• Sink-attention architecture outperforms MLPs and linear models. Figure 9 shows that a simple linear gate underperforms, and even a two-layer MLP with SwiGLU activation (matching parameter count) falls short on retrieval tasks. The sink-attention mechanism — inspired by the observation that attention sinks (Xiao et al., 2024) play a critical role in standard attention — provides learnable "sink keys" that serve as an adaptive baseline for importance scoring. This architectural choice is grounded in attention theory rather than being an arbitrary design decision.

The paper also positions itself within a broader conceptual framework by reformulating several prominent compression methods as special cases of a generalized gating formulation (Figure 3, Table 1). DuoAttention is cast as a constant Boolean gating function mapping hidden states to head-level binary decisions. Expected Attention is cast as a quadratic gating function mapping key states to importance scores. FastGen's pattern identification is cast as a Boolean gating function on input tokens. This reframing is analytically productive: it transforms KV cache compression from "designing heuristics" to "optimizing gating functions," enabling a principled, data-driven approach where the model learns the optimal compression strategy rather than having it hand-crafted by engineers.

The relationship to KVzip is central to the paper's positioning. Fast KVzip is explicitly not trying to outperform KVzip in terms of compression quality — the results consistently show the two methods matching each other (Figures 11, 12). Instead, Fast KVzip aims to match KVzip's quality while eliminating its runtime overhead by moving the expensive reconstruction from inference time to a one-time pre-deployment training phase. This is a distillation play: KVzip runs the full reconstruction process, computes importance scores, and these scores become the supervised training targets for Fast KVzip's lightweight gates. After training, the gates can predict importance scores from hidden states alone, with no reconstruction needed. This positions KVzip not as a competitor but as a teacher — a computationally expensive oracle whose knowledge is compressed into a fast, deployable student.

Finally, the paper positions Fast KVzip as a general-purpose solution compatible with the messy reality of production LLM serving. It explicitly demonstrates compatibility with quantized models (Qwen3-8B-FP8), hybrid sliding-window attention (Gemma3-12B), and various model scales (7B to 14B, with the 30B-A3B MoE variant also tested). The gate training requires only 1M tokens (a $10^{-6}$ fraction of the FineWeb-Edu corpus) and under one H100 hour for 14B-scale models. This positions the method not as a research curiosity requiring exotic infrastructure, but as something a practitioner could implement in an afternoon.

3. Technical Approach

3.1 Reader Orientation

This paper presents a system for compressing the KV cache of frozen-weight large language models by adding lightweight, learnable gating modules to each attention layer that predict which key-value pairs are important enough to retain, enabling models to discard up to 70% of their KV cache with negligible accuracy loss and no runtime reconstruction overhead. The core insight is that the future utility of a KV pair — whether it will receive high attention from future queries — is an intrinsic property decodable directly from the input hidden states at the moment those KV features are computed, eliminating the need for expensive post-hoc reconstruction.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that work together to compress the KV cache during both prefill and decoding:

1. Base LLM (frozen weights) — any pretrained transformer (e.g., Qwen2.5, Qwen3, Gemma3) whose attention layers produce key-value features during the forward pass. The model weights are never modified; only new gating modules are trained.

2. Gating modules (one per attention layer) — lightweight neural networks that take the current hidden state $\mathbf{h} \in \mathbb{R}^D$ as input and output an importance score $\mathbf{s} \in [0, 1]^H$ for each KV head in that layer. These modules use a novel sink-attention architecture with learnable sink keys that serve as an adaptive baseline for scoring.

3. KV cache with eviction logic — the standard key-value cache, modified so that after each chunk during prefill (or periodically during decoding), the system uses the gate's importance scores to identify and discard low-scoring KV pairs, maintaining only those above a retention threshold that respects a total memory budget.

4. Local window buffer — a fixed-size sliding window of recent tokens (4K tokens during prefill, 128 during decoding) whose KV pairs are always retained regardless of gate scores, exploiting the well-documented recency bias in transformer attention.

5. Training pipeline (pre-deployment, one-time) — a distillation procedure where the computationally expensive KVzip reconstruction process is run once on a small corpus (1M tokens of FineWeb-Edu) to produce target importance scores for each KV pair. The gating modules are then trained via stochastic gradient descent with binary cross-entropy loss to predict these scores from hidden states alone, requiring under one H100 GPU hour for 14B-scale models.

Information flows as follows: a sequence of tokens enters the model → at each attention layer, hidden states are projected to QKV as usual AND fed through the gating module → the gating module produces head-wise importance scores for the current token's KV features → the KV cache stores these features along with their scores → when the cache reaches a capacity threshold (determined by the compression budget), low-scoring KV pairs are evicted, keeping only high-scoring ones plus a local window → the compressed KV cache is used for all subsequent attention computations.

3.3 Roadmap for the Deep Dive

• First, the gating mechanism and its computational flow — how gates are integrated into the forward pass, how scores are produced, and how eviction happens during both prefill and decoding stages. This establishes the operational core of the system.
• Second, the interpretation framework — how the authors recast existing compression methods (DuoAttention, Expected Attention, FastGen) as constrained instances of a generalized gating formulation, providing conceptual grounding for why learnable gates are the right abstraction.
• Third, the gate training procedure — the distillation objective, target score computation via KVzip reconstruction, why reconstruction-based targets generalize better than next-token-prediction or QA-based alternatives, and why hidden states outperform key states as gate inputs. This is the critical design choice section.
• Fourth, the sink-attention architecture — the mathematical formulation of the gating module, the role of learnable sink keys, the low-rank projections, and empirical comparisons against MLP and linear alternatives. This is where the specific architectural innovation lives.
• Fifth, efficiency considerations — training cost (time, storage), inference overhead (the buffered decoding strategy that reduces gating latency to ~1%), and memory/latency improvements relative to both uncompressed baselines and KVzip.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that KV cache compression can be reframed as learning to gate — training lightweight modules to predict, from local hidden states alone, the query-agnostic importance scores that an expensive reconstruction process would compute.

---

Gating Mechanism and Computational Flow

The gating mechanism is the central operational component of Fast KVzip. At each attention layer $l$ in the transformer, the authors introduce a gating function:

$g_l : \mathbb{R}^D \rightarrow [0, 1]^H$

where $D$ is the hidden feature dimension of the model and $H$ is the number of KV heads in that specific layer.

What it computes: for a single token's hidden state vector $\mathbf{h} \in \mathbb{R}^D$ at layer $l$, the gating function $g_l$ outputs a vector of $H$ scalar values, each in the range $[0, 1]$, with one value per KV head. Each scalar represents the predicted importance — from 0 (completely discardable) to 1 (must retain) — of the key-value pair that this token will produce for that specific head in that specific layer. This is a per-token, per-head, per-layer prediction: a token's KV features might be critical for head 3 in layer 12 while being useless for head 1 in the same layer.

Why this form: the $[0, 1]$ range enables straightforward thresholding for binary keep/evict decisions — later, the system can select a percentile threshold (e.g., keep the top 30% of KV pairs by score) or apply a fixed budget. The independence across heads is physically meaningful because different attention heads in transformers capture different relationships (syntactic, semantic, positional, etc.), and a KV pair's utility varies substantially by head. The per-layer design reflects the hierarchical nature of transformer representations: early layers process low-level features where retention patterns differ from later layers that capture high-level semantics.

The gating branch operates as an independent computation that does not modify the original attention outputs. As illustrated in Figure 2a, the forward pass at each attention layer now has two parallel paths from the hidden state: (1) the standard QKV projection that feeds into the attention mechanism, and (2) the gating projection that produces importance scores. The attention computation itself is unchanged — it still operates over whatever KV pairs happen to be in the cache. The gate merely determines which pairs get to stay.

Prefill-stage flow. The paper adopts chunked prefill (Agrawal et al., 2024) to reduce peak memory when processing long contexts, with a chunk size of 16K tokens. For each chunk of 16K tokens, the process is:

1. The model processes the chunk through all layers, computing hidden states, attention outputs, and KV features as normal.
2. Simultaneously, at each layer, the gating module processes the hidden states for every token in the chunk and produces importance scores $\mathbf{s} \in [0, 1]^H$ for the KV features just computed.
3. After the chunk is fully processed, a global eviction step occurs across the entire KV cache (both the new chunk's features and any previously retained features): all KV pairs are ranked by their gating scores, and the lowest-scoring pairs are evicted until the total cache size respects the compression budget (e.g., 30% of the original size).
4. A local window of the 4K most recent tokens is always retained, regardless of their gating scores. For contexts shorter than 16K tokens, the paper retains the last 2% of tokens.
5. The next chunk is processed using the compressed KV cache from step 3, reducing peak memory compared to naive chunked prefill that keeps all KV features for all processed chunks.

The local window is a crucial design element that the paper validates in Figure 18 (Appendix B.3). Retaining recent tokens consistently improves performance, though the exact window size (1K to 8K) matters less — performance is comparable across this range. This aligns with the well-known attention sink phenomenon (Xiao et al., 2024): transformers learn to attend heavily to initial tokens (sinks) and recent tokens, and a local window exploits the latter bias without requiring the gate to learn it.

Decoding-stage flow. During autoregressive generation, the KV cache grows by one token per decoding step. Applying gating at every single step would be impractical because each gating computation, while individually cheap, adds up when called hundreds or thousands of times sequentially. The authors report that naive per-step gating causes a 30–60% latency increase due to frequent function calls and memory I/O.

The solution is a buffered, batched approach:

1. The system maintains a small hidden-state buffer of size 128 tokens that caches the most recent hidden states.
2. During decoding, new hidden states are appended to this buffer, and KV features are stored in the full KV cache as usual.
3. When the buffer fills (every 128 decoding steps), the system performs a parallel gating pass: all 128 buffered hidden states are fed through the gating modules simultaneously, producing updated importance scores for the corresponding KV pairs.
4. Based on these scores, the system evicts an equivalent number of KV entries to maintain the cache budget — for example, if the budget is 4K tokens and the cache has grown to 4128, the 128 lowest-scoring KV pairs outside the local window are discarded.
5. A local window of 128 tokens is always preserved at each eviction step, mirroring the prefill design.
6. The hidden-state buffer is then cleared, and the cycle repeats.

This batching strategy reduces the gating overhead to approximately 1% of the model's forward-pass latency on average, making it negligible in practice while still allowing the gate to update importance scores as new context accumulates.

Non-uniform cache structure. The paper applies a non-uniform KV cache eviction structure following Kim et al. (2025a). This means that different attention heads can retain different numbers of KV pairs based on their varying sparsity patterns — some heads might naturally need only 5% of tokens, others might need 80%. The eviction decisions are made independently per head: for a given budget (e.g., 30% overall retention), the system computes a per-head threshold such that the total number of retained KV pairs across all heads and layers equals the budget, but individual heads may vary substantially in their retention rates. Figure 14 visualizes this: approximately half of the heads (primarily in early layers) are highly sparse with retention rates below 5%, while others in middle-to-late layers show uniformly distributed higher retention rates.

---

Interpretation of Gating Formulation

Before detailing the training procedure, the paper provides a conceptual framing that unifies several prominent compression methods as special cases of gating. This is not merely taxonomic — it establishes that the design space for KV compression is fundamentally about designing gating functions, and that prior methods have imposed constraints (constant functions, quadratic forms, Boolean outputs) that limit their expressiveness.

DuoAttention as constant Boolean gating. DuoAttention (Xiao et al., 2025) classifies each attention head as either a "streaming head" that only attends to a local window or a "retrieval head" that attends to the full context. This can be expressed as:

$g_l(\mathbf{h}) = \mathbf{c} \in \{0, 1\}^H$

where $\mathbf{c}$ is a fixed binary vector — constant across all inputs — with 0 indicating streaming heads (evict all non-local KV pairs) and 1 indicating retrieval heads (retain everything). The gating function is input-independent: the decision is made once based on calibration data and never adapts to specific inputs. This captures the coarse-grained, head-level patterns that DuoAttention identifies but cannot adapt to token-level or context-level variation.

Expected Attention as quadratic gating. Expected Attention (Devoto et al., 2025) models the expected attention a KV pair will receive by approximating future query distributions with mean query statistics. The paper characterizes this as a quadratic gating function mapping key states to scores:

$g_l(\mathbf{k}) = f(\mathbf{k}; \boldsymbol{\mu}_q, \boldsymbol{\Sigma}_q)$

where $\mathbf{k}$ is the key state, $\boldsymbol{\mu}_q$ is the mean query vector, and $\boldsymbol{\Sigma}_q$ captures query variance. This provides input-dependent scores (unlike DuoAttention) but constrains the functional form to a specific quadratic model derived from attention mechanics, which may not capture more complex importance patterns.

FastGen as Boolean token gating. FastGen (Ge et al., 2024) identifies attention heads that consistently attend to specific token types (punctuation, special tokens) and applies pattern-based retention. This can be seen as a gating function on the token identity rather than the hidden state:

$g_l(\text{token}) = \mathbf{b} \in \{0, 1\}^H$

where the score depends only on what token is being processed, not on its contextual representation.

Fast KVzip as unconstrained learnable gating. By contrast, Fast KVzip imposes no functional form constraints:

$g_l(\mathbf{h}) = f_\theta(\mathbf{h}) \in [0, 1]^H$

where $f_\theta$ is any differentiable function with parameters $\theta$. This transforms the problem from "design a heuristic" to "optimize a function via gradient descent," enabling the model to discover compression strategies that no human engineer would design. The paper argues this is what enables the method to work across diverse tasks and models without task-specific tuning.

Relationship to Mixture-of-Depths and Mixture-of-Block-Attention. Table 1 positions Fast KVzip relative to two other gating-based transformer efficiency methods. MoD (Raposo et al., 2024) uses gating to conditionally skip attention computation entirely for certain tokens — both the attention operation and KV storage are conditional. MoBA (Lu et al., 2025) uses gating to select which blocks of the KV cache to attend to, but the full KV cache is always stored. Fast KVzip occupies a middle ground: attention is always computed (every token attends to the compressed cache), but KV storage is conditional (low-importance pairs are never stored or are evicted). This distinction matters because it means Fast KVzip preserves the full attention mechanism's ability to attend to any retained token while reducing memory pressure through selective storage.

---

Gate Training Procedure

The training procedure is where the paper makes its most critical design decisions. The goal is to train the gating modules to predict KV importance scores without requiring backpropagation through the full LLM, without degrading the base model's capabilities, and without overfitting to any specific downstream task.

Training objective as distillation from KVzip. Rather than training end-to-end (which would require backpropagation through the full transformer, limiting context length and increasing memory), the paper adopts a distillation approach. The "teacher" is the KVzip reconstruction process (Kim et al., 2025a), which produces target importance scores by explicitly measuring which KV pairs receive attention during context reconstruction. The "student" is the gating module, trained to predict these scores from hidden states alone.

The KVzip reconstruction process works as follows (used only during training, never at inference):

1. A long context is split into sentences.
2. Each sentence is fed back through the frozen LLM as if it were a query.
3. The LLM computes attention over the full KV cache of the complete context.
4. For each KV pair, the maximum attention score it receives across all sentence queries is recorded.
5. These maximum attention scores serve as query-agnostic importance labels: a KV pair that receives high attention from even one sentence during reconstruction is deemed important.

This process is computationally expensive — it essentially processes the context twice — but it captures a task-agnostic notion of KV utility that the paper argues is fundamental. The insight is that reconstruction simulates a distribution of possible queries without needing actual downstream task data, producing scores that generalize because many downstream tasks rely on similar high-level contextualized features.

Training data construction. The authors use the FineWeb-Edu pretraining corpus (Penedo et al., 2024), which has no overlap with any downstream evaluation datasets. The data construction pipeline is:

1. Randomly subsample sequences with context lengths ranging from 10K to 30K tokens, totaling 500K tokens.
2. Concatenate these to construct additional long-context sequences of length 100K tokens, totaling another 500K tokens.
3. Total training data: 1M tokens — approximately a $10^{-6}$ fraction of the full FineWeb-Edu corpus.
4. For each sequence, run the full KVzip reconstruction process to obtain target importance scores for every KV pair at every layer.
5. Store tuples of (hidden state, target score) for each token at each layer, enabling embarrassingly parallel training across samples and layers.

Figure 16 (Appendix B.1) shows that increasing training data size improves performance, particularly on retrieval-intensive tasks (SCBench.KV), and that including the long-context (100K) samples provides additional benefits. However, even the 500K-short dataset (no 100K sequences) shows substantial improvements over baselines, suggesting the method is not critically dependent on massive training data.

Loss function and optimization. The gating modules are trained independently per layer using stochastic gradient descent with binary cross-entropy loss:

$\mathcal{L} = -\left(y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})\right)$

where $y \in [0, 1]$ is the target importance score from KVzip reconstruction (a soft label between 0 and 1 representing the normalized maximum attention the KV pair received) and $\hat{y} \in [0, 1]$ is the gating module's predicted score for that KV pair.

What it computes: for each KV pair at each head in each layer, the loss measures the discrepancy between the predicted importance and the KVzip-derived target. The first term $-y\log\hat{y}$ penalizes the gate when it predicts low importance for a KV pair that KVzip found to be important; the second term $-(1-y)\log(1-\hat{y})$ penalizes the gate when it predicts high importance for a pair KVzip found to be unimportant. The loss is averaged across all tokens, heads, and layers.

Why this form: binary cross-entropy is the appropriate objective for this distillation task because the target scores are themselves probabilities (normalized attention scores), not hard binary labels. It provides well-calibrated gradients — when the prediction is far from the target, gradients are large; when close, gradients are small — and naturally handles the $[0, 1]$ range without requiring clipping or rescaling. The paper reports using a learning rate of 0.2 (unusually high for neural network training, suggesting the gates are well-conditioned), training for 5K update steps with a batch size of 1K, corresponding to 5 epochs over the 1M-token dataset.

Figure 15 (Appendix A.2) shows the training and validation loss trajectories. A notable pattern emerges: earlier layers achieve lower loss values, while middle and later layers show higher loss. The authors hypothesize that "higher layers perform more complex attention mechanisms, which are inherently more difficult to predict from a single hidden state." This is an important observation: it means the gating modules are less accurate at predicting importance for deeper layers, but the compression budget applies uniformly, so the gates must work harder at layers where prediction is inherently harder.

Why reconstruction-based targets over alternatives. This is perhaps the most consequential design choice in the paper. The authors compare three types of target signals (Figure 5):

1. Reconstruction-based (KVzip): maximum attention scores during context self-reconstruction.
2. Next-token prediction: maximum attention scores during standard autoregressive next-token prediction (similar to TrimKV's approach; Bui et al., 2025).
3. Instruction-based QA: maximum attention scores when the model answers questions about the context (similar to Locret's approach; Huang et al., 2024), using the LongAlpaca dataset.

Figure 5 demonstrates that reconstruction-based targets substantially outperform the alternatives on both a synthetic key-retrieval task (SCBench.KV) and a QA task (SQuAD). The performance gap is largest on retrieval, where next-token prediction and QA targets degrade significantly.

Figure 6 provides a mechanistic explanation by visualizing the attention patterns underlying each target type:

• Reconstruction attention is "uniform and structured across KV sequences" — it produces smooth, distributed attention patterns that capture the broad contextual structure of the text.
• Next-token prediction attention is "denser, reflecting more intensive feature contextualization" — it focuses sharply on the tokens immediately relevant to predicting the next word, producing spikier patterns that may miss long-range dependencies.
• QA attention is "sparse as they focus only on query-specific information" — it attends only to tokens relevant to the specific question being asked, discarding most of the context.

The paper hypothesizes that reconstruction is effective because "many downstream applications rely primarily on these high-level contextualized features" — the uniform attention patterns capture the kind of semantic and structural information that is useful across many tasks, while next-token or QA patterns encode task-specific or query-specific utility that fails to transfer. This is a crucial insight: the choice of training target determines what "importance" means, and reconstruction defines importance in a way that is robustly useful across tasks by simulating a distribution of possible attentional needs.

Why hidden states over key states as gate inputs. Figure 7 compares three choices for what representation to feed into the gating module:

1. Hidden states $\mathbf{h} \in \mathbb{R}^D$ — the transformer's intermediate representation after the previous layer's processing, before any attention-specific projections.
2. Key states $\mathbf{k} \in \mathbb{R}^{H \times D_{\text{head}}}$ — the projected key features, post-RoPE encoding, which are what the attention mechanism actually uses.
3. Pre-RoPE key states — the projected key features before rotary position embedding is applied.

Hidden states consistently outperform both key-state variants. The paper suggests two reasons. First, hidden states contain richer information — they are the model's full intermediate representation, encoding not just what the token is but how it relates to the surrounding context, which may be more predictive of future utility than the key projection alone. Second, incorporating position-encoded features (which key states have via RoPE) actually degrades performance, motivating the design choice to "decouple positional information from the gating mechanism and instead leverage recency through a simple local-window strategy." This is a subtle but important architectural insight: the gate should focus on content-based importance (what the token means), leaving position-based importance (how recent the token is) to the explicit local window mechanism that requires no learning.

The paper also includes a negative result on training data construction that the prior example summary omitted: the authors explicitly do NOT train gates on task-specific data, avoiding the domain specialization that plagues TrimKV (which trains separate gates for math vs. general tasks) and Locret (which trains on instruction-QA). A single set of gates trained on the general FineWeb-Edu corpus is used for all evaluations across all tasks and all models. This task-agnostic training is what enables the demonstrated generalization across prefill-intensive retrieval, contextual QA, code comprehension, and decoding-intensive mathematical reasoning.

---

Sink-Attention Gate Architecture

The architectural design of the gating module is where the paper contributes a novel neural network component specifically designed for KV importance prediction. The authors systematically explore the design space and arrive at a sink-attention mechanism inspired by the attention sink phenomenon.

Low-rank projection. Given a hidden state $\mathbf{h} \in \mathbb{R}^D$ at a particular layer, the gate first applies linear projections to obtain low-dimensional query and key representations:

$\mathbf{k} \in \mathbb{R}^{H \times D'}, \quad \mathbf{q} \in \mathbb{R}^{G \times H \times D'}$

where $H$ is the number of KV heads in the layer, $G$ is the grouped-query size (for models using grouped-query attention; Ainslie et al., 2023), and $D'$ is the low-rank projection dimension, set to 16 for all models across all experiments.

What it computes: the hidden state is linearly transformed into a per-head key representation $\mathbf{k}$ and a per-group, per-head query representation $\mathbf{q}$. For a model with 4 KV heads and a grouped-query ratio of 4 (meaning 16 query heads mapped to 4 KV heads), $G = 4$ and $H = 4$, so $\mathbf{q}$ has dimensions $4 \times 4 \times 16 = 256$ scalars per token. These projections are low-rank relative to the original hidden dimension (e.g., for Qwen3 with $D_{\text{head}} = 128$, the projected dimension of 16 is 1/8 of the head dimension).

Why low-rank: the gate must be computationally lightweight — it runs at every layer during both prefill and decoding. Full-dimensional projections would add significant FLOPs and memory. The low-rank design keeps the gate's parameter count small (Table 2 shows 0.18 GB for Qwen3-8B, 0.30 GB for Qwen3-14B) while retaining enough capacity to capture head-specific importance patterns. The sensitivity analysis in Figure 17 shows that performance is "robust to the choice of projection dimension" — larger $D'$ does not meaningfully improve results — suggesting that 16 dimensions already capture the relevant signal.

Weighted normalization. The projections are followed by weighted normalization (Yang et al., 2025a), which the paper mentions but does not elaborate on. This is a normalization technique introduced in Qwen3 that stabilizes training and provides learnable scaling, important here because the gate is trained with a high learning rate (0.2) and needs well-conditioned gradients.

Sink-attention scoring. The core of the architecture is the attention mechanism with learnable sink keys:

$\mathbf{s} = \frac{1}{G} \sum_{j=1}^G \frac{\exp(\mathbf{q}_j^\top \mathbf{k})}{\exp(\mathbf{q}_j^\top \mathbf{k}) + \underbrace{\sum_{r=1}^S \exp(\mathbf{q}_j^\top \mathbf{k}_{\text{sink}}^r) + b_j}_{\text{Learnable sinks}}}$

where:

• $\mathbf{s} \in [0, 1]^H$ is the output importance score vector, one scalar per KV head.
• $\mathbf{q}_j \in \mathbb{R}^{H \times D'}$ is the query for the $j$-th group.
• $\mathbf{k} \in \mathbb{R}^{H \times D'}$ is the key representation for the current token.
• $\mathbf{k}_{\text{sink}}^r \in \mathbb{R}^{H \times D'}$ for $r = 1, \ldots, S$ are learnable sink key vectors — layer-specific parameters that are trained jointly with the projection weights. The paper uses $S = 16$ sink keys across all experiments.
• $b_j \geq 0$ is a learnable scalar bias controlling head-level importance for the $j$-th group.
• The summation over $j = 1, \ldots, G$ averages across the grouped-query groups, producing per-head scores.

What it computes — operational explanation:

1. For each group $j$ and each head $h$, the gate computes a query-key similarity: $\mathbf{q}_{j,h}^\top \mathbf{k}_h$ — how well the token's query representation matches its own key representation for that head. This is a self-similarity score.

2. It also computes similarities to all $S$ learnable sink keys: $\mathbf{q}_{j,h}^\top \mathbf{k}_{\text{sink}, h}^r$ for each sink $r$. These sink keys represent "baseline" importance reference points learned during training.

3. The softmax-style ratio $\exp(\mathbf{q}^\top \mathbf{k}) / (\exp(\mathbf{q}^\top \mathbf{k}) + \sum \exp(\mathbf{q}^\top \mathbf{k}_{\text{sink}}) + b)$ maps these similarities to a probability in $[0, 1]$. A high self-similarity relative to the sink similarities produces a high score (the KV pair is important); a low self-similarity relative to the sinks produces a low score (the KV pair is discardable).

4. The scores are averaged across groups $j$ to produce a single per-head score, since groups share KV heads.

Why sink attention — the conceptual motivation:

Standard attention mechanisms exhibit "attention sinks" (Xiao et al., 2024): certain tokens (often the first token or delimiter tokens) receive disproportionately high attention regardless of their semantic relevance, acting as "sinks" that absorb attention mass. The authors observe that an analogous phenomenon may apply to KV importance: there is a baseline level of importance that certain tokens have simply by virtue of being tokens in a sequence, and the gate needs to measure importance above this baseline.

The learnable sink keys $\mathbf{k}_{\text{sink}}^r$ provide this adaptive baseline. Instead of using a fixed threshold (which would require manual tuning and might not generalize across layers or models), the gate learns what "baseline unimportant" looks like for each head in each layer. The self-similarity $\exp(\mathbf{q}^\top \mathbf{k})$ is compared against the aggregate sink similarity $\sum \exp(\mathbf{q}^\top \mathbf{k}_{\text{sink}}) + b$. When the token's representation is similar to the learned sink keys (suggesting it's a typical, unremarkable token for that head), the score is low. When it is dissimilar to the sinks (suggesting it carries distinctive information), the score is high.

The scalar bias $b_j \geq 0$ provides an additional head-level adjustment: heads that generally retain more tokens will learn lower biases (making it harder for tokens to achieve high scores), while heads that are very selective will learn higher biases. The non-negativity constraint ensures the denominator is always at least $b_j$, maintaining numerical stability.

Why this architecture over alternatives. Figure 9 compares four architectures, all matched for parameter count:

• Linear model: a single linear projection from hidden state to $H$ scores, then sigmoid activation. This is the simplest possible gate — it can only capture linear relationships between hidden states and importance.
• Two-layer MLP with SwiGLU: a standard feedforward network with SwiGLU activation (Shazeer, 2020), providing non-linear capacity.
• Sink-attention without learnable denominator: a variant where $\sum_{r=1}^S \exp(\mathbf{q}_j^\top \mathbf{k}_{\text{sink}}^r) + b_j = 1$ is fixed, making the scoring function a simple sigmoid of the self-similarity.
• Full sink-attention: the proposed architecture.

Results show that while the MLP and simplified sink-attention are competitive on contextual QA tasks, they degrade substantially on retrieval tasks. The linear model performs worst across the board. The full sink-attention uniquely maintains performance on both task types. This suggests that retrieval-heavy tasks require the gate to make fine-grained discrimination between tokens (which the adaptive sink baseline enables) while simpler tasks are more forgiving of approximate importance estimates.

The paper highlights a specific distinction from DeepSeek-V3.2's Lightning Indexer (Liu et al., 2025), which also uses low-rank KV features for sparse attention but introduces additional KV caching for those low-rank features. Fast KVzip's sink keys are fixed per-layer parameters, not cached per-token, avoiding any additional memory footprint.

Hyperparameter sensitivity. Figure 17 (Appendix B.2) analyzes the effects of the number of sink keys $S$ and projection dimension $D'$. The key findings:

• A small number of sink keys ($S = 4$) underperforms, while performance stabilizes around $S = 16$ and shows minimal improvement beyond that. This suggests 16 sink keys are sufficient to model the baseline importance distribution for each head.
• Projection dimension $D'$ shows very flat sensitivity — values from 8 to 64 produce comparable results, with 16 being the chosen default. This robustness implies the gate's capacity bottleneck is not in the projection dimensionality but in the attention-based scoring mechanism itself.

The paper uses identical hyperparameters across all models (Qwen2.5, Qwen3, Gemma3, different scales), setting $S = 16$ and $D' = 16$ universally without model-specific tuning. This is a practical strength: deployers do not need to perform hyperparameter searches for each new model.

---

Efficiency Considerations

Training efficiency. Table 2 quantifies the gate training cost across model scales on a single H100 GPU:

• Qwen3-8B: 0.59 H100 hours, 0.18 GB storage for gate parameters.
• Qwen3-30B-A3B: 0.70 H100 hours, 0.11 GB storage (smaller hidden state in this MoE model reduces gate size).
• Qwen3-14B: 0.83 H100 hours, 0.30 GB storage.

These are remarkably low numbers. Training a complete KV cache compression system for a 14B-parameter model takes less than one GPU-hour and produces less than 300 MB of additional parameters — a tiny fraction of the model's own weight file. The training is embarrassingly parallel across samples and layers because the target scores are precomputed and the gates are trained independently per layer, enabling efficient GPU utilization.

Inference overhead. The paper reports that the buffered decoding strategy reduces gating overhead to approximately 1% of the model's forward-pass latency on average. This is achieved by amortizing the gating computation across 128-token batches rather than computing it at every step. The prefill overhead is similarly minimal because the gating shares the same hidden states the attention mechanism already computes — the additional projections and sink-attention scoring are small compared to the main attention computation, especially with FlashAttention-2's optimized kernels.

Memory and latency improvements. Figure 10 shows the practical benefits on Qwen2.5-7B-1M with a 30% KV budget ratio:

• Prefill (Figure 10a): Fast KVzip reduces both memory usage and prefill time compared to the no-compression baseline. At 320K context length, the memory reduction is substantial (from approximately 80 GB to approximately 45 GB). Compared to KVzip, Fast KVzip's prefill time is much lower because it eliminates the reconstruction pass — KVzip's prefill time at 320K is approximately double the baseline, while Fast KVzip's is below the baseline.
• Decoding (Figure 10b): Fast KVzip reduces decoding latency and memory compared to the baseline. KVzip provides similar decoding speed (its reconstruction happens during prefill only, so decoding is unaffected), but Fast KVzip achieves this without paying the prefill-time cost.

The memory reduction during prefill is particularly important because it reduces peak memory usage — the maximum GPU memory required at any point during processing. In chunked prefill without compression, the KV cache grows with each chunk, and peak memory occurs when the final chunk is being processed with all previous KV features in memory. Fast KVzip evicts within each chunk, so the KV cache size plateaus rather than growing linearly, substantially reducing the peak.

The paper uses a non-uniform KV cache eviction structure following Kim et al. (2025a) for all experiments. This means the compression is applied independently per head, allowing heads with naturally sparse attention patterns to retain far fewer KV pairs than heads with dense patterns. This is more efficient than uniform eviction (which would force all heads to retain the same fraction) because it allocates the memory budget where it provides the most benefit.

Integration with existing model features. The paper explicitly tests compatibility with:

• Quantized weights: Qwen3-8B-FP8 uses dynamic FP8 quantization, and Fast KVzip works without modification (Figure 12).
• Hybrid sliding-window attention: Gemma3-12B uses a mix of global and sliding-window attention heads, and Fast KVzip is applied only to the global attention KV cache (which dominates memory in long-context scenarios), demonstrating that the method composes with architectural variants.
• Grouped-query attention: The gating formulation naturally handles GQA by averaging across query groups (the $1/G$ summation), meaning the same gate architecture works whether the model uses multi-head, grouped-query, or multi-query attention.

4. Key Insights and Innovations

Innovation 1: Reformulating KV Cache Compression as Gate Function Optimization

The paper's most distinctive conceptual contribution is not any particular gating architecture but rather the reinterpretation of diverse KV cache compression methods as constrained instances of a unified gating framework. Prior work treated compression heuristics (H₂O, SnapKV), attention-pattern analysis (DuoAttention, FastGen), and query-distribution modeling (Expected Attention) as fundamentally different approaches — one was about tracking cumulative attention, another about head classification, another about statistical approximation. The paper cuts through this apparent diversity by showing they differ only in the functional form and constraints imposed on a common gating function $g_l$ that maps representations to importance scores (Figure 3).

This reframing is intellectually significant because it transforms the field's problem statement. Before this work, KV cache compression was largely a heuristic design problem: engineers observed attention patterns, identified regularities, and hand-crafted rules to exploit them. After this reframing, it becomes a function approximation problem: design a sufficiently expressive, differentiable $g_l$, provide it with training signal that captures a task-agnostic notion of KV utility, and let optimization discover the compression strategy. The shift is analogous to the transition from hand-crafted computer vision features to learned convolutional filters — it opens the door to methods that a human designer would never invent.

The taxonomy in Figure 3 and Table 1 is not merely descriptive; it is diagnostically productive. By identifying that prior methods impose specific constraints — DuoAttention uses constant Boolean functions, Expected Attention uses quadratic forms, FastGen uses token-identity gating — the paper explains why these methods fail in particular regimes. A constant function cannot adapt to input-dependent variation in KV utility. A quadratic form in key space cannot capture complex importance patterns encoded in richer hidden state representations. Token-identity gating cannot account for contextual variation in a token's importance. Each constraint limits expressiveness in a way that causes specific failure modes (e.g., DuoAttention's inability to do fine-grained token-level retention, Expected Attention's reliance on key-only information). Fast KVzip's unconstrained $g_\theta(\mathbf{h})$ is not an arbitrary architectural choice — it is the logical consequence of identifying and removing these constraints.

This framing also unifies the design space for future work. Rather than proposing yet another heuristic, subsequent methods can be understood and compared by their choice of (1) gating function class, (2) training signal, and (3) input representation. The paper's extensive ablation of these three dimensions (architecture in Figure 9, target signal in Figure 5, input in Figure 7) provides a template for systematic exploration that the field previously lacked.

The comparison to Mixture-of-Depths and Mixture-of-Block-Attention in Table 1 further contextualizes Fast KVzip within a broader family of gating-based transformer efficiency methods. The distinction — always compute attention, conditionally store KV features — identifies a specific point in the design space that was previously unexplored. MoD conditionally skips both attention and storage; MoBA always stores the full KV cache and uses gating for sparse access. Fast KVzip's "always attend, conditionally store" represents a third regime with distinct memory-compute tradeoffs that neither prior approach captures.

This is a fundamental conceptual advance rather than an incremental refinement. It does not directly improve any metric but provides the intellectual scaffolding that makes the subsequent architectural and training innovations possible. The field can now discuss KV compression in terms of gating expressiveness, training signal quality, and generalization, rather than in terms of attention-pattern heuristics and recency biases. This is a rare case where a paper's most important contribution is a way of thinking rather than a specific mechanism.

---

Innovation 2: Reconstruction-Based Distillation as Task-Agnostic Importance Ground Truth

The paper's second major insight is that the target signal used to train KV importance predictors determines not just the quality but the generality of the resulting compression, and that context reconstruction provides a uniquely task-agnostic notion of KV utility. This may sound like a training detail, but it is the linchpin that enables Fast KVzip to work across retrieval, QA, code comprehension, and mathematical reasoning without task-specific tuning — a capability that prior learned compression methods (TrimKV, Locret, DMS) conspicuously lack.

The evidence for this claim's significance is the negative result component of Figure 5: gates trained on next-token prediction attention patterns or instruction-QA attention patterns degrade substantially on retrieval tasks, despite using identical architecture and training data. This is not a small performance difference — it is a qualitative failure mode where the gate systematically discards information needed for retrieval because its training signal never taught it that such information matters. TrimKV (Bui et al., 2025) experiences exactly this failure, requiring separate models for mathematical reasoning versus general language tasks (Table 4). The paper's reconstruction-based target avoids this specialization entirely.

The intellectual contribution is the identification of why reconstruction generalizes. Figure 6 provides the diagnostic: reconstruction attention patterns are "uniform and structured" — they distribute attention broadly across the context, capturing the high-level semantic and structural features that many downstream tasks require. Next-token prediction creates denser, more localized patterns focused on immediate prediction needs. QA creates sparse patterns focused on query-specific information. These visualizations are not merely descriptive; they explain a mechanistic failure mode. If your training signal says "tokens 47–52 are critical because they contain the answer to this specific question," your gate will learn to retain answer-bearing tokens and discard everything else. When deployed on a retrieval task where the "answer" is a specific key hidden in arbitrary positions, the gate will evict it because it never learned that arbitrary tokens can be important.

The paper's hypothesis that reconstruction simulates a distribution of possible queries — by feeding each sentence back as if it were a query — is a theoretical contribution to understanding what makes a compression signal task-agnostic. It is not that reconstruction is a magic source of importance information; it is that reconstruction marginalizes over a broad query distribution implicitly, without needing actual task data. This connects to the idea of amortized inference: the reconstruction process computes the maximum attention a KV pair would receive under any query in the "distribution of sentences from this document," and that maximum is a robust statistic for downstream tasks whose queries are drawn from a related distribution.

The choice to use this expensive reconstruction signal for distillation rather than inference is itself an innovation. KVzip demonstrated that reconstruction-based scores produce excellent compression quality, but at prohibitive runtime cost. Fast KVzip's key move is recognizing that this cost only needs to be paid once, during training, and that the resulting importance scores can serve as supervised targets for a lightweight predictor. This is a classic distillation pattern, but applied to a novel domain: distilling an algorithmic process (reconstruction) rather than a model's output distribution. The teacher is not a larger network but a computational procedure, and the student learns to predict the procedure's output from inputs that the procedure itself uses intermediate representations of. This is a clever abuse of the fact that the hidden states — which the procedure processes — contain information about the procedure's eventual output.

The practical consequence is that this is not an incremental training recipe refinement. It is the difference between a method that works on one benchmark and a method that works across twelve benchmarks spanning four task categories without modification. Prior learned compression methods (TrimKV, Locret) achieved their results by training on task-specific data, making their reported performance contingent on task-distribution match at deployment. Fast KVzip's reconstruction-based training eliminates this contingency, making the method genuinely deployment-ready in a way that task-specific approaches are not.

---

Innovation 3: The Sink-Attention Mechanism as an Adaptive Importance Baseline

While the gating framework and reconstruction-based training are conceptual innovations, the sink-attention architecture is the paper's primary architectural contribution — and it is more subtle than simply "attention works better than MLPs." The innovation is the recognition that predicting KV importance requires not just scoring how "important" a token is, but measuring that importance relative to an adaptive, learned baseline, and that the attention sink phenomenon provides a natural mechanism for implementing this baseline.

The critical design element is the learnable denominator in the scoring function:

$s = \frac{\exp(\mathbf{q}^\top \mathbf{k})}{\exp(\mathbf{q}^\top \mathbf{k}) + \sum_{r=1}^S \exp(\mathbf{q}^\top \mathbf{k}_{\text{sink}}^r) + b}$

This is not a standard attention mechanism — the "keys" being attended to are fixed per-layer parameters, not input-dependent features — nor is it a standard feedforward network. It is a hybrid where the token's representation is scored against itself (via $\mathbf{q}^\top \mathbf{k}$) to produce a raw importance signal, then normalized by comparison to learned reference points ($\mathbf{k}_{\text{sink}}^r$) that encode what "baseline unimportance" looks like for each head.

The architectural ablation in Figure 9 demonstrates why this matters. A linear model fails because it cannot capture the non-linear, context-dependent relationship between hidden states and importance. A two-layer MLP with matched parameter count improves but still underperforms on retrieval — suggesting that generic non-linearity is insufficient and that the specific comparative structure of the scoring matters. The simplified sink-attention variant without a learnable denominator — effectively $s = \sigma(\mathbf{q}^\top \mathbf{k})$ — is competitive on QA but degrades on retrieval, supporting the claim that the adaptive baseline is specifically important for tasks requiring fine-grained token discrimination.

The connection to attention sink theory (Xiao et al., 2024) is more than terminological. The original attention sink phenomenon describes how certain tokens absorb disproportionate attention mass, acting as a baseline that stabilizes attention distributions. The paper inverts this idea: instead of sinks as recipients of attention (tokens that receive high scores), they become reference points that define the baseline against which importance is measured. A token with high $\exp(\mathbf{q}^\top \mathbf{k})$ relative to $\sum \exp(\mathbf{q}^\top \mathbf{k}_{\text{sink}})$ is one that stands out from the baseline; a token where these terms are comparable is unremarkable. This inversion is conceptually elegant because it repurposes a known transformer property for a new computational purpose.

The architecture also makes an interesting representational choice that the paper does not fully theorize but that has practical consequences: the query and key are both derived from the same hidden state. This means the gate computes a form of self-predictive importance — how much does this token's representation "stand out" relative to learned prototypes? This is distinct from approaches that use cross-attention between the current token and the full context (which would require access to other tokens' representations and violate the "local computation" requirement). The self-similarity formulation means the gate's computation is O(1) per token — it does not depend on context length — while still capturing a notion of importance that correlates with downstream attention utility.

The robustness to hyperparameters (Figure 17) — $S = 16$ works across all models, $D'$ from 8 to 64 performs comparably — is itself an insight. It suggests the architecture has found a "natural" representational capacity for this task, where additional parameters provide diminishing returns because the core challenge is not representational capacity but rather the choice of what to represent (comparative scoring against learned baselines). This contrasts with MLP-based gates where architectural hyperparameters (width, depth, activation) strongly affect performance and require per-model tuning.

This is a fundamental architectural contribution — not a minor variation on existing gating mechanisms. The sink-attention design is novel as a neural network component (it is not standard attention, not an MLP, not a linear probe), and its specific properties (adaptive baseline, self-similarity scoring, per-head specialization via multiple sink keys) are tailored to the KV importance prediction problem in ways that generic architectures are not.

---

Innovation 4: The Negative Result That Positional Encoding Degrades Gating Performance

One of the paper's most counterintuitive findings — and one that has significant implications for how gating mechanisms should be designed — is that incorporating positional information into the gating input degrades performance rather than improving it. Figure 7 shows that using key states with rotary position embedding (RoPE) as gate inputs underperforms using raw hidden states without explicit position encoding, and even pre-RoPE key states (which contain the key projection but not the position signal) underperform hidden states.

This is surprising because position is clearly important for KV utility — recent tokens are attended to more heavily, and the local window mechanism exists precisely because recency matters. One might therefore expect that providing positional information to the gate would help it predict which tokens will be important. The paper finds the opposite.

The interpretation the paper offers — that positional information should be "decoupled from the gating mechanism and instead leverage recency through a simple local-window strategy" — is a design principle with broader applicability. It suggests that for content-based importance prediction (which tokens carry semantically important information), positional signals are actually a confound: they cause the gate to upweight recent tokens regardless of content, which interferes with learning which tokens are substantively important. The explicit local window then handles the position-based retention separately, as a non-learned rule.

This finding is significant because it contradicts the natural design intuition that "more information is better" for a learned predictor. It demonstrates that the gate benefits from representational specialization: give it only the information relevant to content-based importance (hidden states), and handle position-based importance through a separate, non-learned mechanism. This is a form of inductive bias — the architecture is structured so that the gate doesn't have to learn to ignore position, because position isn't in its input.

The broader implication is that gating mechanisms for KV compression should not try to learn everything. Position is well-understood (recency bias is robust and predictable), so it should be handled by a fixed rule. Content importance is complex and context-dependent, so it should be learned. Mixing the two forces the learned component to disentangle them, which Figure 7 shows it does imperfectly. This principle — identify what is predictable by simple rules and handle it outside the learned system — could inform future gating designs and may explain why some prior learned approaches underperform: they ask the gate to simultaneously learn content importance and position-based retention, and the latter interferes with the former.

This is a diagnostic insight rather than a performance gain per se, but it is the kind of finding that prevents future researchers from pursuing dead ends. The negative result is actionable: if you're building a gating mechanism, do not feed positional information into it; handle position separately. This is a small but intellectually crisp contribution that the paper's systematic ablation methodology surfaces cleanly.

---

Innovation 5: The Buffered Decoding Strategy as a Practical Latency Solution

While the sink-attention architecture and reconstruction-based training are the paper's conceptual contributions, the buffered decoding strategy — maintaining a hidden-state buffer of 128 tokens and performing gating in parallel rather than at every step — is the innovation that makes the method practically deployable. The paper reports that naive per-step gating during decoding causes a 30–60% latency increase, which would make the method unusable in latency-sensitive applications regardless of its compression quality. The buffered approach reduces this to approximately 1%.

This is not a deep theoretical contribution, but it addresses the primary failure mode of prior learned compression methods: they worked in benchmarks but were too slow to deploy. KVzip itself is the canonical example — its compression quality is excellent, but its runtime reconstruction cost makes it impractical for many production scenarios. Fast KVzip's buffered decoding ensures that the method's practical performance (wall-clock latency, throughput) matches its benchmark performance (accuracy at a given compression ratio).

The specific design choice — a buffer size of 128 tokens — reflects an empirical tradeoff. Larger buffers reduce amortized overhead further (fewer gating passes per token) but delay the gate's ability to react to new information. Smaller buffers provide more responsive importance updates but increase overhead. The paper does not extensively ablate this choice (it is mentioned as a fixed parameter), suggesting that 128 was found to be a robust default, but the principle — batch gating computations to amortize overhead — is general.

The significance of this contribution is that it closes the gap between algorithmic quality and system efficiency. Many KV compression papers report accuracy at a given retention ratio without measuring whether the compression overhead cancels out the memory savings in terms of actual wall-clock performance. Fast KVzip's Figure 10 demonstrates that the method improves both accuracy-per-byte and latency-per-token relative to uncompressed baselines and prior methods. The buffered decoding strategy is what makes this possible during autoregressive generation, where per-step overhead is most damaging.

This is an incremental systems contribution rather than a fundamental advance, but it is essential for the paper's claim of practical deployability. Without it, Fast KVzip would be another method that works well on paper but adds unacceptable latency in production — the same criticism it levels at KVzip.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation spans two broad categories: prefill-intensive tasks and decoding-intensive tasks. For prefill-intensive evaluation, the paper uses SCBench (Li et al., 2025), a multi-query benchmark comprising nine tasks including synthetic key retrieval from RULER (Hsieh et al., 2024), long-context QA from LongBench (Bai et al., 2024), and code comprehension tasks. Additionally, the authors include OpenAI MRCR (OpenAI, 2025), a multi-round conversational retrieval benchmark, and SQuAD (Rajpurkar et al., 2016), a multi-query QA dataset. Maximum context lengths reach up to 170K tokens using the Qwen3 tokenizer. For decoding-intensive tasks, evaluation uses AIME24 (AIME, 2025) and MATH (Hendrycks et al., 2021), with maximum decoding lengths of 32K and 16K tokens respectively, following the R-KV baseline setting (Cai et al., 2025).

• Base model(s). The paper evaluates on a diverse set of state-of-the-art open-source LLMs: Qwen2.5-7B-1M and Qwen2.5-14B-1M (Yang et al., 2025b), Qwen3-8B and Qwen3-14B, Qwen3-8B-FP8 (Yang et al., 2025a), and Gemma3-12B (Team et al., 2025). Each model is evaluated using its native precision. Gemma3-12B employs a hybrid sliding-window attention mechanism where Fast KVzip is applied only to the global attention KV cache, which dominates memory usage in long-context scenarios. Qwen3-8B-FP8 adopts dynamic FP8 quantization, testing compatibility with compressed-weight deployments. The Qwen3-30B-A3B MoE variant is also used for training efficiency measurements (Table 2) but is not a primary evaluation target. These models are chosen to span different scales (7B to 14B), different pretraining/post-training regimes (Qwen2.5 vs. Qwen3), different attention architectures (standard vs. hybrid sliding-window), and different precision formats (BF16 vs. FP8), demonstrating that the method is not tied to any single model family.

• Metrics. For prefill-intensive tasks, performance is measured as task-specific accuracy scores on each benchmark dataset (e.g., retrieval accuracy for SCBench.KV, F1 for SQuAD, exact match or accuracy for LongBench subsets). For Figures 11 and 12, results are reported as raw accuracy at varying KV cache budget ratios. For the model-averaged results in Figure 12, scores are normalized relative to full-cache performance prior to averaging — that is, each dataset's score is divided by the score achieved with no KV compression (budget ratio = 1.0), yielding a percentage of full-cache accuracy. For decoding-intensive tasks (Figure 13), AIME24 results are reported as the average score over 16 random seeds, while MATH uses standard accuracy metrics. The KV cache budget ratio is expressed as a fraction of the original uncompressed cache size, with 0.3 meaning 70% of KV pairs are evicted.

• Baselines. The paper compares against a comprehensive set of state-of-the-art KV cache compression methods:

  • KVzip (Kim et al., 2025a): the reconstruction-based query-agnostic method that serves as both a baseline and the teacher for Fast KVzip's distillation training. Included in prefill-intensive evaluations.
  • SnapKV (Li et al., 2024): a heuristic method that analyzes attention patterns from the prompt's final tokens to identify important KV pairs. Included in prefill-intensive evaluations.
  • Expected Attention (Devoto et al., 2025) with Ada-KV (Feng et al., 2024): a query-agnostic method using quadratic key-state models, combined with adaptive budget allocation. Included in prefill-intensive evaluations.
  • DuoAttention (Xiao et al., 2025) with KVzip score (Kim et al., 2025a): a head-classification method that designates retrieval vs. streaming heads, using KVzip scores for the importance signal. Included in prefill-intensive evaluations.
  • R-KV (Cai et al., 2025): a redundancy-aware compression method designed for decoding-intensive scenarios that continuously compresses the KV cache during generation, covering both context and generated tokens. Included in decoding-intensive evaluations.
  • TrimKV (Bui et al., 2025): a gating-based method trained on next-token prediction targets. Included in both prefill-intensive and decoding-intensive evaluations where applicable. The paper notes that TrimKV trains specialized models for different domains (mathematics vs. general language), and where official general-task models are available (Qwen3-4B-Instruct-2507), they are used for comparison (Figure 19). For decoding-intensive tasks, the paper additionally evaluates the early stopping of thinking strategy introduced in Qwen3 (Yang et al., 2025a), where the model terminates the thinking process upon reaching a predefined token budget and is forced to generate the final answer. Some baselines are specifically designed for either prefill-intensive or decoding-intensive scenarios, and comparisons are reported only within their applicable settings.

• Generation budget / compute accounting. Compute for compression methods is measured in terms of KV cache budget ratio — the fraction of the original uncompressed KV cache that is retained after compression. Budget ratios are typically swept from 0.2 to 1.0 (20% to 100% retention). For prefill-intensive tasks, all methods are evaluated under the query-agnostic setting following KVzip (Kim et al., 2025a), meaning the compression decision is made once during prefill and applies to all subsequent queries on that context. For decoding-intensive tasks, the paper adopts the R-KV setting (Cai et al., 2025) where the KV cache is continuously compressed during decoding to match a fixed-size budget (e.g., 4K tokens), covering both the original context and newly generated tokens. Inference efficiency comparisons (Figure 10) use PyTorch with FlashAttention-2 (Dao, 2024) on a single H100 GPU, measuring prefill time (seconds) and peak memory usage (GB) at context lengths from 160K to 320K tokens.

• Cross-validation / statistical protocol. The paper conducts a single set of gate training per model using the FineWeb-Edu corpus (Penedo et al., 2024) and evaluates on all downstream tasks without any task-specific fine-tuning or adaptation. Training and evaluation datasets are disjoint: FineWeb-Edu has no overlap with SCBench, RULER, LongBench, MRCR, SQuAD, AIME24, or MATH. The results in Figure 11 and 12 use the same trained gates across all tasks and budget ratios — there is no per-task or per-budget tuning. This is a key aspect of the evaluation design: it tests whether the gates genuinely learn a task-agnostic notion of KV importance, rather than overfitting to specific benchmarks. For AIME24, results are averaged over 16 random seeds to account for stochastic variation in the reasoning process. No cross-validation or statistical significance testing is reported for the main benchmark results, though the gate training process uses a validation loss for monitoring convergence (Figure 15).

Main Quantitative Results

Prefill-Intensive Tasks: Fast KVzip Matches KVzip While Outperforming All Other Baselines

The headline result for prefill-intensive scenarios appears in Figure 11 and Figure 12. Fast KVzip maintains full-cache performance at a 30–40% KV cache budget ratio across all 12 prefill-intensive benchmark datasets while matching the compression quality of KVzip — the most accurate prior method — without incurring KVzip's runtime reconstruction overhead.

Figure 11 presents per-dataset results for Qwen2.5-7B-1M across KV cache budget ratios from 0.2 to 1.0, organizing the 12 tasks into three categories: retrieval-intensive, contextual understanding, and high context redundancy. The key patterns:

• Fast KVzip tracks the full-cache baseline closely down to a budget ratio of 0.3–0.4. On most tasks, the performance curve for Fast KVzip remains essentially flat from budget ratio 1.0 down to 0.3, then begins a gradual decline below 0.3. This is the "near-lossless" regime the paper claims: at 30% retention (70% eviction), accuracy is indistinguishable from storing the full KV cache.

• Fast KVzip and KVzip produce nearly overlapping curves across all datasets and budget ratios. This is the central empirical validation of the distillation approach: the student gating modules, trained via reconstruction-based distillation, recover the teacher's (KVzip's) compression decisions with sufficient fidelity that downstream task performance is indistinguishable. The paper explicitly states this parity, noting that Fast KVzip "matches the performance of KVzip, which requires twice the prefilling computational cost for compression" (Section 4.2). This near-overlap holds across retrieval-intensive tasks (SCBench.KV, SCBench.SCROLLS, SCBench.SQuALITY, MRCR), contextual understanding tasks (SCBench.En.QA, SCBench.En.MCQ, SCBench.En.SUM, SQuAD), and high-redundancy code tasks (SCBench.RepoQA, SCBench.CodeCompose). The consistency across task types is notable because prior methods often show task-dependent degradation patterns.

• Fast KVzip substantially outperforms all non-KVzip baselines. SnapKV, Expected Attention with Ada-KV, and DuoAttention with KVzip score all show earlier and steeper performance degradation as the budget ratio decreases. On retrieval-intensive tasks (SCBench.KV, MRCR), the gap is particularly pronounced: SnapKV and Expected Attention begin diverging from full-cache performance at budget ratios around 0.6–0.7, while Fast KVzip maintains accuracy down to 0.3. On SCBench.SCROLLS, the advantage is similarly large. On contextual QA tasks (SCBench.En.QA, SQuAD), the gap is somewhat narrower but still consistent. This pattern — larger advantages on retrieval, smaller but still clear advantages on QA — mirrors the architectural findings in Figure 9, where the sink-attention architecture's benefits were most pronounced on retrieval tasks.

• On some tasks, Fast KVzip shows modest performance improvements over the full-cache baseline at intermediate compression ratios. The paper attributes this to "the denoising effect of attention induced by KV cache compression" (Section 4.2), citing Ye et al. (2025). The mechanism is that removing certain KV pairs can actually improve attention quality by eliminating noise or distraction, a phenomenon previously observed in differential attention mechanisms.

Figure 12 demonstrates generalization across model families by presenting averaged relative performance across all 12 datasets for five different models: Qwen2.5-7B-1M, Qwen3-8B, Qwen2.5-14B-1M, Gemma3-12B, and Qwen3-8B-FP8. Results are normalized to full-cache performance for each model-dataset pair. The key findings:

• The performance pattern is consistent across models. All five models show that Fast KVzip maintains near-100% relative performance down to budget ratios of 0.3–0.4, while SnapKV and Expected Attention diverge earlier. This is particularly significant because these models differ in training regime (Qwen2.5 vs. Qwen3), attention architecture (standard vs. Gemma3's hybrid sliding-window), and precision (BF16 vs. FP8).

• Qwen3-8B and Qwen3-8B-FP8 show nearly identical relative performance, confirming that the method is compatible with weight quantization without modification. The FP8 variant requires no special handling of the gating modules themselves — they operate on hidden states, which are computed in the model's native precision.

• Gemma3-12B, despite using hybrid sliding-window attention, benefits comparably to models with full global attention. Fast KVzip is applied only to the global attention KV cache (which dominates memory in long contexts), and the results validate that this selective application is sufficient.

• The larger model (Qwen2.5-14B-1M) shows similar compression tolerance to the smaller models. There is no evidence that larger models require different compression ratios or that the gates trained on 1M tokens fail to scale — despite the 14B model having more layers and more heads, the same training recipe and hyperparameters ($S = 16$, $D' = 16$) produce comparable compression quality.

Decoding-Intensive Tasks: Fast KVzip Preserves Reasoning Performance Where Early Stopping Fails

Figure 13 presents results on mathematical reasoning benchmarks (AIME24 and MATH) for Qwen3-8B and Qwen3-14B, comparing Fast KVzip against R-KV and the early-stopping-of-thinking baseline. The key findings:

• Fast KVzip achieves near-lossless performance at a KV budget of 4K tokens. On AIME24 with Qwen3-8B, the full-cache baseline (no compression, no budget constraint) achieves a certain accuracy; Fast KVzip at a 4K budget matches or nearly matches this performance. The exact numerical scores are visible in Figure 13's bar charts, though the paper does not quote them in the main text. On MATH with Qwen3-8B, the pattern is similar: Fast KVzip at 4K budget is visually comparable to the full-cache baseline.

• Fast KVzip outperforms R-KV at the same KV budget. The R-KV baseline, which uses redundancy-aware heuristics rather than learned gates, shows lower accuracy than Fast KVzip at both 4K and 8K budget sizes on AIME24 for Qwen3-8B. The gap is consistent across both model scales.

• The early-stopping-of-thinking baseline drastically degrades reasoning performance. This method forces the model to terminate its thinking (chain-of-thought) process early to fit within a token budget, then generate the final answer. Figure 13 shows that this causes a substantial accuracy drop compared to both the full-cache baseline and Fast KVzip at the same budget. On AIME24 with Qwen3-14B, the early-stopping bar is markedly lower. The paper interprets this as evidence that "the model requires a sufficient length of thinking to deduce correct answers, and that Fast KVzip indeed maintains the necessary KV features for general inference, including reasoning procedures" (Section 4.3). In other words, you cannot simply truncate the reasoning process to save memory — you need to compress the KV cache so that the full reasoning process can fit within the available memory budget.

• Qwen3-14B shows the same patterns as Qwen3-8B. The larger model benefits similarly from Fast KVzip's compression, with near-lossless performance at 4K budget on AIME24 and MATH. This consistency across scales suggests the gating mechanism captures scale-invariant properties of attention, not artifacts of a particular model size.

The key takeaway from the decoding-intensive results is that Fast KVzip enables long-chain reasoning within constrained memory by compressing the KV cache rather than truncating the reasoning process. The distinction matters because the model's reasoning capability depends on maintaining the full context of its thinking chain — early stopping forces the model to conclude prematurely, while KV compression allows it to think fully but with a more memory-efficient representation of that thinking.

Inference Efficiency: Fast KVzip Reduces Both Latency and Memory vs. No Compression

Figure 10 quantifies the practical efficiency benefits on Qwen2.5-7B-1M with a 30% KV budget ratio, measuring prefill time and peak memory across context lengths from 160K to 320K tokens on a single H100 GPU.

• Prefill (Figure 10a): Fast KVzip reduces prefill time compared to the no-compression baseline, and dramatically reduces it compared to KVzip. At 320K context length, the no-compression baseline takes a certain prefill time; Fast KVzip is modestly faster (the reduction being due to processing fewer KV features in subsequent attention operations). KVzip's prefill time is approximately double the baseline at 320K because it must re-process the entire context during its reconstruction phase. Fast KVzip eliminates this reconstruction entirely, bringing prefill time below the no-compression baseline. For peak memory usage during prefill, Fast KVzip shows substantial reduction: at 320K, memory drops from approximately 80 GB (no compression) to approximately 45 GB. KVzip's peak memory is intermediate — higher than Fast KVzip because it must temporarily store the full KV cache during reconstruction before compression.

• Decoding (Figure 10b): Fast KVzip reduces decoding latency and memory compared to the no-compression baseline at all measured context lengths. The paper notes that KVzip provides similar decoding speed to Fast KVzip (since KVzip's reconstruction happens during prefill only, and decoding uses the compressed cache), and the KVzip decoding curve is omitted from Figure 10b for clarity. The memory advantage during decoding is consistent with the prefill pattern: storing 30% of the KV cache reduces memory proportionally, and this benefit persists throughout autoregressive generation.

The efficiency results validate the paper's central value proposition: Fast KVzip matches the compression quality of the most accurate prior method (KVzip) while eliminating the runtime overhead that made that method impractical for latency-sensitive deployment. The gains are not merely in FLOPs or abstract compute units — they translate directly to wall-clock time and peak memory, the practical metrics that determine deployment feasibility and cost.

Training Efficiency: Under One GPU-Hour for 14B-Scale Models

Table 2 quantifies the gate training cost:

• Qwen3-8B: 0.59 H100 hours, 0.18 GB storage for gate parameters.
• Qwen3-30B-A3B: 0.70 H100 hours, 0.11 GB storage.
• Qwen3-14B: 0.83 H100 hours, 0.30 GB storage.

These numbers are remarkably low. The training time is dominated by the forward passes needed to precompute hidden states and KVzip target scores for 1M tokens, plus the subsequent gate optimization (5K update steps with batch size 1K). The storage requirements are negligible relative to the model weights themselves (a 14B model in BF16 is approximately 28 GB, so 0.30 GB of gate parameters represents roughly a 1% increase). The 30B-A3B model has smaller gates (0.11 GB) because its hidden state dimension is smaller than the dense models despite the larger total parameter count (most parameters are in the MoE's expert feedforward networks, not in the attention layers where the gates operate).

Ablation Studies and Robustness Checks

The paper includes extensive ablation studies in the appendices (Section B, Figures 16–19) and in the main method section (Figures 5, 7, 9). While the prior-written sections already cover the gate architecture ablation (Figure 9) and the target signal / gate input ablations (Figures 5, 7), the remaining ablations address the method's sensitivity to training data, hyperparameter choices, and inference configurations.

Training data size and composition (Figure 16, Appendix B.1): Increasing the amount of FineWeb-Edu training data from 250K to 1M tokens yields performance improvements, particularly on SCBench.KV, the synthetic key-retrieval task. The 1M-token configuration (the paper's default) outperforms 500K and 250K variants. To isolate the effect of long-context training data, the authors construct a "500K-short" dataset by removing the 100K-token concatenated sequences from the 1M-token dataset. This degrades performance compared to the full 1M-token dataset, demonstrating that including long-context samples (where KV compression must handle genuinely long-range dependencies) is beneficial for gate training. However, even the 250K-token configuration shows substantial improvements over baselines, indicating the method is not critically dependent on massive training data.

Gate architecture hyperparameters (Figure 17, Appendix B.2): The number of sink keys $S$ and projection dimension $D'$ are varied for Qwen2.5-7B-1M on SCBench.KV. With $S = 4$, performance is noticeably lower than with $S = 16$ (the default), but performance stabilizes at $S = 16$ with minimal further improvement at $S = 32$ or $S = 64$. For projection dimension, values from $D' = 8$ to $D' = 64$ produce comparable results, with $D' = 16$ being the chosen default. This robustness is practically important: deployers do not need to tune these hyperparameters per model or per task. The finding that 16 sink keys suffice suggests that the "baseline importance distribution" for each attention head can be captured by a relatively small number of reference vectors.

Local window size (Figure 18, Appendix B.3): Retaining a local window of recent tokens consistently improves performance compared to no local window, validating the design choice to decouple positional recency from gate-based content scoring. Window sizes from 1K to 8K tokens produce comparable results, with performance being "robust to the choice of local window size." The paper uses 4K tokens for prefill and 128 tokens for decoding, but the flat sensitivity curve suggests these choices are not critical.

Comparison to TrimKV on general tasks (Figure 19, Appendix C): When compared to the official TrimKV model for Qwen3-4B-Instruct-2507 on general language tasks (using prefill-intensive benchmarks from Figure 11, averaged and normalized to full-cache performance), Fast KVzip substantially outperforms TrimKV, particularly on retrieval tasks. This is notable because TrimKV trains separate gating models for mathematical reasoning vs. general language using distinct datasets (Table 4), while Fast KVzip uses a single set of gates trained on general text data. The result supports the paper's claim that reconstruction-based training targets produce more task-generalizable compression than next-token prediction targets.

Training loss convergence (Figure 15, Appendix A.2): The training and validation loss curves show stable convergence across all attention layers. Earlier layers achieve lower loss values, while middle and later layers show higher loss — the authors hypothesize this reflects "more complex attention mechanisms" in deeper layers that are harder to predict from a single hidden state. The losses converge without signs of overfitting (training and validation curves track closely), which is expected given the small parameter count of the gates relative to the 1M-token training set.

Gate prediction visualization (Figure 20, Appendix C): A token-level comparison of Fast KVzip's predicted scores against KVzip's target scores on SCBench.En.QA (a dataset not used during training) shows that the gates capture token-level dynamics without over-smoothing. The per-layer averaged scores track the target distribution closely, with the dashed threshold line indicating the cutoff that achieves 36% retention. This visualization provides qualitative evidence that the gates learn fine-grained, context-sensitive importance predictions rather than collapsing to a uniform or layer-constant score.

Critical Assessment

Claim 1: Fast KVzip maintains near-lossless performance while evicting up to 70% of the KV cache.

The evidence for this claim is strong and consistent across the 12 prefill-intensive benchmarks in Figure 11. On nearly all tasks, the performance curve remains flat from budget ratio 1.0 down to 0.3, with degradation beginning below that threshold. The claim of "near-lossless" at 30% retention is visually supported — the Fast KVzip curve overlays the full-cache baseline at 0.3 on most datasets. However, the reader should note: "near-lossless" does not mean "zero degradation." On some tasks, there is a small but visible dip at 0.3 (e.g., SCBench.SQuALITY, SCBench.CodeCompose), and the claim is better characterized as "degradation is small enough to be practically acceptable" rather than "literally indistinguishable from full cache." The paper's own data shows that at more aggressive compression ratios (0.2 budget, meaning 80% eviction), performance does drop meaningfully on retrieval tasks, so the 70% eviction figure represents an approximate ceiling rather than a hard guarantee.

A genuine limitation: the 30–40% budget ratio that achieves near-lossless performance is presented as a single range, but Figure 11 shows that this threshold varies somewhat by task. On SCBench.KV (synthetic key retrieval), performance holds well even at 0.2, while on SCBench.SQuALITY, there is a slight decline by 0.3. The paper does not quantify this task-dependent variation, and a practitioner deploying Fast KVzip would need to determine the appropriate budget ratio for their specific task distribution — the 30–40% range is a guideline, not a guarantee.

Claim 2: Fast KVzip matches the compression performance of KVzip while eliminating runtime reconstruction overhead.

This claim is well-supported by Figure 11, where Fast KVzip and KVzip curves are visually nearly identical across all datasets and budget ratios. The paper's key differentiator is not better compression quality than KVzip, but equivalent quality with dramatically lower runtime cost (Figure 10). This is an appropriate and honest positioning: the method does not claim to surpass the reconstruction oracle, only to match it efficiently.

However, a nuance: the claim holds for the specific models and benchmarks tested. The distillation from KVzip to Fast KVzip's gates is performed on FineWeb-Edu data with the same model that will be used at inference time. If a deployer wanted to use Fast KVzip on a model not in the tested set (e.g., a Llama-series model), they would need to run the full KVzip reconstruction process on that model to generate training targets — which requires the very computational expense Fast KVzip is designed to avoid. The method's efficiency story is therefore conditional on the availability of pre-trained gates. For the specific models tested (Qwen2.5, Qwen3, Gemma3), the gates are provided; for arbitrary new models, the upfront training cost — while modest — must be paid. The paper is transparent about gate training costs (Table 2), but the framing of "eliminates runtime overhead" should be understood as "amortizes the one-time training overhead across all subsequent inferences."

Claim 3: Fast KVzip generalizes across diverse tasks (retrieval, QA, code, math) and diverse model families.

This claim is strongly supported by the breadth of evaluation. The 12 prefill-intensive benchmarks (Figure 11) span synthetic retrieval, long-document QA, summarization, and code comprehension — a notably diverse set. The decoding-intensive results (Figure 13) extend to mathematical reasoning, a domain with fundamentally different attention patterns than document understanding. The model diversity (Figure 12) covers different pretraining regimes, attention architectures, and precision formats. This is a substantial evaluation scope that addresses the primary weakness of prior learned methods: TrimKV requires task-specific training, Locret is QA-only, but Fast KVzip works across the board with a single set of gates.

A limitation worth noting: all evaluations use models from the Qwen and Gemma families, which share architectural similarities (they are all relatively modern, dense transformer variants with GQA and RoPE). The paper does not test on, for example, Llama-3, Mistral, or DeepSeek models. While the architectural diversity within the tested set is meaningful (hybrid attention in Gemma3, MoE in 30B-A3B), the claim of "generality" would be stronger with cross-family validation. Additionally, the Gemma3-12B evaluation only compresses the global attention KV cache, not the sliding-window attention cache — this is a reasonable design choice but means the evaluation does not fully test Fast KVzip's behavior when applied to sliding-window attention layers.

Claim 4: The sink-attention architecture is necessary for strong performance, particularly on retrieval tasks.

Figure 9 supports this claim clearly: the linear model underperforms substantially, the two-layer MLP and simplified sink-attention (no learnable denominator) are competitive on QA but degrade on retrieval, and the full sink-attention uniquely maintains performance across both task types. This is a clean ablation result with clear practical implications.

However, the ablation is parameter-matched — all architectures have approximately the same number of parameters — but not compute-matched. The full sink-attention involves more complex operations (multiple exponentiations, summation over sink keys) than a simple MLP. The paper does not report the latency difference between architectures during inference. If the sink-attention mechanism adds meaningful per-token computation compared to a simpler MLP, the efficiency claim ("~1% overhead") might not hold for the MLP alternative — the MLP would have even lower overhead. The paper's claim that the sink-attention is worth its cost is implicit in the performance results but not explicitly validated with latency measurements of alternative architectures.

Claim 5: Reconstruction-based training targets generalize better than next-token prediction or instruction-QA targets.

Figure 5 provides clean evidence: reconstruction-trained gates outperform next-token-prediction-trained and QA-trained gates on both SCBench.KV and SQuAD. This is a critical ablation because it explains why prior gating methods (TrimKV with next-token prediction, Locret with QA) fail to generalize. The visualization in Figure 6 provides mechanistic interpretation: reconstruction attention is more uniform and structured, while next-token and QA attention are sparser and more task-specific.

A missing experiment: the paper does not test what happens when multiple target signals are combined. Would training on reconstruction + next-token prediction + QA targets produce even more robust gates, or would the task-specific signals contaminate the generalization? Given the paper's framing that reconstruction already captures the "high-level contextualized features" that downstream tasks need, the hypothesis would be that adding other signals provides no benefit — but this is untested.

Claim 6: Positional information degrades gating performance and should be handled by a separate local window mechanism.

Figure 7 supports this: hidden states (no explicit position encoding) outperform key states (with RoPE) and pre-RoPE key states. The paper's interpretation — that position is a confound for content-based importance prediction — is plausible and supported by the data. However, the ablation tests only the choice of gate input, not the mechanism by which position is handled. The local window is always present (as a separate mechanism), so the experiment is: given that recency is already captured by the local window, does adding positional information to the gate help? The answer is no. But this does not fully test the counterfactual: if the local window were removed and positional information were provided to the gate, would performance degrade below the no-window, no-position baseline? The current design treats position and content as separable by design, and the ablation validates that given this separation, positional features in the gate are harmful. But it does not prove that a single unified mechanism (gate with positional features, no separate window) could not work — it only shows that the separated design is effective.

Weaknesses and missing experiments:

1. No confidence intervals or statistical testing. The main results (Figures 11, 12, 13) report point estimates without error bars. For the normalized average in Figure 12, the variability across the 12 datasets is effectively a measure of consistency, but per-dataset variability (from multiple runs or multiple random seeds) is not reported. This makes it difficult to assess whether small differences between methods (e.g., Fast KVzip vs. KVzip at specific budget ratios) are statistically meaningful or within noise.

2. Single training corpus. All gate training uses FineWeb-Edu, which is a web-text corpus. Would gates trained on code-heavy data perform better on code comprehension tasks? Would gates trained on mathematical text perform better on AIME24? The paper's task-agnostic claim implies the answer should be "no significant difference," but this is untested. If there is a domain gap, it would affect how practitioners choose training data for their specific deployment domain.

3. No end-to-end serving throughput measurement. Figure 10 reports prefill time and decoding latency for a single sequence, but production serving performance depends on throughput (queries per second) under batching. The paper does not measure how Fast KVzip affects maximum batch size (the compressed KV cache enables larger batches) or how the buffered decoding strategy interacts with continuous batching schedulers. These are systems-level metrics that would strengthen the deployment practicality claim.

4. Limited budget ratios during decoding. The decoding-intensive experiments (Figure 13) report results at 4K and 8K budget sizes, but do not sweep a range of budget ratios as the prefill experiments do. This makes it difficult to determine the compression ratio at which reasoning performance begins to degrade — the "near-lossless" threshold for decoding may differ from the prefill threshold.

5. No comparison to feature-dimension compression or quantization-only baselines. The paper focuses exclusively on token-level eviction, but complementary techniques like KV quantization (KIVI, Liu et al., 2024b) or feature-channel compression (Think, Xu et al., 2025) could be combined with Fast KVzip. The paper does not establish where token-level eviction alone stands relative to these orthogonal approaches, nor whether the combination would yield multiplicative benefits.

6. SQuAD evaluation details are sparse. SQuAD is mentioned as a multi-query QA dataset, but the paper does not specify whether it uses SQuAD v1.1 or v2.0, what context length is used, or how multi-query evaluation is structured (multiple questions per context). Given SQuAD's typical context lengths are much shorter than 170K tokens, it is unclear how informative this benchmark is for long-context compression evaluation specifically.

7. The "denoising effect" is mentioned but not systematically studied. The paper notes that Fast KVzip sometimes outperforms the full-cache baseline (Section 4.2) and attributes this to "the denoising effect of attention." This is an interesting observation, but the paper does not analyze which tasks show this effect, at what compression ratios, or whether specific attention heads are responsible. It remains a qualitative observation rather than a studied phenomenon.

Despite these limitations, the experimental evaluation is thorough by the standards of the KV cache compression literature. The breadth of tasks (prefill-intensive + decoding-intensive, retrieval + QA + code + math), model diversity, and systematic ablations substantially exceed what is typical for the field. The central claims — that Fast KVzip matches KVzip's compression quality with negligible overhead and generalizes across tasks — are well-supported by the reported data. The main gaps are in statistical rigor (error bars), systems-level throughput evaluation, and cross-corpus training validation, which represent opportunities for future work rather than fatal flaws in the current evidence.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Numbers

The assumption or constraint. The entire Fast KVzip framework depends on pre-computed KV importance scores from the KVzip reconstruction process to serve as training targets for the gating modules. The paper is explicit about this dependency: the gate training procedure requires "target scores derived from KVzip's reconstruction process" (Section 3.3), which involves re-feeding the entire training context through the frozen LLM — essentially doubling the forward-pass computation for every training token. While the paper reports gate training time as under one H100 GPU-hour for 14B-scale models (Table 2), this figure accounts only for the gate optimization step (stochastic gradient descent on pre-computed hidden-state/target-score tuples). It does not include the cost of running the full KVzip reconstruction process to generate those target scores in the first place.

The paper acknowledges this implicitly in its description of the training procedure — "reconstruction can be performed using parallelized forward passes" (Section 3.3) — but it never quantifies the computational cost of this reconstruction phase. The 1M training tokens must be processed twice: once to extract hidden states, and once (via KVzip) to produce the target importance scores. For a 14B-parameter model processing 1M tokens with chunked prefill at 16K-token chunks, the reconstruction step alone requires processing the equivalent of 1M additional forward-pass tokens (since each sentence is fed back as a query). This cost is not amortized or reported.

The consequence. A practitioner seeking to deploy Fast KVzip on a new model — one not in the set of models for which the authors provide pre-trained gates — must first implement or replicate the KVzip reconstruction pipeline to generate training targets. This is not a trivial engineering task: it requires modifying the inference code to perform sentence-level reconstruction, collecting attention scores at every layer, and normalizing them into importance targets. The computational cost of this step, while one-time, may be substantial for very large models or for organizations without access to the specific KVzip implementation used by the authors.

More importantly, the one-time cost is not a fixed constant — it scales with the model size and the amount of training data. If a practitioner decides they need more than 1M tokens of training data (e.g., for a domain-specific deployment where FineWeb-Edu is out of distribution), the reconstruction cost scales linearly. The paper's ablation in Figure 16 shows that increasing training data from 250K to 1M tokens improves performance, suggesting that practitioners with demanding accuracy requirements might want even more training data — and would pay proportionally higher reconstruction costs. The "under one GPU-hour" figure is therefore a lower bound that excludes the most computationally intensive step in the pipeline.

What evidence exists in the paper. The paper does not measure or report the reconstruction cost. Figure 15 shows training loss curves for the gate optimization step, but the process of generating the target scores that make those loss curves possible is treated as a given. Figure 16 explores training data size effects on downstream performance, but does not report the wall-clock time required to generate the training data at each size. Table 2 reports gate training time and storage, with the implicit assumption that target scores are already available. The paper never states the total end-to-end cost from raw model to deployed gates, which is the cost a practitioner actually needs to budget for.

Mitigation status. Not addressed. The authors do not discuss the reconstruction cost, do not propose methods to reduce it, and do not explore whether cheaper alternatives (e.g., using a smaller model for reconstruction, or using a subset of sentences rather than full reconstruction) could produce sufficiently high-quality training targets. This is a significant omission given that the paper's central value proposition is eliminating inference-time overhead — if the training-time overhead is itself substantial, the practical advantage over KVzip narrows, since KVzip's cost is incurred per-inference while Fast KVzip's equivalent cost is incurred once during training. The tradeoff is favorable for high-volume deployments (amortize one-time training cost over many inferences) but unfavorable for low-volume or single-use scenarios, and the paper does not help practitioners evaluate where the crossover point lies.

---

6.2 Hard Problems Where the Base Model Produces No Correct Solution Receive No Benefit

The assumption or constraint. Fast KVzip compresses the KV cache by predicting which stored key-value pairs will be useful for future attention. This fundamentally assumes that the KV features needed for correct task performance exist in the cache to begin with — that is, that the base model processes the context in a way that produces the necessary intermediate representations. For problems that are genuinely outside the model's capability range, where its attention patterns fail to capture the relevant information regardless of the amount of context stored, no compression method that operates by filtering existing KV features can help.

The paper does not explicitly state this limitation, but it is an inescapable consequence of the method's design. Fast KVzip is a selection mechanism, not a generation mechanism: it chooses which KV pairs to keep from those the model has already produced, but it cannot create new KV features or restructure the model's attention patterns to capture information it otherwise misses. If the base model's attention is fundamentally inadequate for a task — if it simply does not know how to attend to the right information — Fast KVzip cannot compensate.

The consequence. There exists a class of tasks — analogous to the "hardest problems" (difficulty bin 5) in the PaLM 2-S* scaling analysis — where Fast KVzip will provide zero benefit regardless of compression ratio, because the base model's full-cache performance on those tasks is already near zero. The paper's evaluation does not systematically characterize this boundary. All reported results show Fast KVzip maintaining or approaching full-cache performance, but this is selection bias: the benchmarks are chosen because the base models achieve non-trivial performance on them. If a practitioner deploys Fast KVzip on a task where the base model struggles (e.g., a very long document with subtle multi-hop reasoning that the base model simply cannot track), compression will not make things worse (the gate still retains the "best" KV pairs), but it also will not make things better, and the compute spent on gate training and inference is wasted relative to simply running the base model with the full (uncompressed) cache.

What evidence exists in the paper. The paper does not explicitly evaluate this limitation. All benchmarks — SCBench, MRCR, SQuAD, AIME24, MATH — are tasks on which the tested models (Qwen2.5, Qwen3, Gemma3) achieve non-trivial full-cache accuracy. There is no "stress test" on deliberately adversarial or out-of-distribution inputs where the base model's performance is near floor. The paper also does not report per-task full-cache accuracies for all benchmarks, making it impossible for a reader to identify which tasks fall into the "model already struggles" regime. The closest indirect evidence comes from the performance curves in Figure 11: on tasks where the full-cache baseline accuracy is relatively low (some contextual QA tasks, some code comprehension tasks), the gap between full-cache and compressed-cache Fast KVzip narrows or compresses at higher budget ratios. But this is correlation, not systematic characterization.

Mitigation status. Not addressed. The paper does not discuss the capability boundary, does not provide guidance on when Fast KVzip is unlikely to help, and does not propose mechanisms for detecting at inference time whether the base model is in a regime where compression is counterproductive. This is a meaningful gap for practitioners: a deployment engineer needs to know whether investing in Fast KVzip integration will yield benefits for their specific task distribution, and the paper provides no diagnostic framework for making this determination beyond "run the full benchmark suite on your task and see."

---

6.3 Training Data Domain Mismatch May Affect Compression Quality on Specialized Content

The assumption or constraint. All gate training in the paper uses the FineWeb-Edu corpus (Penedo et al., 2024), a web-text dataset. The training data consists of "randomly sampled sequences ranging from 10K to 30K tokens, yielding a total of 1M training tokens," plus additional concatenated 100K-token sequences (Section 3.3, Appendix A.1). The paper explicitly characterizes this as a "task-agnostic reconstruction objective" (Abstract, Section 3.3) and demonstrates strong generalization across the evaluated benchmarks. However, the content domain of the training data is held constant: web text, primarily educational in nature (FineWeb-Edu is a quality-filtered subset of web crawl data). The evaluation benchmarks span a range of content types — synthetic key-value retrieval (RULER), book-length narratives (SCBench.SCROLLS), code repositories (SCBench.RepoQA, SCBench.CodeCompose), mathematical reasoning (AIME24, MATH), and Wikipedia-based QA (SQuAD) — but the gates are always trained on web text.

The assumption is that the attention patterns learned from reconstructing web-text contexts transfer to these diverse content types. The paper provides evidence that this transfer works for the specific models and benchmarks tested, but does not establish that the transfer is universal or that domain mismatch between training data and deployment content is safe.

The consequence. For a practitioner deploying Fast KVzip on a specialized domain — medical records, legal documents, financial filings, non-English text, highly technical scientific literature — the question is: will gates trained on general web text accurately predict KV importance for content whose linguistic structure, information density, and attention patterns differ substantially from the training distribution? The paper provides no direct evidence on this question.

Consider the extreme case: code comprehension (SCBench.RepoQA, SCBench.CodeCompose). The paper's Figure 11 shows Fast KVzip performing well on these tasks despite being trained on web text. This is encouraging but not definitive — code in the evaluated benchmarks is primarily Python and common languages, which may share structural properties with the technical web text in FineWeb-Edu. A deployment on, say, legacy COBOL codebases or highly obfuscated source code might exhibit different attention patterns that the web-text-trained gates fail to capture. Similarly, mathematical reasoning (AIME24) involves sequences of symbols, equations, and structured reasoning chains that have no analogue in web text; Figure 13 shows good performance, but only for Qwen3 models that already have strong math capabilities. A model less capable at math might show different gating behavior on math content.

More concretely: if KV importance in legal documents depends on attention to definitional sections, cross-reference clauses, and citation anchors — patterns that may not be well-represented in web text — then gates trained on web text might systematically underweight these tokens, leading to higher-than-expected accuracy degradation on legal QA tasks. The paper provides no framework for diagnosing or mitigating such domain mismatches.

What evidence exists in the paper. The paper intentionally demonstrates breadth — 12 prefill benchmarks across retrieval, QA, summarization, and code, plus 2 math benchmarks — which provides evidence that web-text-trained gates transfer across content types within the set of tested domains. However, this evidence is necessarily bounded by the benchmark selection. None of the benchmarks involve specialized professional domains, non-English text, or content with radically different linguistic properties from web text. The paper's claim of "task-agnostic" training is about the objective (reconstruction rather than next-token prediction or QA), not about the data distribution, and the paper conflates these two dimensions. A gate trained with a task-agnostic objective on domain-mismatched data may still underperform on out-of-domain content.

Mitigation status. Not addressed as a limitation. The paper does not discuss domain transfer, does not evaluate on specialized-domain benchmarks, and does not explore whether fine-tuning the gates on a small amount of domain-specific data would improve performance. Given that gate training is cheap (~1 H100 hour), domain-specific gate fine-tuning would be a natural mitigation — re-running the reconstruction process on domain-specific text and fine-tuning the existing gates rather than training from scratch — but this is not explored or suggested.

---

6.4 The Method Has Not Been Tested Across Model Architectures Outside the Qwen/Gemma Families

The assumption or constraint. All evaluations in the paper use models from two families: Qwen (Qwen2.5-7B/14B-1M, Qwen3-8B/14B, Qwen3-8B-FP8, Qwen3-30B-A3B) and Gemma (Gemma3-12B). The paper claims that Fast KVzip "demonstrates the generality of our approach" (Abstract), but this generality is tested only within a narrow architectural range. All tested models share key design features: they are relatively modern dense transformers (the 30B-A3B is MoE but the attention layers are architecturally similar), they all use rotary position embeddings (RoPE), they all use grouped-query attention (GQA), and they are all trained by organizations (Alibaba, Google) whose pretraining recipes may share common design patterns.

Notably absent from the evaluation are Llama-family models (Meta), Mistral-family models, DeepSeek-family models, or older architectures that use multi-head attention (MHA) rather than GQA. These models differ in ways that could affect gating behavior: different normalization placements (pre-norm vs. post-norm), different activation functions, different attention implementations, and different pretraining data mixtures that produce different attention patterns.

The consequence. The paper's strongest claim — that Fast KVzip provides "near-lossless performance while evicting up to 70% of the KV cache" — is empirically supported only for Qwen and Gemma models. A practitioner using Llama-3-8B, Mistral-7B, or DeepSeek-V2 has no direct evidence that Fast KVzip will work for their model. Given that the method requires per-model gate training (the gates are layer-specific and depend on the model's hidden state representations), a practitioner must either (a) trust that the approach transfers without empirical validation, or (b) implement the full KVzip reconstruction pipeline, train gates, and run their own evaluation — a non-trivial engineering investment with uncertain payoff.

The risk is not merely that the exact 70% eviction number might differ. It is that fundamental properties of the attention patterns — which the gating mechanism relies on to predict KV importance from hidden states — might differ across model families in ways that make the core insight (that hidden states encode future KV utility) less robust. For example, if a model architecture uses a different attention formulation (e.g., linear attention, or attention with different masking patterns), the relationship between hidden states and KV importance that the paper demonstrates for Qwen/Gemma might weaken or disappear entirely.

What evidence exists in the paper. The paper evaluates five distinct models, but all are closely related. Qwen2.5-7B-1M and Qwen2.5-14B-1M are the same architecture at different scales. Qwen3-8B and Qwen3-14B share the same architecture and pretraining regime. Qwen3-8B-FP8 is a quantized version of Qwen3-8B. Gemma3-12B is the only genuinely distinct architecture, and it is notable that the paper acknowledges a limitation with Gemma3 — "for the hybrid model, we only compress the global attention KV cache" (Section 4.1) — which means the evaluation on Gemma3 is not fully comparable to the Qwen evaluations. The Qwen3-30B-A3B MoE model is mentioned only in the training efficiency table (Table 2) and is not evaluated on any downstream benchmarks.

The paper does not discuss this architectural scope limitation, does not acknowledge that "generality" is tested within a narrow band of the model design space, and provides no theoretical argument for why the approach should transfer to fundamentally different architectures.

Mitigation status. Not addressed. The paper presents its model diversity as a strength — testing across "a diverse set of state-of-the-art open-source large language models" (Section 4.1) — without acknowledging that this "diversity" is limited to two model families. The authors do not discuss which architectural properties are necessary for Fast KVzip to work, do not speculate on transfer to other families, and do not provide guidance for practitioners using non-Qwen/non-Gemma models. This is a significant limitation for a method that claims generality and targets practical deployment.

---

6.5 The Local Window Strategy Fragments the Compression Design Space and Prevents End-to-End Learning

The assumption or constraint. Fast KVzip decouples position-based retention from content-based retention: a fixed-size local window of recent tokens (4K for prefill, 128 for decoding) is always retained regardless of gating scores, while the gating mechanism handles content-based importance for all other tokens. The paper validates this separation empirically (Figure 7 shows that positional features degrade gate performance; Figure 18 shows that a local window improves performance). However, this separation is architecturally imposed, not learned. The gate has no mechanism to determine whether a particular token in the window is actually important; it simply cannot evict window tokens. Conversely, the gate has no mechanism to signal that an old token is so important that it should be treated with window-like protection.

The consequence. The local window size — 4K tokens for prefill, 128 for decoding — is a hyperparameter that the paper treats as fixed across all models and tasks. But the optimal window size likely depends on the task. For a task requiring fine-grained local reasoning over recent context (e.g., code completion where the last few lines are critical), a larger window may be beneficial. For a task requiring long-range retrieval where the window consumes budget that could be better allocated to older, content-important tokens, a smaller window may be optimal. The paper's sensitivity analysis (Figure 18) shows that performance is "comparable across local window sizes ranging from 1K to 8K tokens" — but this analysis is on a single task (SCBench.KV) with a single model (Qwen2.5-7B-1M). It does not establish that window-size insensitivity holds across tasks and models.

More fundamentally, the fixed window creates a structural inefficiency in the compression budget. If the model's attention patterns for a particular task exhibit strong recency bias (most attention goes to the last 2K tokens), retaining a 4K window wastes budget on 2K tokens that rarely receive attention. If the attention patterns exhibit weak recency bias (important information is distributed uniformly across the context), the window is too small and compresses all equally, potentially evicting distant-but-important tokens. The gate cannot adapt the window size dynamically because the window is a non-learned fixed mechanism. This means the compression is suboptimal relative to what an end-to-end learned system — one where the gate itself learns to handle recency — could achieve.

The paper's argument against end-to-end learning of position is that positional features in the gate input degrade performance (Figure 7). But this only shows that naively adding positional features to the current architecture degrades performance; it does not demonstrate that a different architecture — perhaps one that jointly models content and position in a more sophisticated way — could not outperform the separated design. The fixed window is a pragmatic engineering choice that works well, but it represents a hard-coded prior that limits the method's adaptivity.

What evidence exists in the paper. Figure 18 demonstrates that window size is not highly sensitive on the tested task. Figure 7 shows that adding positional information to the gate degrades performance, supporting the separation design. However, neither experiment tests whether a learned mechanism for handling recency (beyond a fixed window) could improve compression. The paper does not ablate the interaction between window size and compression budget — for a fixed total budget, the window consumes a fixed fraction, and the remaining budget is allocated by the gate. As the total budget shrinks, the window consumes a larger fraction, potentially starving the content-based gating of resources. This interaction is unexplored.

Mitigation status. Partially mitigated by empirical validation. The sensitivity analysis in Figure 18 shows robustness to window size over a reasonable range, suggesting that the fixed window is not catastrophically fragile. However, the paper does not address the more fundamental limitation: the inherent suboptimality of a non-learned separation between positional and content-based retention. The authors do not propose adaptive window mechanisms or explore architectures that could jointly learn both signals.

---

6.6 The Buffered Decoding Strategy Introduces a Latency-Accuracy Tradeoff That Is Not Characterized

The assumption or constraint. During decoding, Fast KVzip does not update importance scores at every token generation step. Instead, it maintains a hidden-state buffer of size 128 and computes gating decisions in parallel when the buffer fills (Section 3.1). The paper reports that naive per-step gating causes a "30–60% latency increase" and that the buffered approach reduces this to "approximately 1% of the model's forward-pass latency on average" (Section 3.1). This is presented as a pure improvement — the buffered strategy eliminates overhead without meaningful downside.

However, the buffered strategy introduces a staleness problem: for up to 127 tokens after a gating pass, the importance scores in the KV cache are based on hidden states that are now out of date. New tokens have been generated, the context has shifted, and attention patterns may have changed. Yet the gate does not update its scores — meaning that KV pairs that have become important in light of recent context cannot be promoted, and KV pairs that have become irrelevant cannot be demoted, until the next buffer flush. During this window, the KV cache is operating with stale importance estimates.

The consequence. In the worst case, the 128-token window between gating updates could span a critical reasoning transition. Consider a model generating a chain-of-thought solution to a math problem. The model might spend several steps exploring one approach, then realize it is wrong and pivot to a different approach at step 73 of the 128-token buffer. The KV pairs associated with the now-abandoned approach might have received high importance scores during the last gating pass (because they were relevant to the exploration), while KV pairs associated with the new approach might have received low scores (because they seemed irrelevant at the time). The gate cannot update scores until step 128, meaning the cache retains potentially useless KV pairs from the abandoned approach while potentially having evicted useful pairs for the new approach — all because the importance estimates are 55 steps stale.

The paper does not characterize how often such transitions occur in practice, nor does it measure whether the accuracy impact of stale scores is measurable. The "approximately 1% overhead" claim masks a latency-accuracy tradeoff: the buffer size controls this tradeoff (smaller buffer = lower staleness but higher overhead), and the paper's fixed choice of 128 is an unexamined point on what might be a meaningful Pareto frontier.

What evidence exists in the paper. The paper does not ablate the buffer size. The decoding results in Figure 13 show that Fast KVzip performs well at 4K and 8K budget sizes on AIME24 and MATH, but these results are the aggregate outcome after many decoding steps — they do not isolate whether performance degrades specifically during the stale windows. The paper does not compare per-step vs. buffered gating in terms of accuracy (only in terms of latency overhead), so the reader cannot assess how much accuracy, if any, is sacrificed by the batching strategy.

Mitigation status. Not addressed as a tradeoff. The paper presents the buffered decoding strategy as a pure efficiency optimization and does not discuss staleness, does not ablate buffer size against accuracy, and does not explore adaptive buffering strategies (e.g., flushing the buffer early when the generation appears to be transitioning between reasoning modes). The buffer size of 128 appears to be a single empirically chosen value, not the result of a systematic tradeoff analysis. For practitioners concerned about worst-case behavior on tasks with volatile attention patterns (e.g., multi-turn reasoning, debate-style generation), the lack of staleness characterization is a meaningful gap.

7. Implications and Future Directions

How This Work Changes the Landscape

Fast KVzip does not introduce a new compression algorithm in the conventional sense — it does not propose a better heuristic for which KV pairs to discard, nor a more sophisticated attention-pattern analysis, nor a cleverer way to exploit sparsity. What it introduces is a reframing of KV cache compression as a function approximation problem rather than a heuristic design problem, and it provides the empirical scaffolding — the gating architecture, the training target, the input representation — that makes this reframing productive. This is a conceptual shift whose significance lies not in surpassing KVzip's compression quality (it doesn't — it matches it) but in demonstrating that the expensive algorithmic process KVzip performs at runtime can be distilled into a lightweight, feed-forward predictor that runs with negligible overhead.

The magnitude of this shift is best characterized as a methodological pivot with practical consequences, not a paradigm shift. The basic insight — that KV importance is predictable from hidden states — is elegant but not earth-shattering; it is the kind of observation that, once stated, feels almost obvious. What is not obvious, and what the paper establishes through systematic ablation, is that realizing this insight requires getting several non-trivial design choices exactly right: the training target must be reconstruction-based (next-token prediction and QA targets fail to generalize), the gate input must be hidden states without positional encoding (key states and RoPE-encoded features degrade performance), and the architecture must use sink-attention with learnable baseline keys (MLPs and linear models underperform, particularly on retrieval). These are not incremental hyperparameter tweaks — they are design principles that collectively define a new approach to learned KV compression, and the paper's identification of these principles through controlled experiments is its primary intellectual contribution.

The paper resolves a specific contradiction that has plagued the learned KV compression literature. Prior gating-based methods — Locret, TrimKV, DMS — demonstrated that learned importance prediction could work, but only within narrow domains: Locret on QA tasks, TrimKV on mathematical reasoning (with separate models for general language), DMS on programming. The implicit message was that learned KV compression was inherently task-specific — that you had to train on the same kind of data you would evaluate on, and that cross-task generalization was not achievable. Fast KVzip falsifies this by demonstrating a single set of gates, trained on 1M tokens of general web text with a reconstruction objective, that transfers across retrieval, QA, summarization, code comprehension, and mathematical reasoning. This does not mean the task-specific methods were wrong about their own results — it means they were wrong about the source of the task-specificity. The bottleneck was not the gating architecture or the model capacity; it was the training signal. Next-token prediction and QA attention patterns encode task-specific utility (Figure 6), and gates trained on them inherit that specificity. Reconstruction attention patterns encode task-agnostic utility, and gates trained on them generalize. This is a clean, mechanistic explanation that converts a confusing pattern of conflicting results into a coherent picture.

The paper also reorients the field's relationship to KVzip. Before Fast KVzip, KVzip represented a frustrating tradeoff: it achieved the best compression quality in the field, but at a runtime cost that made it impractical for latency-sensitive deployment. The implicit research question was "can we beat KVzip's quality with less overhead?" Fast KVzip answers a different question: "can we match KVzip's quality by paying the overhead once, during training, rather than at every inference?" This is a distillation argument — KVzip is not a competitor but a teacher — and it suggests that future work on KV compression should not treat reconstruction-based methods as baselines to surpass, but as oracles to distill. The consequence is that the research frontier shifts from "design a better compression algorithm" to "design a better distillation pipeline," where the expensive oracle can be made increasingly sophisticated (better reconstruction objectives, larger training corpora, multi-model ensembles) as long as the resulting importance scores can be effectively compressed into a lightweight predictor.

The research directions this paper makes more attractive include: distillation of increasingly sophisticated KV importance oracles into feed-forward predictors; adaptive and dynamic gating mechanisms that adjust compression ratios during inference rather than at fixed intervals; joint optimization of KV compression with other inference-time efficiency techniques (quantization, sparse attention, speculative decoding); and theoretical analysis of what properties of hidden states encode future attention utility. The research directions this paper makes less attractive include: designing new heuristic importance metrics based on attention-pattern analysis (the paper shows these are fragile compared to learned gates); training task-specific compression models (the paper shows this is unnecessary if the training signal is task-agnostic); and investing in ever-more-complex reconstruction procedures for runtime deployment (the paper shows these can be distilled into cheap feed-forward predictors). None of these directions are "dead," but Fast KVzip shifts the burden of proof: a new method must now demonstrate either better compression quality than Fast KVzip at matched overhead, or lower overhead than Fast KVzip at matched quality, and the bar for both is high.

Follow-Up Research This Work Enables

Cross-architecture validation on Llama, Mistral, and DeepSeek families. The paper evaluates exclusively on Qwen and Gemma models — two families from organizations (Alibaba, Google) whose pretraining recipes may share design patterns. The central claim that hidden states encode KV importance in a way that transfers across tasks and models is tested for task diversity but not for architectural diversity. A strong follow-up would replicate the full pipeline on Llama-3-8B, Mistral-7B, and DeepSeek-V2-Lite: run KVzip reconstruction on 1M tokens of FineWeb-Edu (or an equivalent general corpus), train sink-attention gates with identical hyperparameters (S = 16, D' = 16, learning rate 0.2, 5K steps), and evaluate on the same 12 prefill-intensive benchmarks. The key question is whether the 70% eviction threshold transfers — if Llama models require 50% retention for near-lossless performance while Qwen models achieve it at 30%, that reveals architecture-dependent compressibility that the current paper cannot detect. A negative result (gates fail to transfer to non-Qwen architectures) would be equally informative: it would suggest that Fast KVzip's success depends on properties of Qwen's attention patterns (perhaps their pretraining data mixture, their RoPE frequency, or their normalization scheme) that are not universal, narrowing the method's scope substantially.

Dynamic difficulty-adaptive compression ratios. The paper applies a fixed global compression budget (e.g., 30% retention) across all tokens and all contexts. But Figures 11 and 13 show that different tasks tolerate different compression ratios — retrieval tasks maintain accuracy at 20% budget while some QA tasks degrade slightly at 30%. This suggests an opportunity for context-adaptive compression: use the gate's own confidence (the distribution of predicted importance scores) to estimate how compressible the current context is, then adjust the budget dynamically. An easy context where most tokens score near 0 or near 1 (clear separation between important and unimportant) can be compressed more aggressively; a difficult context where scores are uniformly middling (no clear separation) needs a larger budget. The experiment would be: at the end of prefill, compute the entropy or variance of the gate's predicted scores across all KV pairs, use this as a difficulty metric, and select a budget ratio from a pre-computed lookup table (analogous to the difficulty-conditioned compute-optimal policy in the PaLM 2-S* paper). The evaluation would measure whether adaptive budgeting achieves better accuracy-per-average-byte than fixed budgeting across the 12-benchmark suite, and whether the gate's confidence is a reliable difficulty signal. The infrastructure for this experiment exists in the paper (the gates already produce per-token scores; only the budget selection logic needs to be added).

Combining Fast KVzip with KV quantization for multiplicative compression. The paper focuses exclusively on token-level eviction (removing entire KV pairs), but orthogonal compression dimensions — particularly KV quantization (KIVI; Liu et al., 2024b) and feature-channel pruning (Think; Xu et al., 2025) — address different aspects of the KV cache footprint. Token eviction reduces the number of stored KV pairs; quantization reduces the bits per stored value; channel pruning reduces the dimensionality of each stored vector. The combination could be multiplicative: evict 70% of tokens (3.3× reduction) and quantize the remaining values to 2-bit precision (8× reduction from BF16) for a total ~26× compression. The experiment would be: apply Fast KVzip gates at a fixed budget ratio (e.g., 30%), then apply KIVI-style asymmetric quantization to the retained KV cache, measuring both accuracy on the 12-benchmark suite and wall-clock memory/latency on long contexts. The key question is whether the two compression mechanisms interact adversarially — quantized KV values might have different attention properties that change which tokens are important, and gates trained on full-precision KV features might make suboptimal decisions when the retained features are quantized. If the combination works without retraining, it establishes a practical deployment recipe; if it requires gate retraining with quantization-aware targets, it identifies a new research problem.

Training-free gate transfer across models via representation alignment. The paper trains separate gates for each model because the hidden state representations are model-specific — a gate trained on Qwen3-8B hidden states cannot be applied to Qwen2.5-7B-1M because the representations live in different spaces. This is a practical barrier: a new model requires new gate training, which requires running the expensive KVzip reconstruction pipeline. A research direction is to investigate whether gates can transfer across models if the hidden states are first aligned. The experiment would be: take a gate trained on Qwen3-8B, and at inference time, insert a lightweight linear adapter (trained via a small amount of paired hidden-state data from both models on the same text) that maps the target model's hidden states into the source model's representation space before feeding them to the gate. The adapter would be trained with a reconstruction-like objective: minimize the MSE between the source gate's predictions on source hidden states and the adapted gate's predictions on mapped target hidden states, using a small corpus (e.g., 10K tokens) of paired forward passes. The evaluation would measure whether an adapted gate approaches the performance of a natively trained gate, and how the adaptation data requirement scales with the architectural distance between models. A positive result would dramatically reduce the barrier to deploying Fast KVzip on arbitrary models; a negative result would confirm that gate training is inherently model-specific and motivate research into universal importance predictors.

Stress-testing on adversarial long-context scenarios. The paper's evaluation uses standard benchmarks where the base model achieves non-trivial performance. This leaves open the question of how Fast KVzip behaves when the model is pushed to its capability limits — very long contexts (500K+ tokens), highly ambiguous retrieval tasks, or adversarial inputs designed to trick the gate into evicting critical information. A stress-test suite would include: (a) needle-in-haystack with the needle placed at positions that are systematically distant from both the beginning (attention sinks) and the end (local window), testing whether the gate's content-based scoring can identify importance without positional cues; (b) multi-hop reasoning where the critical tokens are distributed across the context with no surface-level signal of their importance (e.g., the answer depends on combining information from three sentences separated by thousands of tokens of irrelevant text); (c) adversarial prompts where the context contains "distractor" passages that are semantically similar to the query but irrelevant, testing whether the gate confuses semantic similarity with importance; (d) context-switching scenarios where the model must process two unrelated documents concatenated, testing whether the gate's scores on the first document are affected by the presence of the second. These stress tests would map the failure modes of the current method and identify which design choices (training target, architecture, local window size) are responsible for robustness or fragility in each scenario.

Practical Applications and Downstream Use Cases

High-throughput batched serving with larger batch sizes. The most immediate practical application of Fast KVzip is in LLM serving systems where GPU memory is the binding constraint on batch size. In a continuous batching serving setup (e.g., vLLM, TensorRT-LLM), the KV cache for all in-flight sequences must fit in GPU HBM alongside the model weights. With full KV caching, a Qwen2.5-14B-1M model processing 100K-token contexts might fit only 2-4 sequences per GPU before running out of memory. Compressing the KV cache to 30% with Fast KVzip (Figure 10a shows ~45 GB peak memory vs. ~80 GB uncompressed at 320K) triples the available KV cache headroom, enabling 3× larger batch sizes and proportionally higher throughput. For a deployment serving thousands of queries per hour, this directly translates to reduced GPU-hours and lower infrastructure costs. The 1% decoding overhead from buffered gating is negligible compared to the throughput gain from larger batches, making this a pure win for throughput-bound deployments. The specific numbers from the paper (Figure 10, 30% budget, Qwen2.5-7B-1M) provide a concrete starting point for capacity planning: if your current deployment is memory-bound at batch size B, Fast KVzip at 30% budget should support approximately 3B.

Long-context document understanding with constrained hardware. Many practical applications — contract review, scientific literature synthesis, repository-level code understanding — require processing documents that are 50K-200K tokens long, but the GPUs available for inference (e.g., A10G with 24 GB, L40S with 48 GB, or even consumer GPUs like RTX 4090) cannot fit both model weights and full KV caches for such contexts. Fast KVzip reduces the KV cache footprint proportionally to the budget ratio, making it possible to run long-context inference on hardware that would otherwise be memory-infeasible. For example, a Qwen2.5-7B-1M model with full 128K-token KV cache requires approximately 32-40 GB of KV cache memory alone (depending on precision and implementation), exceeding the capacity of a 24 GB GPU even with no model weights loaded. At 30% budget, the KV cache drops to ~10-13 GB, fitting comfortably alongside model weights on a 24 GB card. This is not a throughput play — it is an enablement play: Fast KVzip makes long-context inference possible on hardware where it was previously impossible, democratizing access to long-context LLM capabilities for researchers and practitioners without access to 80 GB H100s.

Cost-efficient multi-turn conversational agents with long memory. Multi-turn conversational agents — customer support bots, AI assistants with persistent memory, interactive document Q&A — accumulate KV cache across turns as the conversation history grows. Without compression, the KV cache grows linearly with the total conversation length, eventually constraining either the maximum conversation duration or the maximum number of concurrent users. Fast KVzip enables these agents to maintain a compressed representation of the conversation history, dropping low-importance turns (e.g., pleasantries, redundant information) while retaining critical context (e.g., user preferences stated early in the conversation, factual claims that might be referenced later). The key advantage over simple sliding-window approaches (which only retain recent turns) is that Fast KVzip can retain arbitrary tokens from anywhere in the conversation history based on their content importance, not just their recency. The buffered decoding strategy (128-token buffer) is well-suited to this use case because conversation turns are typically tens to hundreds of tokens, meaning importance updates happen at natural conversation boundaries. A deployment engineer could set the KV budget to accommodate, say, a 30-minute conversation within a fixed memory allocation, and Fast KVzip would automatically manage which parts of the history are preserved.

Long-chain reasoning with memory-constrained decoding. The paper's decoding-intensive results (Figure 13) demonstrate a specific application: mathematical reasoning with chain-of-thought processes that can run to tens of thousands of tokens. The alternative — early stopping of thinking (truncating the reasoning process to fit a memory budget) — causes a drastic performance drop because the model cannot complete its reasoning. Fast KVzip enables the full reasoning process to proceed while compressing the accumulated KV cache, allowing long reasoning chains on hardware with fixed memory allocations. This is particularly relevant for reasoning models (like Qwen3's thinking mode, o1-style models) where the internal chain-of-thought may be 10-50× longer than the final answer. A deployment serving reasoning queries could allocate a fixed 4K-token KV budget (as in Figure 13) and let the model reason for arbitrarily long, with Fast KVzip compressing older reasoning steps to make room for new ones. The paper's result that Fast KVzip achieves near-lossless performance at 4K budget on AIME24 provides a concrete operating point: you can serve reasoning queries with a 4K KV cache and expect accuracy comparable to unlimited memory.

When to Prefer This Method

The paper positions Fast KVzip explicitly against KVzip — not as a quality improvement, but as an efficiency improvement — and against heuristic methods (SnapKV, Expected Attention) as a quality improvement. This yields clear decision rules grounded in the reported results:

• Prefer Fast KVzip over KVzip when you need KVzip-quality compression (near-lossless at 30% budget) but cannot tolerate KVzip's 2× prefill latency overhead. This covers most latency-sensitive production deployments, especially those with high query volumes where the one-time gate training cost (~1 H100 hour per model) is easily amortized. The paper's Figure 10 provides the quantitative justification: at 320K context length, Fast KVzip's prefill time is below the no-compression baseline while KVzip's is ~2× the baseline.

• Prefer Fast KVzip over heuristic methods (SnapKV, Expected Attention, DuoAttention) when your task distribution includes multi-query scenarios (multiple questions per long context), retrieval tasks, or any setting where heuristic sparsity patterns risk discarding information needed for future queries. The paper's Figure 11 shows SnapKV and Expected Attention diverging from full-cache performance at budget ratios of 0.6–0.7, while Fast KVzip holds near-lossless down to 0.3. The gap is largest on retrieval tasks, making Fast KVzip the clear choice for retrieval-augmented generation, document QA with multiple queries, and conversational agents with long memory.

• Prefer Fast KVzip over TrimKV, Locret, or DMS when you need a single compression model that works across diverse tasks without task-specific training. The paper's Table 4 documents that TrimKV requires separate models for math vs. general language; Fast KVzip uses one set of gates for all 14 evaluated benchmarks. If your deployment serves a mix of tasks (some retrieval, some QA, some code), the task-specific methods require either multiple models (with switching logic) or accepting degraded performance on out-of-distribution tasks. Fast KVzip eliminates this operational complexity.

• Consider the no-compression baseline (or KVzip directly) when inference volume is very low and the one-time gate training cost is not amortizable, or when the model is not one of the tested families (Qwen, Gemma) and the practitioner cannot run the KVzip reconstruction pipeline to generate training targets. In these cases, the practical advantage of Fast KVzip — amortizing training cost over many inferences — does not materialize, and the simpler (if slower) approach of running KVzip at inference time, or accepting the full memory cost, may be preferable. The crossover point depends on query volume, model size, and engineering resources; the paper does not quantify this threshold but the training cost numbers in Table 2 (~0.6-0.8 H100 hours) provide a lower bound for the amortization calculation.