Inference Optimization2025~102 minarXiv:2506.05345

Inference-Time Hyper-Scaling with KV Cache Compression

score 9Adrian Łańcucki, Konrad Staniszewski, Piotr Nawrot +1NVIDIA↑ 30Jun 5, 2025cs.LG

🎯 Pitch

Compressing the KV cache instead of retaining all tokens during reasoning can dramatically boost LLM accuracy on math and coding tasks under fixed latency—adding over 12 points on AIME 24 by generating more tokens within the same memory read budget. The key is a new method that delays token eviction during training, preserving reasoning quality even at 8× compression, where standard sparse attention fails.

1. Executive Summary

This paper introduces inference-time hyper-scaling, a framework that uses KV cache compression to expand the effective token budget for reasoning within a fixed compute budget, and proposes Dynamic Memory Sparsification (DMS), a novel trainable token eviction method that achieves high compression ratios with only ~1K training steps while preserving accuracy better than training-free sparse attention baselines. Evaluated on reasoning benchmarks (AIME 24, MATH-500, GPQA Diamond, LiveCodeBench) with Qwen-R1 distilled models (1.5B–32B), DMS achieves average Pareto frontier gains over vanilla models of 12.5 points on AIME 24, 8.6 on GPQA, and 9.7 on LiveCodeBench at equivalent memory reads, with throughput improvements up to 5× over the uncompressed baseline. The paper establishes that KV cache compression enables more favorable inference-time scaling than full KV retention only when the compression method maintains accuracy at high ratios—a requirement that DMS meets through its key design choice of delayed eviction, which preserves reasoning capabilities that immediate eviction otherwise destroys.

2. Context and Motivation

The Core Problem: KV Cache Size Is the Real Bottleneck for Inference-Time Scaling

The paper identifies a fundamental mismatch between how the field conceptualizes inference-time scaling and how Transformer LLMs actually consume compute during generation. The prevailing mental model treats generated tokens as the unit of inference cost: scale sequential reasoning (longer chains) or parallel reasoning (more diverse samples), and cost scales linearly with the total number of tokens produced (Muennighoff et al., 2025; Brown et al., 2024). This framing is natural because it mirrors how we think about training FLOPs—more data means more compute.

However, in auto-regressive Transformers, generation cost is not bottlenecked by the number of generated tokens per se, but by the size of the key–value (KV) cache that accumulates during generation. The KV cache stores the key and value representations for every previously generated token, and at each new generation step, the model must attend to this entire cache. Two critical consequences follow:

Memory exhaustion: The KV cache grows linearly with the total number of tokens generated (the sum of prompt length plus all generated tokens across all parallel reasoning chains). For large models with long context windows deployed on GPUs with finite VRAM, this quickly becomes the limiting factor on how many parallel reasoning threads or how long a chain can be generated. Exceeding GPU memory forces costly offloading or simply prevents scaling beyond a certain point.
Attention becomes memory-bound: The computational cost of attention is dominated not by the FLOPs of computing attention scores, but by the time required to read the KV cache from high-bandwidth memory (HBM) into the compute units. As sequence length grows, this memory transfer time increasingly dominates the latency of each generation step. The paper formalizes this in Appendix I with latency calculations showing that KV cache reads can account for the majority of per-step inference time at longer context lengths and larger batch sizes (Figure 9).

This means the standard inference-time scaling playbook—"generate more tokens via longer chains or more parallel samples"—is actually constrained by how efficiently those tokens can be represented in memory, not by how many tokens the model can produce per unit FLOP. The paper's core insight is that by compressing the KV cache, one can generate more tokens within the same budget of memory reads and peak memory usage, thereby achieving higher reasoning accuracy at equivalent cost. The authors term this hyper-scaling: scaling beyond what vanilla attention's memory footprint would permit.

Why This Problem Matters Now

The timing of this work is driven by the rapid adoption of inference-time scaling as a primary strategy for improving reasoning capabilities (OpenAI o1, DeepSeek R1, Guo et al., 2025). As models are increasingly deployed with "thinking" modes that generate thousands of reasoning tokens before answering, the KV cache bottleneck shifts from a long-context niche concern to a mainstream deployment challenge. The paper makes this concrete in Figure 4: even at moderate batch sizes (32–128 sequences in parallel), the latency curves for vanilla models begin to rise steeply well within the context lengths used for serious reasoning (e.g., 8K–32K tokens). The batch size here represents both the number of parallel reasoning threads per query and the number of concurrent user queries being served, making this directly relevant to production throughput.

The problem is also acute for edge deployment and memory-constrained hardware, where peak KV cache memory can prevent loading larger models or serving multiple parallel reasoning chains altogether. Even on datacenter GPUs (the paper uses NVIDIA H100 SXM), the combination of large batch sizes for throughput and long sequences for reasoning quality pushes against VRAM limits that KV cache compression directly addresses.

A subtler point: the paper's analysis in Figure 4 shows that the context length at which attention becomes memory-bound depends on both model scale and batch size, meaning there is no single threshold where KV cache compression "kicks in" as beneficial. Instead, the benefit is continuous and configuration-dependent, making it a general-purpose optimization rather than a solution only for extreme long-context scenarios.

Where Prior Approaches Fall Short

The paper evaluates three families of existing KV cache efficiency methods and identifies specific limitations that motivate DMS:

Training-free KV cache eviction methods (TOVA, H2O) fail at high compression ratios. These methods use heuristics—such as attention scores (TOVA; Oren et al., 2024) or cumulative attention (H2O; Zhang et al., 2023a)—to decide which tokens to evict from the KV cache during inference. They are attractive because they require no model modification and incur minimal overhead. However, they make static, uniform eviction decisions based on simple signals, and at the compression ratios needed for meaningful hyper-scaling (4×–8×), accuracy degrades severely. This is especially problematic for models using Grouped Query Attention (GQA; Ainslie et al., 2023), where multiple query heads share the same KV heads. With GQA, evicting a KV token affects all query heads in the group simultaneously, amplifying the destructive impact of any single poor eviction decision. The paper's results bear this out: in Table 1, H2O at 4× compression drops from 47.0% to 14.7% on GSM8K (a 32-point loss), and TOVA drops to 20.2%. On long-context tasks like Needle in a Haystack, the collapse is even more dramatic (96.4% → 13.4% for H2O at 4×).

The fundamental issue is that these heuristics are reactive: they decide which tokens to keep based on past attention patterns, but cannot anticipate which tokens will be important for future generation steps. This is a classic exploration–exploitation problem in a sequential decision setting, and simple heuristics systematically make myopic eviction choices that remove information the model later needs.

Quest (selective retrieval without eviction) trades memory savings for accuracy. Quest (Tang et al., 2024) takes a different approach: rather than permanently evicting tokens, it keeps the full KV cache in memory but only retrieves the most relevant pages (fixed-size blocks of contiguous KV items) for each attention computation, using a heuristic based on the highest-magnitude dimensions of key vectors. This avoids the permanent information loss of eviction, which explains why Quest maintains much better accuracy than TOVA/H2O at high compression ratios (Table 1: Quest at 4× retains 39.9% on GSM8K vs. 20.2% for TOVA).

However, Quest has two critical limitations for hyper-scaling:

It does not reduce memory load. The full KV cache stays in HBM, so Quest cannot address the peak memory bottleneck. For hyper-scaling—where the goal is to fit more parallel reasoning chains or longer sequences within a fixed memory budget—this is a dealbreaker.
It actually adds memory overhead. To perform efficient page selection, Quest must store additional page representations (compressed summaries of each page's contents). This means Quest slightly increases rather than decreases total memory usage.

Furthermore, because Quest uses a separate top-k for each query head in GQA models, it can retrieve more distinct KV blocks than necessary (since different query heads in a group may select different pages), potentially increasing memory transfers compared to methods that uniformly compress. The paper notes this implementation detail in Appendix H.1. The consequence is visible in the Pareto frontier comparisons: Quest sometimes overlaps with or trails vanilla models on the memory-reads axis (Table 8: Quest vs. Vanilla shows negative or near-zero improvements on several model–task combinations), meaning its accuracy retention doesn't translate into a better latency–accuracy trade-off in all regimes.

Dynamic Memory Compression (DMC) requires expensive training. DMC (Nawrot et al., 2024) addresses the accuracy problem of eviction by learning a per-head, per-token decision: at each step, the model decides whether to append the new KV pair to the cache (standard behavior) or merge it with the most recent entry via weighted averaging. This is more flexible than eviction because it can preserve information through merging rather than simply discarding it. And because the decisions are learned end-to-end (via stochastic reparameterization with a Gumbel-sigmoid relaxation), DMC achieves much better accuracy than training-free eviction at high compression ratios.

The limitation is data and compute cost. The paper's own experiments (Section 5.3, Figure 5, right panel) show that DMC requires an order of magnitude more training tokens than DMS to reach comparable performance. The original DMC paper reported needing ~44K training steps to reach 8× compression, with performance degrading when data was halved. At the model scales where inference-time scaling is most impactful (7B–32B parameter reasoning models), this training cost can be prohibitive. Moreover, the paper finds (Table 1, Appendix F, Table 4) that DMC can collapse catastrophically when extrapolating to context lengths beyond those seen during retrofitting—on Llama 3.2 1B Instruct, DMC at CR2 drops from 99.0% to 0.0% on Needle in a Haystack when moving from 3K to 4K context length. This extrapolation failure makes DMC fragile for deployment in variable-length settings.

How This Paper Positions Itself

The paper's conceptual contribution is reframing inference-time scaling through the lens of memory efficiency rather than token throughput. The key equation is: for a fixed budget of KV cache memory reads (which drives latency) and peak memory tokens, KV cache compression allows generating more tokens of reasoning than vanilla attention. The paper provides the first systematic investigation of whether this translation actually yields accuracy gains, testing the hypothesis that "by making the KV cache leaner, could we scale the length and number of reasoning threads and enhance the accuracy of existing LLMs for an equivalent compute budget?" (Section 2.1).

The technical contribution, DMS, is positioned as a practical compromise between the two extremes of prior work:

From training-free eviction: DMS adopts the simplicity of token eviction (just remove tokens, don't try to merge them). This avoids the complexity of DMC's weighted averaging and its associated training difficulties.
From DMC: DMS adopts the idea of learning eviction decisions via retrofitting, using the same stochastic reparameterization and auxiliary compression loss. This provides the accuracy benefits of learned decisions over heuristic ones.
Novel to DMS: The key innovation is delayed eviction—the model decides now that a token should be evicted, but the eviction doesn't take effect until a future time step (controlled by a sliding window parameter $w$ , default 256 tokens). This delay gives the model time to extract and integrate information from soon-to-be-evicted tokens, transforming eviction from a destructive operation into one that preserves critical context. The paper shows (Section 5.3, Figure 5, left) that immediate eviction causes rapid accuracy degradation, while delayed eviction with even a modest 16-token window maintains performance.

DMS is thus cheaper to train than DMC (by ~60× in the reasoning model experiments, reaching CR 4× in 300 steps vs. thousands for DMC) while achieving better accuracy than training-free methods. It also naturally extrapolates beyond its training context length (unlike DMC), and its eviction mechanism reduces both memory reads and peak memory (unlike Quest). These properties make DMS suitable as the enabler of inference-time hyper-scaling: a compression method that is cheap enough to apply to existing models, accurate enough to preserve reasoning quality at high compression ratios, and memory-efficient enough to translate compression into genuine budget headroom for longer or more parallel reasoning traces.

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems-and-methods paper that introduces both a framework (inference-time hyper-scaling) and a technique (Dynamic Memory Sparsification, or DMS) for making that framework practical. The system being built is a modified Transformer LLM that learns to selectively evict tokens from its key–value cache during generation, enabling the model to generate longer reasoning chains or more parallel reasoning threads within the same memory and latency budget as an uncompressed model. The core idea is that KV cache compression can be used not just to reduce inference cost for a fixed task, but to reinvest the saved memory and bandwidth into more extensive reasoning, yielding higher accuracy at equivalent compute — but only if the compression method preserves the model's reasoning abilities at high compression ratios.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that interact during both training and inference:

Base LLM (Transformer with GQA) — The pretrained language model (Qwen-R1 distilled, Qwen3, or Llama 3.2 Instruct) that generates reasoning traces. It uses Grouped Query Attention, where multiple query heads share a single set of KV heads.
Eviction Decision Module — A lightweight mechanism added to each attention head that, at every generation step, predicts a binary decision α_t ∈ {0, 1} indicating whether the current token's key–value pair should be marked for future eviction. During training, this is a Gumbel-sigmoid stochastic neuron; during inference, it produces hard binary decisions. Critically, this module reuses an existing neuron from the query vector rather than adding new parameters.
Sliding Window / Delayed Eviction Buffer — A fixed-size window (default 256 tokens) that holds tokens after they are marked for eviction but before they are actually removed from the KV cache. This delay allows the model to attend to and integrate information from soon-to-be-evicted tokens before they disappear.
Compression-Aware Training Pipeline — During retrofitting (continued training of the pretrained LLM), the model is trained with:
- A logit distillation loss against the original (uncompressed) model as teacher
- An auxiliary one-sided L1 loss that pushes the average eviction rate toward a target compression ratio
- A linearly annealed target compression ratio that increases from 1× to the desired CR over the course of training
- A causal attention mask constructed from the continuous eviction decisions α_t that makes marked tokens progressively less visible

Information flows as follows: at each generation step t, the hidden state h_t is projected through a learned linear transformation to produce a continuous eviction score α_t → this score is converted to a binary eviction flag → the flag enters the sliding window buffer → after w steps, flagged tokens are physically evicted from the KV cache → the model attends only to un-evicted tokens and tokens still in the window → the process repeats for the next token.

3.3 Roadmap for the Deep Dive

First, the formal mechanism of eviction decisions: how α_t is computed from hidden states, the Gumbel-sigmoid reparameterization, and why a specific bias initialization (b = -5) matters for training stability.
Second, the delayed eviction via sliding window: how the attention mask M_α is constructed from a sequence of eviction decisions, why immediate eviction fails, and how the window size w controls the tradeoff between compression and accuracy.
Third, the training objective and compression schedule: the two-term loss function (distillation + auxiliary compression loss), the linear annealing of the target compression ratio, and why logit distillation is preferred over standard LM loss.
Fourth, the inference-time mechanism: how continuous decisions are binarized, how the sparse KV cache interacts with PagedAttention, and performance considerations (no extra parameters, no new read/write operations).
Fifth, the hyper-scaling evaluation framework: how configurations are parameterized as W-L-CR tuples (width, length, compression ratio), how budgets are measured in KV cache token reads and peak memory, and how Pareto frontiers are computed to compare methods at equivalent cost.

3.4 Detailed, Sentence-Based Technical Breakdown

Eviction Decisions: From Hidden States to Binary Flags

At the core of DMS is a per-token, per-head binary classifier that decides whether a newly generated token's key–value pair should eventually be evicted from the KV cache. The decision is made at the moment the token is generated, but it does not take effect until later (see Section 3.4 on delayed eviction). This is distinct from training-free eviction methods (TOVA, H2O) which make eviction decisions retroactively based on attention scores of past tokens.

For a given attention head at generation step t, the model receives a hidden state vector h_t ∈ ℝ^d (where d is the model's hidden dimension). This is the same hidden state that feeds into the query, key, and value projections of standard attention. The eviction decision is computed as:

$\alpha_t \sim \text{Gumbel-sigmoid}(\mathbf{h}_t \mathbf{w}^\top + b, \tau)$

where w ∈ ℝ^d is a trainable weight vector, b is a scalar bias, τ is the temperature of the Gumbel-sigmoid distribution, and α_t ∈ [0, 1] is a continuous value representing the "probability" of marking this token for eviction.

Symbol definitions:

h_t: the hidden state vector at time step t (same vector that feeds into query/key/value projections of standard attention)
w: a trainable weight vector of dimension d, initialized to all zeros (w = [0, ..., 0]^⊤)
b: a scalar bias set to -5 to offset the logits and start training with α_t ≈ 0 (preventing early eviction)
τ: the Gumbel-sigmoid temperature parameter, set low to encourage near-binary decisions
α_t: the continuous eviction score in [0, 1], where 1 means "mark for eviction" and 0 means "keep"

What it computes: the raw logit h_t w^⊤ + b is a scalar score for how "evictable" the current token is. This logit passes through the Gumbel-sigmoid function, which adds Gumbel-distributed noise and applies a sigmoid, producing a value in [0, 1]. During training, this is a stochastic relaxation: α_t is continuous and can take intermediate values between 0 and 1. The Gumbel noise provides gradient estimates for what is ultimately a discrete decision, enabling end-to-end training via the reparameterization trick (Louizos et al., 2018). During inference, the noise is removed and the sigmoid is thresholded at 0.5 to produce a hard binary decision.

Why this form: the Gumbel-sigmoid reparameterization is a standard technique for learning discrete stochastic decisions with gradient-based optimization. It converts the non-differentiable sampling of a binary variable into a differentiable operation by separating the randomness (Gumbel noise) from the parameter-dependent part (the sigmoid of the logit). This allows the model to learn which tokens to evict through standard backpropagation. The alternative would be reinforcement learning (e.g., REINFORCE), which has higher variance and is less sample-efficient. The initialization w = 0 and b = -5 ensures that at the start of training, sigmoid(0 - 5) ≈ 0.007, meaning the model initially keeps almost all tokens. This prevents catastrophic forgetting in early training steps — if the model started evicting tokens aggressively before learning which ones are important, it would irreversibly damage its language modeling capabilities. The low temperature τ pushes the Gumbel-sigmoid toward producing values close to 0 or 1 even during training, reducing the gap between training behavior (continuous) and inference behavior (discrete).

Parameter Implementation: Borrowing a Neuron

Rather than adding a new parameter vector w for every attention head (which would increase model size), DMS reuses an existing neuron from the query projection. Specifically, for each group of query heads that share a KV head (in GQA), the first neuron of the first query head is repurposed:

The first dimension of q_{t,first} (the query vector for the first head in the group) is extracted and treated as the logit for α_t.
During the initial phase of continued training (first 2000 steps), this dimension is gradually zeroed out: q_{t,first}[0] ← q_{t,first}[0] × (1 - t/2000), where t is the current training step.
After this warm-up phase, the model enters the main DMS retrofitting phase, where it learns to use this repurposed neuron for eviction decisions.
During both training and inference, q_{t,first}[0] is set to 0 for the actual attention computation, so the eviction signal does not contaminate the attention scores.

What this computes: a zero-parameter mechanism for producing per-head eviction decisions. The original query projection W_q already maps h_t to a d_k-dimensional query vector; the first dimension of this projection effectively becomes the w in the eviction equation, but it is a row of an existing weight matrix rather than a new parameter. The gradual zeroing out during warm-up prevents a sudden change in the query representation that could destabilize training.

Why this form: the paper explicitly compares this to DMC's approach, which extracts α_t from key representations (affecting all query heads in a group identically). By extracting from a query head instead, DMS allows per-head eviction decisions even within a GQA group, providing more fine-grained control. The authors also note (Appendix C) that one could alternatively use a "separate, trainable, zero-initialized projection from the hidden state to extract α_t, effectively eliminating the need for continued training" — the query-neuron approach is chosen because it requires no additional parameters and minimal continued training, but the separate-projection alternative is available if even cheaper retrofitting is desired.

Constructing the Attention Mask from Eviction Decisions

During training, a sequence of eviction decisions α_{1:T} for a sequence of length T is used to construct an additive mask M_α ∈ (-∞, 0]^{T×T} that is added to the unnormalized attention scores QK^⊤ before the softmax. This mask implements the delayed eviction mechanism in a fully differentiable way.

For attention head positions (i, j) where i is the query position (current token attending) and j is the key position (past token being attended to), the mask is constructed as follows:

Causal mask positions (where j > i, i.e., future tokens): set to -∞, same as standard causal attention.
Within sliding window (where i - j ≤ w, i.e., recent tokens): set to 0, meaning no masking — these tokens are fully visible regardless of their eviction decision. This implements the "delay" because even tokens marked for eviction remain accessible while they are within the window.
Outside sliding window (where i - j > w, i.e., tokens that have aged out of the window): set to log(1 - α_j). This means:
- If α_j = 0 (token not marked for eviction): log(1 - 0) = 0, so the token remains fully visible.
- If α_j = 1 (token marked for eviction): log(1 - 1) = log(0) = -∞, so the token is completely masked (evicted).
- If α_j is intermediate (during training): the token is partially masked, with the degree of masking proportional to α_j.

What it computes: a visibility schedule for every past token from the perspective of every current token. Tokens are fully visible for w steps after generation, regardless of their eviction status. After w steps, their visibility is determined by their eviction decision: kept tokens remain at full visibility, evicted tokens become completely invisible (+ -∞ attention). During training, the continuous α_j values create a smooth transition from visible to invisible, providing gradient signal for the eviction decisions.

Why this form: the log(1 - α) formulation has a specific mathematical property: when the mask is added to attention logits and passed through softmax, a mask value of log(1 - α) on token j multiplies that token's attention weight by (1 - α_j) relative to what it would be with no masking. This means α_j directly controls the proportion by which token j's influence is reduced. The sliding window (positions 0 to w set to mask value 0) is the mechanism that implements delayed eviction: even tokens with α_j = 1 remain visible for w steps, giving the model time to extract their information. The authors emphasize (Section 5.3) that without this delay — i.e., setting mask positions to log(1 - α_j) immediately — accuracy degrades rapidly. The sliding window is therefore not just a detail but the central mechanism that makes learned eviction viable.

The mask is never materialized as a full T×T matrix. As noted in Section 3.1 (Performance Considerations), it can be compactly passed as a vector of eviction decisions α_{1:T} and applied efficiently using existing sparse attention kernels.

Training Objective: Distillation Plus Compression Loss

DMS is trained with a two-term objective that balances accuracy preservation with compression:

$\mathcal{L} = \mathcal{L}_D + \mathcal{L}_{\text{aux}}$

Symbol definitions:

L_D: the logit distillation loss — the KL divergence between the DMS model's output logits and the original (uncompressed) teacher model's output logits
L_aux: the auxiliary compression loss, defined below

The distillation loss L_D follows Hinton et al. (2015): the original pretrained LLM acts as the teacher, and the DMS-retrofitted model as the student. At each training step, both models process the same input, and the student is trained to match the teacher's output distribution over the vocabulary. This is in contrast to DMC (Nawrot et al., 2024), which uses standard language modeling loss (cross-entropy against ground-truth next tokens).

Why distillation over LM loss: the paper provides two justifications (Section 4). First, the original data mixtures used to train the base models are rarely public, making it impossible to replicate the exact pretraining distribution. Distillation only requires unlabeled text (any distribution can be used), since the teacher provides the target. Second, and more importantly for this work's focus, the authors found that logit distillation "provides greater robustness to shifts in training data" and "is especially beneficial for fragile LLMs with lower parameter counts." Empirical evidence: prior work on model compression (Sreenivas et al., 2024; Minixhofer et al., 2025) has observed that distillation preserves capabilities better than training from scratch on proxy data when the original data distribution is unavailable.

The auxiliary compression loss L_aux is a one-sided L1 penalty that encourages the model to achieve a target average compression ratio:

$\mathcal{L}_{\text{aux}} = \max\left(\alpha^* L H T - \sum_{l \in L} \sum_{h \in H} \sum_{t \in T} \alpha_{lht}, 0\right)$

Symbol definitions:

α^*: the target compression rate, defined as (1 - 1/CR) where CR is the target compression ratio. For 4× compression, α^* = 0.75 (75% of tokens should be evicted); for 8×, α^* = 0.875.
L: number of transformer layers
H: number of KV attention heads per layer
T: sequence length
α_{lht}: the eviction decision for layer l, head h, token position t
The sum ∑_{l,h,t} α_{lht}: total "amount of eviction" across all layers, heads, and positions

What it computes: the total eviction mass the model would need to produce to achieve exactly the target compression ratio is α^* × L × H × T (this is the total number of tokens that should be marked for eviction if every head evicted at the target rate). The loss is the shortfall between this target and the actual sum of all α values. The max(..., 0) makes it one-sided: the model is penalized for evicting too little (not meeting the compression target), but NOT penalized for evicting too much (exceeding the target). The result is a non-negative scalar that is zero when the average eviction rate meets or exceeds α^*.

Why this form: the one-sidedness is crucial. If the loss were symmetric (e.g., |target - actual|), the model would be penalized for compressing more than the target, which would prevent individual heads from specializing to higher compression ratios. This allows heterogeneity: some heads can compress heavily (e.g., 90% eviction) while others compress lightly or not at all, as long as the average meets the target. This is consistent with the observation (Figure 6, right) that early layers are compressed less than later layers, suggesting the model naturally learns to preserve more tokens in early layers where information is less redundant. The L1 form (rather than L2) provides a constant gradient regardless of the size of the shortfall, which the authors find produces more stable training. This loss is identical to the one used in DMC (Nawrot et al., 2024).

The distillation loss and auxiliary loss are combined without a weighting hyperparameter (i.e., equal weight of 1.0 each). The paper does not discuss tuning this balance, suggesting that equal weighting works robustly across model scales.

Compression Ratio Annealing Schedule

The target compression ratio is not set to its final value at the start of training. Instead, it is linearly annealed from 1× (no compression) to the desired CR over the course of training:

$\text{CR}(t) = \frac{t}{100} + 1$

where t is the training step counter. This means:

At step 0: CR = 1× (no compression, α^* = 0)
At step 100: CR = 2× (α^* = 0.5)
At step 300: CR = 4× (α^* = 0.75)
At step 700: CR = 8× (α^* = 0.875)

What it computes: a schedule that gradually increases the compression pressure on the model over the course of training. The auxiliary loss L_aux uses the α^* corresponding to the current CR at each step.

Why this form: the linear schedule in CR (not in α^*) means the compression target increases by 1× every 100 steps. This produces a family of models at different compression ratios from a single training run — at any point, one can extract a model checkpoint with the CR achieved so far. The authors note this explicitly: "Crucially, annealing the CR generates a family of models with different compression ratios from a single retrofitting run." The specific choice of 100 steps per unit of CR was determined empirically: the paper states that "The amount of required data depends directly on the context length of retrofitted models and the target compression ratio: higher ratios necessitate larger datasets." The 100-step rate was sufficient for the models tested (up to 32B parameters) but might need adjustment for larger models or longer contexts. Notably, unlike DMC, DMS omits the third phase of training with a fixed compression ratio, as the authors found it "provided negligible benefits for DMS."

Training Hyperparameters and Data

The training configuration varies by model family, as detailed in Table 3. Key settings:

Batch size: 1024 for most models (following the original Llama recipe; Touvron et al., 2023), reduced to 256 for Qwen3-8B (due to its 32K context length requiring more memory per sample).
Context length: 4096 tokens for Llama 3.2 1B Instruct, 8192 for Qwen-R1 distilled models (to accommodate the longer sequences in AIME and MATH-500 reasoning traces), 32768 for Qwen3-8B.
Learning rate: 1e-5 (for 1B models) to 3e-5 (for 7B and 32B models), using AdamW optimizer.
Precision: bfloat16 for model parameters, FP32 for optimizer states (standard mixed-precision training).
Training data for Qwen-R1 models: the OpenR1-Math-220k dataset, consisting of high-quality reasoning traces sampled from DeepSeek R1, with additional filtering using Math-Verify to retain only traces that reach the correct mathematical solution. Logit distillation uses these traces as input; the teacher model processes the same input to produce target logits.
Training data for Llama 3.2 1B instruct: a curated set of programming examples (Python, C, C++) plus synthetic data generated by prompting the model itself on linear algebra problems from the DeepMind mathematics dataset. Notably, this synthetic data is NOT filtered for correctness (unlike the Qwen-R1 data).
Hardware: NVIDIA H100 GPUs using Megatron-LM framework. GPU hours range from 10 (Llama 1B, one CR unit) to 345 (Qwen-R1 32B, one CR unit).

The authors note that retrofitting cost scales primarily with model size and context length. A full run from CR 1× to CR 8× requires 700 training steps, which takes approximately 70 GPU hours for Llama 1B, 525 GPU hours for Qwen-R1 7B, and 2,415 GPU hours for Qwen-R1 32B (7 × the per-unit cost in Table 3).

Delayed Eviction: The Central Mechanism

The paper's ablation (Section 5.3, Figure 5 left) directly compares two eviction timing policies:

Immediate eviction: at time step t + w, the decision α_{t+w} determines whether the past token at position t is evicted. In other words, the eviction decision is made after the token has been in the cache for w steps, based on the model's state at the time of potential eviction. This is closer to how training-free methods (TOVA, H2O) work: they look at the current state and decide which past tokens to remove.

Delayed eviction (DMS default): at time step t, the decision α_t determines whether the token just generated at position t will be evicted at future time t + w. The decision is made before the token's information has been used by future tokens, and the token remains accessible for w steps to allow information extraction.

What it computes: in both cases, a token survives for at most w steps in the KV cache after being marked (or after the decision is made to evaluate it). The difference is when the decision is made relative to the token's generation. In delayed eviction, the model immediately assesses the token's future importance and signals intent; in immediate eviction, the model waits and assesses past tokens' current relevance.

Why delayed eviction works better: the paper's empirical result is stark: with a sliding window of only 16 tokens, delayed eviction maintains accuracy up to 4× compression on GSM8K (47.0% → 41.0% for vanilla → DMS at 4×), while immediate eviction collapses regardless of window size (Figure 5 left). The authors hypothesize (Section 3.1) that this is because "decoder-only models heavily attend to recent tokens" (citing Xiao et al., 2024; Jiang et al., 2024). Delayed eviction allows the model to anticipate which tokens will become important and give future steps a chance to attend to them before removal. Immediate eviction forces the model to judge a token's importance only after the window has passed, by which point critical information may already be lost if the token was needed by intermediate steps. In essence, delayed eviction makes the model proactive (planning ahead for what to keep) rather than reactive (desperately trying to preserve what's left).

An additional subtlety: the paper reports that "DMS retains reasoning capabilities with a window of only 16 tokens up to a compression ratio of 4×" (Section 5.3). This means the information extraction window can be surprisingly short — the model doesn't need hundreds of tokens of access to each evicted token to preserve its reasoning. This efficiency is what enables aggressive compression without accuracy loss.

DMS During Inference

At inference time, the DMS mechanism operates as follows (Section 3.2, Figure 2a):

At each generation step t, the hidden state h_t is projected through the learned weight vector w (or the repurposed query neuron) to produce a logit.
The logit is passed through a sigmoid and thresholded: α_t^{bin} = ⌊sigmoid(h_t w^⊤ + b)⌋ ∈ {0, 1}. The Gumbel noise is removed; decisions are hard binary.
If α_t^{bin} = 1, the key–value pair (k_t, v_t) is marked for eviction. It is inserted into the KV cache normally (so the current and immediately subsequent tokens can attend to it), but scheduled for removal at step t + w.
At step t + w, any token marked for eviction at step t is physically removed from the KV cache. Under the assumption that KV cache entries are stored with positional metadata, this removal can be implemented by simply allowing the storage for that position to be overwritten by an incoming token — no explicit deletion operation is needed.
The sparsified KV cache is stored using PagedAttention (Kwon et al., 2023), which allocates memory in fixed-size pages. Individual attention heads can have different effective sequence lengths because they evict different numbers of tokens, so pages are allocated per-head.

Performance considerations (Section 3.1, Section 3.2): the authors make several claims about inference efficiency:

DMS introduces no new read/write operations on the KV cache beyond what standard attention already performs. Evicted tokens are simply overwritten; there is no compaction or garbage collection needed.
The attention mask M_α is never materialized as a full matrix. It is passed as a compact vector of eviction decisions and applied using existing sparse attention kernels (FlexAttention; Dong et al., 2024, and FlashMask; Wang et al., 2025a).
During the prefilling phase (the initial forward pass on the prompt), DMS can additionally leverage sparsity to avoid computing attention for tokens that will be evicted, using techniques from MInference (Jiang et al., 2024) or similar sparse prefilling methods.
Because evicted tokens are overwritten rather than explicitly deleted, the peak memory usage is reduced: only non-evicted tokens plus tokens in the sliding window occupy KV cache memory at any time. For a target compression ratio CR, the effective memory usage is approximately 1/CR × total tokens + w (the sliding window overhead), rather than total tokens as in vanilla attention.

Sparse Attention Mask: Training vs. Inference Gap

A key implementation detail (Section 3.1) is how the continuous training mask (values in (-∞, 0]) becomes discrete inference behavior (hard eviction). During training, the mask values log(1 - α_j) create a soft, differentiable transition. Tokens with intermediate α_j values are partially attended to, providing gradient signal that pushes α_j toward either 0 or 1 (since the compression loss penalizes the average, and the low Gumbel-sigmoid temperature encourages polarization). By the end of training, most α_j values are close to 0 or 1, minimizing the gap between training behavior (continuous soft masking) and inference behavior (hard binary eviction). The low temperature τ in the Gumbel-sigmoid is the key knob controlling this gap: lower τ means sharper decisions during training, closer to inference behavior, but potentially noisier gradients. The paper uses a fixed low τ (value not explicitly specified in the main text or appendix) that empirically balances these concerns.

Inference-Time Hyper-Scaling Evaluation Framework

The evaluation framework (Section 5.1) is designed to test whether KV cache compression enables better reasoning at equivalent cost. The key conceptual move is defining "cost" not in tokens generated, but in memory operations:

Primary efficiency metrics:

KV cache token reads: the total number of KV cache entries that are loaded from HBM and attended to, summed across all generation steps and all attention heads. This is an "implementation-agnostic proxy for attention compute" because memory bandwidth, not FLOPs, is the bottleneck. For a vanilla model generating L tokens with H KV heads across n layers, each step reads L × H × n entries; at compression ratio CR, this becomes approximately (L/CR + w) × H × n for DMS.
Peak tokens in memory: the maximum size of the KV cache (in number of tokens) at any point during generation. This determines whether the configuration fits in GPU VRAM.
Throughput: measured as tokens per second at the maximum batch size that fits in memory for each configuration, using an NVIDIA H100 SXM GPU with a Hugging Face Transformers + FlashAttention implementation.

Configuration space: each experiment is parameterized as a tuple W-L-CR:

W (width): number of parallel reasoning chains generated (parallel scaling)
L (length): maximum sequence length per chain (sequential scaling)
CR (compression ratio): the DMS compression ratio (1× = vanilla model)

The total token budget (conceptually) is W × L, but the actual compute cost is determined by how densely those tokens occupy the KV cache. A configuration of 4-8K-4× (4 parallel chains of 8K tokens with 4× compression) uses approximately the same KV cache reads as 1-8K-1× (one chain of 8K tokens uncompressed), but generates 4× more reasoning traces.

Pareto frontier computation: for each method (Vanilla, DMS, TOVA, Quest, H2O, DMC), the set of all W-L-CR configurations is plotted with accuracy on the y-axis and the efficiency metric (reads, memory, or throughput) on the x-axis. The Pareto frontier is the set of points for which no other configuration of the same method achieves both higher accuracy and better efficiency. When comparing methods, the paper computes the average improvement as the integrated difference between their Pareto frontiers over the largest common budget interval (Appendix G):

$\text{Improvement}_{A \text{ over } B} = \frac{\int_{x \in I} (A(x) - B(x)) dx}{|I|}$

where I is the shared budget interval, A(x) is the best accuracy of method A at budget x, and B(x) is the best accuracy of method B at budget x. Linear interpolation is used for budget values where direct measurements are not available. This metric captures the average accuracy gain across the entire range where the methods can be compared, rather than just the maximum gain.

Why this framework: it directly tests the hyper-scaling hypothesis: if compressing the KV cache by CR× frees up capacity to generate CR× more tokens (either longer chains or more parallel ones), then a compressed model should achieve higher accuracy than an uncompressed model at the same memory cost — but only if the compression preserves reasoning quality. The multiple budget metrics (reads, memory, throughput) disentangle different aspects of efficiency: reads test latency, memory tests deployability on constrained hardware, and throughput tests production serving scenarios.

4. Key Insights and Innovations

Innovation 1: Reframing Inference-Time Scaling as a Memory Bandwidth Problem, Not a Token Budget Problem

The paper's most foundational conceptual move is disentangling two quantities that prior work implicitly conflated: the number of tokens generated during reasoning and the actual compute cost of generating them. The dominant mental model in inference-time scaling research (Snell et al., 2024; Muennighoff et al., 2025; Brown et al., 2024) treats generated tokens as the unit of inference compute — scale up parallel samples or sequence length, and cost scales proportionally. This framing is natural because it mirrors how we think about training FLOPs, and it leads to optimization strategies organized around token allocation: how many tokens to spend, whether to spend them sequentially or in parallel, and how to allocate them adaptively by problem difficulty.

This paper argues that this framing is wrong at the hardware level for auto-regressive Transformer generation. The real bottleneck is not the FLOPs of producing tokens but the memory bandwidth consumed by reading the growing KV cache from HBM at each generation step. The paper makes this argument concrete through the latency analysis in Appendix I (formalized in Equations 2–6), showing that KV cache reads dominate per-step latency once sequence length and batch size cross modest thresholds (Figure 9). This means that the relationship between "tokens generated" and "compute consumed" is non-linear in a way that standard scaling analyses miss: generating 4× more tokens costs more than 4× the latency because the per-token cost itself grows with sequence length (since each new token must attend to a larger cache).

From this diagnostic, the paper derives a prescriptive reframing: inference-time scaling should be optimized over memory operations (KV cache reads, peak memory tokens), not over generated tokens. The insight isn't that KV caches are large — that's well-known — but rather that reasoning quality per unit memory bandwidth is the right optimization objective, and that compressing the KV cache enables expanding the effective token budget within a fixed memory bandwidth budget. This is what the paper terms "hyper-scaling": scaling beyond what vanilla attention's linear cache growth would permit at equivalent cost.

This reframing has substantial implications beyond this paper. It suggests that future work on inference-time scaling should report accuracy against memory bandwidth metrics (not just token counts or wall-clock time on particular hardware), because token counts obscure the non-linear relationship between generation and cost. It also implies that KV cache compression methods should be evaluated not on their ability to preserve accuracy at fixed output length, but on their ability to shift the Pareto frontier of accuracy vs. memory reads — which is precisely the evaluation framework the paper introduces (Figures 3, 7; Appendix G).

The evidence for why this reframing matters is in the Pareto frontier comparisons themselves. The paper shows that Quest — which preserves the full KV cache and only selectively retrieves from it — achieves reasonable accuracy at high compression-equivalent retrieval rates, but its Pareto frontier on the memory-reads axis is often worse than or equivalent to vanilla models (Table 8: Quest vs. Vanilla shows near-zero or negative average improvements on AIME 24 and MATH-500 for several model sizes). This is a direct consequence of Quest not reducing memory load: it trades away the memory savings that hyper-scaling requires. The framing makes this limitation visible in a way that accuracy-at-fixed-budget comparisons would not.

Innovation 2: Delayed Eviction as a Learned Anticipatory Mechanism

The paper's central technical innovation is not that KV cache tokens can be evicted (TOVA, H2O, and others already do this), nor that eviction decisions can be learned (DMC learns to merge/append, which is a superset of eviction). The innovation is the specific temporal structure of the eviction mechanism: the model decides now to evict a token later, creating a window during which the rest of the model can extract information from the soon-to-be-discarded representation before it disappears.

This is conceptually distinct from all prior approaches:

Training-free eviction methods (TOVA, H2O) make eviction decisions retrospectively: when the cache is full, they look at past tokens and decide which to remove based on historical attention scores. These decisions are reactive and myopic — they cannot anticipate which tokens will be needed for future computation.
DMC makes decisions immediately about whether to merge or append, but the decision takes effect right away (in the attention mask for subsequent steps). There is no explicit delay between the decision and its consequence.
Standard sliding window attention (e.g., StreamingLLM; Xiao et al., 2024) keeps a fixed window of recent tokens and discards everything older, with no learned per-token selectivity.

DMS's delayed eviction creates a proactive, anticipatory mechanism: the model signals at token generation time whether that token is likely to be important enough to keep, and then the sliding window provides a fixed grace period during which all future tokens can attend to it regardless of the eviction flag. This transforms eviction from a lossy compression-after-the-fact into a scheduled information transfer: the model has w steps to read and integrate whatever it needs from the token before the scheduled deletion occurs.

The significance of this design is supported by the ablation in Figure 5 (left). When the same learned eviction mechanism is implemented with immediate eviction (the decision is made at the time of eviction rather than at token generation), accuracy collapses regardless of window size. This is not the model simply needing more training — it's evidence that the anticipatory timing matters qualitatively. The model benefits from making the decision before seeing how the token will be used by subsequent steps, which forces it to learn a forward-looking assessment of token importance rather than a retrospective one.

Why does anticipatory eviction work better? The paper hypothesizes (Section 3.1) that decoder-only models "heavily attend to recent tokens" (citing Xiao et al., 2024; Jiang et al., 2024), but this explanation is incomplete. A deeper interpretation: by the time a token reaches the eviction boundary in immediate eviction, the model has already had w steps to attend to it — the information extraction has already happened or failed. The decision at that point is about whether to keep extracting beyond w steps, which is a harder problem because it requires predicting future information needs from a single token's residual utility. In delayed eviction, the decision is made at token generation time based on the token's intrinsic properties (is this token likely to contain information that future reasoning steps will need?), which is a more learnable signal because it depends on the token's content rather than on the complex dynamics of future attention patterns.

This conceptual distinction — proactive vs. reactive compression — is a genuine contribution to the design space of learned KV cache management. It's not merely a hyperparameter choice (window size) but a fundamentally different information flow that makes the learning problem easier.

Innovation 3: Empirical Proof That Sparse Attention Alone Suffices for High-Ratio Compression

Prior to this work, the dominant assumption in learned KV cache compression was that merging is necessary for high compression ratios. DMC (Nawrot et al., 2024) explicitly argued for weighted averaging of tokens over simple eviction, on the grounds that eviction permanently discards information while merging can preserve a summary of evicted tokens' content. This argument is intuitive: if you're compressing 8×, surely you need to retain some representation of the discarded 87.5% of tokens, and merging provides that through weighted averaging.

DMS provides a strong empirical counter-argument: sparse attention (binary keep/evict decisions) is sufficient to achieve 8× compression with minimal accuracy loss, provided the decisions are learned and temporally structured (delayed). The evidence:

Table 1: DMS at 4× compression maintains 41.0% on GSM8K vs. 47.0% for vanilla (a 6-point drop), while DMC at 4× drops to 3.6% — a catastrophic 43-point collapse. On Needle in a Haystack at CR2, DMS achieves 97.8% vs. DMC's 0.0% (Table 4).
Figure 3 and associated tables (10–12): DMS at 4× and 8× compression places points on or near the Pareto frontier across AIME 24, MATH-500, GPQA Diamond, and LiveCodeBench for models from 1.5B to 32B parameters.
Table 2: DMS 8× on Qwen3-8B scores within 1.2 points of vanilla on GPQA Diamond (57.6 vs. 58.8) and within 2 points on AIME 2024 (73.0 vs. 75.0), while enabling 5× higher throughput (Figure 8).

This is not an incremental improvement over DMC — it's a fundamental finding about what KV cache compression requires. The implication is that the hard part of KV cache compression is deciding which tokens to keep, not representing the ones you discard. A binary keep/evict decision, when made well (via learned anticipatory decisions), preserves enough information that the discarded tokens' content doesn't need to be summarized. The sliding window provides sufficient temporal overlap that the model can extract what it needs from a token before eviction, and the per-head heterogeneity (different heads evict different tokens) provides sufficient representational diversity that no single important signal is lost across all heads simultaneously.

This finding has substantial practical implications: training a model to make binary eviction decisions (DMS) is much cheaper than training it to make continuous merging decisions (DMC). The paper quantifies this in Figure 5 (right): DMS achieves higher scores than DMC while using 8× fewer training tokens, and in practice the reasoning models were trained with 60× less data than DMC would require (300 steps to CR4 vs. thousands). This data efficiency difference is not just a training artifact — it reflects the relative difficulty of the two learning problems. Learning which tokens to keep requires distinguishing important from unimportant content, which is a relatively well-defined classification problem. Learning how to merge tokens requires learning a continuous compression function that preserves arbitrary downstream information, which is a much harder generative modeling problem.

The negative result for DMC on Llama 3.2 1B Instruct (Table 1) is also informative: the paper notes that DMC's collapse may be due to "its more challenging training objective amplified by the limited 1B model capacity." This suggests that binary eviction is not just cheaper but also more robust to model scale — smaller models can learn to decide what to keep more easily than they can learn to merge representations.

Innovation 4: The Sliding Window as a Learnable Buffer, Not a Fixed Architectural Constraint

The paper repurposes the sliding window — a well-known mechanism in efficient attention (Xiao et al., 2024; Jiang et al., 2024; Oren et al., 2024) — into something conceptually different from its typical use. In standard sliding window attention, the window is the mechanism of compression: all tokens outside the window are discarded, and only tokens inside the window are retained. The window size directly determines the compression ratio, and the model has no say in which tokens survive beyond the window boundary.

In DMS, the sliding window serves a fundamentally different role: it is a fixed-capacity information transfer buffer that decouples the decision to evict from the execution of eviction. The compression ratio is determined by the learned eviction decisions α_t, not by the window size. The window exists only to provide a grace period during which soon-to-be-evicted tokens remain accessible. This means the window can be small relative to the total sequence length — the paper shows that a window of only 16 tokens suffices for Llama 3.2 1B Instruct at 4× compression (Figure 5, left), and uses a window of 256 tokens as the default for the reasoning models.

This repurposing matters because it changes what the window size parameter controls. In standard sliding window attention, increasing the window size directly reduces compression (more tokens kept), creating a direct accuracy-efficiency tradeoff. In DMS, the window size controls the quality of information extraction before eviction: larger windows give the model more time to attend to evicted tokens, potentially improving accuracy, but they don't fundamentally change how many tokens are eventually kept (which is controlled by the learned α_t and the target CR in the auxiliary loss). This decoupling means DMS can achieve high compression ratios with modest window sizes — the model learns to use the brief window efficiently rather than relying on a large window as a crutch.

The evidence for the window's distinct role is visible in the extrapolation behavior (Appendix F, Table 4). DMS models can extrapolate to context lengths beyond their training length (e.g., 8K evaluation on a 4K-trained model for NIAH at CR2, achieving 99.4% vs. DMC's 0.0%), but only up to a point — at CR3, the 8K NIAH score drops to 24.2% for the same model. This pattern suggests that the learned eviction policy generalizes across context lengths (the model correctly identifies evictable tokens regardless of total sequence length), but the fixed window size becomes a bottleneck when the sequence is long enough that the ratio of window-to-sequence becomes too small. This is a qualitatively different failure mode from DMC, which collapses entirely beyond its training context length regardless of CR, and from training-free methods, which degrade uniformly with higher CR regardless of context length.

This reconceptualization of the sliding window — from a compression mechanism to a scheduled information transfer buffer — is a small but genuine conceptual contribution. It shows that the two functions that are typically conflated in sliding window architectures (deciding what to keep and how long to keep it) can and should be separated, with the former learned and the latter fixed.

5. Experimental Analysis

Evaluation Methodology

Datasets. The primary evaluation uses four reasoning benchmarks: AIME 24 (30 competition-level math problems), MATH-500 (500 competition-level math problems; Hendrycks et al., 2021b), GPQA Diamond (graduate-level multiple-choice questions in physics, chemistry, and biology; Rein et al., 2024), and LiveCodeBench (coding problems from 2024-08-01 to 2025-01-31; Jain et al., 2025). For general-purpose evaluation (Tables 1 and 2), additional datasets include GSM8K (grade-school math; Cobbe et al., 2021), MMLU (factuality; Hendrycks et al., 2021a), HellaSwag (common-sense QA; Zellers et al., 2019), Needle in a Haystack (NIAH; Kamradt, 2023), Variable Tracking (VT; Hsieh et al., 2024), MMLU-Pro (Wang et al., 2024b), HumanEval (coding; Chen et al., 2021), IFEval (instruction following; Zhou et al., 2023), and ArenaHard v0.1 (conversation; Li et al., 2025).
Base models. The primary models are Qwen 2.5 distilled from DeepSeek R1 at three scales: 1.5B, 7B, and 32B parameters (Guo et al., 2025), all using Grouped Query Attention (GQA). Additionally, Qwen3-8B distilled from Qwen3-235B-A22B (Yang et al., 2025) is evaluated for broader capability assessment, and Llama 3.2 1B Instruct (Grattafiori et al., 2024) serves as a non-reasoning baseline for method ablations and general-purpose evaluation. These models span 1B to 32B parameters and include both reasoning-specialized (R1-distilled) and general-purpose instruction-tuned variants, covering the regime where inference-time scaling is most impactful.
Metrics. For MATH-500 and AIME 24, accuracy is measured via exact match after mapping outputs to a unified math representation (using MathVerify; Kydlíček and Gendenberger, 2025). For GPQA Diamond, accuracy is the fraction of questions where the model selects the correct choice among four options. For LiveCodeBench, the metric is pass@all — a success is counted if any generated sequence passes the provided unit tests. For general-purpose benchmarks (GSM8K, MMLU, HellaSwag, NIAH, VT), standard accuracy metrics from the Language Model Evaluation Harness (Gao et al., 2024) are used. The primary efficiency metrics are: (i) KV-cache token reads, defined as the total number of items in the KV cache attended to at each generation step, summed across steps (an implementation-agnostic proxy for attention compute, since loading the KV cache from memory dominates latency at long sequences); (ii) peak tokens in memory, the maximum KV-cache size during generation; and (iii) throughput, measured as tokens per second at the maximum batch size that fits in GPU memory for each configuration.
Baselines. The paper compares against: (i) Vanilla models (no KV cache compression); (ii) TOVA (Token Omission via Attention; Oren et al., 2024), a training-free method that evicts tokens with the lowest attention weight within a sliding window; (iii) H2O (Heavy-Hitter Oracle; Zhang et al., 2023a), which evicts tokens based on cumulative attention scores, keeping a separate recent-token window; (iv) Quest (Tang et al., 2024), which retains the full KV cache but selectively retrieves only the most relevant pages using a heuristic based on high-magnitude key dimensions; and (v) DMC (Dynamic Memory Compression; Nawrot et al., 2024), which learns per-head decisions to either append or merge tokens via weighted averaging, requiring substantial continued training.
Generation budget and compute accounting. Each experiment is parameterized as a tuple W-L-CR: W is the number of parallel reasoning chains (width), L is the maximum sequence length per chain, and CR is the compression ratio (1× for vanilla, 4× or 8× for compressed models). The compute budget for comparison is not measured in generated tokens but in the efficiency metrics listed above (KV-cache token reads, peak memory, throughput). This is the core conceptual move: two configurations with different W-L-CR tuples can have the same memory footprint (e.g., 4-8K-4× has similar peak memory to 1-8K-1×), and the comparison is made at equivalent cost. For training-free baselines (TOVA, H2O, Quest), the KV-budget (maximum cache size) is set to (input_length + max_generation_length) / CR. For H2O, this budget is evenly split between recent tokens and heavy-hitter tokens. Quest is permitted full dense attention during prefilling and uses a block size of max(16, 2 × CR), with a separate top-k for each query head, giving it an advantage over other methods in terms of retrieval flexibility but no memory reduction.
Cross-validation and statistical protocol. For AIME 24, results are averaged over 10 runs with different random seeds; for MATH-500, over 3 runs. For the general-purpose benchmarks with Llama 3.2 1B Instruct (Table 1), standard errors computed by the Language Model Evaluation Harness are reported in Appendix G, Table 6. Pareto frontier improvements (Tables 8 and 9) are computed as the average integrated difference between methods' Pareto frontiers over the largest common budget interval, using linear interpolation for unmeasured budget values. Confidence intervals for the Pareto frontier metrics are not reported, which is a limitation of the current analysis.

Main Quantitative Results

Inference-Time Hyper-Scaling Results (Accuracy vs. Memory Reads)

The headline results appear in Figure 3 (top row) for AIME 24 and LiveCodeBench, and Figure 7 (top row) for MATH-500 and GPQA Diamond. Across all model scales and datasets, DMS achieves the best Pareto frontiers when comparing accuracy against KV-cache token reads, consistently outperforming both vanilla models and the best training-free baselines.

AIME 24 (Figure 3, top-left): For Qwen-R1 32B, DMS configurations (primarily at 4× and 8× compression) form a Pareto frontier that lies substantially above the vanilla frontier. The average Pareto improvement of DMS over vanilla is 12.0 points on the memory-reads axis (Table 8). For comparison, Quest achieves only a 2.0-point average improvement over vanilla on this model–dataset combination. At a specific configuration: vanilla Qwen-R1 32B at 1-32K-1× scores 70.0% on AIME 24 (Table 10), while DMS at 4× compression with the same sequence length (1-32K-4×) scores 73.3% — a 3.3-point gain at one-quarter the KV-cache reads. At 8× compression, DMS still achieves 73.3% (Table 12), matching the 4× result and surpassing vanilla by 3.3 points at one-eighth the memory reads.

The gains are even larger for smaller models: Qwen-R1 1.5B shows a 10.6-point average DMS improvement over vanilla (Table 8), and Qwen-R1 7B shows 15.0 points. This pattern — larger relative gains for smaller models — appears consistently and suggests that smaller models benefit more from the additional token budget that compression enables, likely because their base reasoning capabilities are more constrained and benefit more from extended exploration.

LiveCodeBench (Figure 3, top-right): The pattern holds for code generation. DMS achieves an average Pareto improvement of 9.7 points over vanilla for Qwen-R1 32B, 7.9 points for 7B, and 7.3 points for 1.5B (Table 8). Quest achieves more modest improvements: 3.8, 5.0, and 2.5 points respectively. DMS over Quest shows 5.6, 3.4, and 4.9 points of improvement — meaning DMS's learned eviction decisions provide substantial gains over Quest's selective retrieval even though Quest retains all tokens in memory.

MATH-500 (Figure 7, top-left): Gains are smaller on MATH-500 because the benchmark is closer to saturation — vanilla Qwen-R1 32B already achieves 94.8% accuracy at 32K context (Table 10). DMS at 4× compression maintains 94.6% at one-quarter the memory reads, and at 8× achieves 94.6% as well (Table 12). The average Pareto improvement of DMS over vanilla is 1.6 points for 32B, 1.0 for 7B, and 4.2 for 1.5B (Table 8). Quest shows negative average improvements against vanilla for 1.5B and 7B (−1.8 and −0.6 points respectively) — on this near-saturated benchmark, Quest's lack of memory savings means it cannot translate its accuracy preservation into Pareto improvements.

GPQA Diamond (Figure 7, top-right): DMS achieves average Pareto improvements of 8.6, 4.1, and 4.8 points over vanilla for 32B, 7B, and 1.5B models respectively (Table 8). At direct comparison (Table 10): vanilla Qwen-R1 32B at 1-32K-1× scores 63.1%, while DMS at 4× scores 63.6% — a 0.5-point improvement at one-quarter the reads. At 8× (Table 12), DMS scores 64.6% — actually surpassing vanilla by 1.5 points. This counterintuitive result (higher accuracy at lower cost) is possible because the 8× compression enables generating more tokens within the same memory budget during inference-time scaling, which on GPQA Diamond translates to better reasoning chains.

Summary across all datasets (Figure 1, Tables 8 and 9): Averaging the Pareto frontier gains across all model scales for each dataset, DMS improves over vanilla by 12.5 points on AIME 24, 2.3 on MATH-500, 5.8 on GPQA Diamond, and 8.3 on LiveCodeBench for the memory-reads metric. DMS over Quest shows improvements of 9.4 points (AIME 24), 3.2 (MATH-500), and 4.6 (LiveCodeBench) where the comparison is feasible (NA for some dataset-scale combinations where Pareto frontiers are disjoint; Table 8).

A crucial pattern visible in the Pareto frontier plots: most DMS points at 4× and 8× compression lie on the Pareto frontier, while vanilla points at 1× compression are consistently Pareto-dominated by some DMS configuration. The configurations that lie on the frontier typically combine both sequential and parallel scaling (intermediate W and L values), not just one or the other — confirming that hyper-scaling requires exploiting both dimensions of the expanded token budget.

Inference-Time Hyper-Scaling Results (Accuracy vs. Peak Memory)

The memory-load results (Figure 3, middle row; Figure 7, middle row) compare accuracy against peak tokens in memory. Since Quest does not reduce memory, the primary comparison here is DMS vs. TOVA (the best training-free method for memory reduction) vs. vanilla.

AIME 24: DMS achieves average Pareto improvements over vanilla of 14.6 points (32B), 15.7 (7B), and 17.3 (1.5B) on the memory axis (Table 9). TOVA shows substantially smaller improvements: 2.6, −0.2, and 5.3 points respectively. DMS over TOVA shows 8.1, 15.6, and 9.6 points — a clear indication that learned eviction preserves reasoning quality far better than heuristic eviction at the same memory reduction.

GPQA Diamond: The pattern is similar: DMS over TOVA shows 6.3, 6.5, and 5.6 points average improvement for 32B, 7B, and 1.5B models (Table 9). At direct comparison (Table 11): vanilla Qwen-R1 7B at 32K context scores 51.5% on GPQA Diamond; DMS at 4× compression scores 48.5% (a 3-point drop); TOVA at 4× compression scores 47.5% (a 4-point drop). The TOVA-DMS gap is modest in this direct comparison but widens substantially when looking at the full Pareto frontier, indicating that DMS's advantage comes from maintaining accuracy across a wider range of budget configurations, not just at the single-configuration level.

LiveCodeBench: DMS over TOVA shows 11.1, 6.0, and 4.0 points improvement for 32B, 7B, and 1.5B (Table 9). The gap grows with model scale, suggesting that for larger models with more complex reasoning patterns, learned eviction decisions become increasingly important relative to heuristic decisions.

Throughput Results

The throughput measurements (Figure 3, bottom row; Figure 4; Appendix I) translate the memory-read improvements into actual serving speed on NVIDIA H100 SXM GPUs.

Throughput vs. accuracy (Figure 3, bottom): For AIME 24 and LiveCodeBench, DMS configurations achieve higher throughput at equivalent accuracy compared to vanilla. The throughput is measured at the maximum batch size that fits in GPU memory for each configuration, meaning DMS can serve more parallel queries because its compressed KV cache occupies less VRAM per query. This is a direct consequence of reduced peak memory: a DMS model at 4× compression can fit approximately 4× more parallel sequences in the same GPU memory.

Qwen3-8B throughput (Figure 8): This figure specifically compares vanilla Qwen3-8B against DMS variants at 4×, 8×, and 12× compression on LiveCodeBench. DMS at all compression ratios enables higher throughput than vanilla while matching or exceeding accuracy. At the highest throughput point, DMS 12× achieves approximately 5× higher throughput than vanilla at the same accuracy level. This demonstrates that KV cache compression's benefits extend beyond single-query latency to multi-query serving scenarios, which is the relevant metric for production deployments.

Latency analysis (Figure 4, Appendix I): The latency of a single generation step is approximately constant at short context lengths but begins to rise once the KV cache becomes large enough that memory reads dominate computation. The threshold at which this happens depends on batch size and model scale. Figure 4 (top) shows that for Qwen-R1 models at batch size 32, the latency curve for the 32B model rises around 2K–4K tokens, while the 1.5B model remains flat until approximately 8K tokens — larger models have proportionally larger MLP and attention projection FLOPs that can better hide memory latency. Figure 4 (bottom) shows that the threshold also depends on batch size: at batch size 32, latency rises around 2K–4K tokens for Qwen3-8B; at batch size 128, it rises almost immediately. This means KV cache compression provides latency benefits across a wide range of practical configurations, not just at extreme context lengths.

DMS for General-Purpose Use (Non-Scaling Evaluation)

The paper evaluates whether DMS models can serve as general-purpose LLMs (i.e., at equivalent generated token lengths, not reinvesting the savings into more tokens) to establish that the compression itself does not degrade base capabilities.

Qwen-R1 distilled models (Tables 10–12): At fixed W=1 (single reasoning chain) with equivalent maximum length, DMS at 4× compression largely preserves accuracy compared to vanilla. On AIME 24: vanilla 32B scores 70.0% at 32K, DMS 4× scores 73.3% (Table 10). On MATH-500: vanilla 32B scores 94.8%, DMS 4× scores 94.6% (Table 10). On GPQA Diamond: vanilla 32B scores 63.1%, DMS 4× scores 63.6% — DMS actually slightly outperforms vanilla at one-quarter the memory reads. At 8× compression (Table 12), degradation is more noticeable but often modest: AIME 24 32B drops from 70.0% to 73.3% (actually an increase), MATH-500 32B drops from 94.8% to 94.6%, and GPQA Diamond 32B increases from 63.1% to 64.6%. The 32B model at 8× compression never underperforms vanilla on these three benchmarks at fixed generation length. For smaller models, 8× compression shows more degradation: Qwen-R1 1.5B drops from 84.8% to 80.0% on MATH-500 and from 30.0% to 23.3% on AIME 24 (Table 12).

Qwen3-8B (Table 2): Evaluated across 7 diverse benchmarks with thinking mode enabled, DMS 8× with a 512-token sliding window achieves: GPQA Diamond 57.6 (vanilla 58.8, −1.2), MMLU-Pro 73.5 (74.2, −0.7), AIME 2024 73.0 (75.0, −2.0), MATH-500 95.5 (95.1, +0.4), HumanEval 89.6 (87.8, +1.8), IFEval 88.8 (90.3, −1.5), ArenaHard v0.1 89.7 (88.4, +1.3). The DMS model is within 2 points on all benchmarks and exceeds vanilla on MATH-500, HumanEval, and ArenaHard. This is notable because it demonstrates that the learned sparsification does not merely preserve capabilities but can sometimes improve them, possibly due to a regularization effect similar to dropout.

Llama 3.2 1B Instruct (Table 1): Evaluated across short-context (GSM8K, MMLU, HellaSwag) and long-context (NIAH, VT) tasks at compression ratios 2×, 3×, and 4×. DMS with a 16-token window is the most robust method overall. On GSM8K at 4×: DMS 41.0% vs. vanilla 47.0% (−6.0), compared to TOVA 20.2% (−26.8), H2O 14.7% (−32.3), Quest 39.9% (−7.1), and DMC 3.6% (−43.4). On NIAH at 3×: DMS 93.6% vs. vanilla 96.4% (−2.8), while TOVA drops to 40.2%, H2O to 17.2%, and DMC to 1.8%. On VT at 4×: DMS achieves 67.6%, significantly outperforming vanilla (55.8%, +11.8), while all other methods degrade (TOVA 33.8%, H2O 12.6%, Quest 49.6%, DMC 4.0%). This is a striking result: DMS can actually improve long-context reasoning through its learned sparsification, possibly because aggressive eviction of redundant tokens acts as an attention sharpening mechanism that helps the model focus on the truly relevant context.

A critical detail in Table 1: the DMC model collapses catastrophically on long-context tasks (NIAH, VT) at all compression ratios, dropping to near-zero scores even at CR2. The paper attributes this to DMC's "more challenging training objective amplified by the limited 1B model capacity" (Section 5.2 footnote) and, in Appendix F, notes that DMC "collapses quickly" but this collapse "does not occur for shorter contexts and a larger non-GQA model" (see Table 5 for Llama 2 7B results). This highlights a practical advantage of DMS: its simpler binary eviction objective is more robust to model scale and generalizes better beyond the training context length.

Context-length extrapolation (Appendix F, Table 4): DMS models trained at 4K context length can extrapolate to 8K evaluation. On NIAH at CR2: DMS achieves 99.4% at 8K (vs. 96.4% vanilla at 4K training length), showing strong extrapolation. On VT at CR2: DMS achieves 56.0% at 8K (vs. 55.8% vanilla at 4K). However, at CR3, 8K NIAH drops to 24.2% for DMS, indicating that the combination of high compression and extrapolated context length eventually exceeds the model's capacity. DMC, in contrast, collapses to 0.0% on NIAH at 4K for CR2 — it fails even within its training context length. Quest extrapolates well (95.6% at 3× CR, 8K NIAH) because it never actually removes tokens from memory, but this comes at the cost of no memory reduction.

Ablation Studies and Robustness Checks

Delayed vs. immediate eviction (Figure 5, left): On Llama 3.2 1B Instruct evaluated on GSM8K 0-shot, delayed eviction (the DMS default) maintains accuracy substantially better than immediate eviction across all compression ratios and window sizes tested. With a 16-token window: at CR 2×, delayed achieves approximately 46.5% vs. immediate at approximately 42%; at CR 3×, delayed achieves approximately 45.5% vs. immediate below 35%; at CR 4×, delayed achieves 41.0% vs. immediate collapsing below 10%. Larger windows help both policies but do not close the gap: even with a 256-token window, immediate eviction at CR 4× drops to roughly 30%, while delayed eviction at CR 4× with a 16-token window already achieves 41%. This is the central ablation justifying DMS's design — it demonstrates that the temporal structure of eviction decisions (anticipatory vs. reactive) matters qualitatively, not just the presence of a sliding window.

Data efficiency comparison with DMC (Figure 5, right): On the same Llama 3.2 1B Instruct setup, DMS achieves higher GSM8K accuracy than DMC while using an order of magnitude fewer training tokens. At CR 4×, DMS reaches approximately 41% accuracy with roughly 2K–4K training tokens, while DMC requires approximately 16K+ tokens to reach similar performance (and plateaus lower). The figure shows DMS converging to its final accuracy within ~2K tokens, while DMC continues to require more tokens. This ablation justifies the claim that DMS requires substantially less training data than DMC — approximately 8× fewer tokens to reach comparable accuracy, and 60× less in the reasoning model experiments (300 steps to CR4 for DMS vs. thousands for DMC).

Sliding window size (Figure 5, left; Table 1): For Llama 3.2 1B Instruct, DMS with a 16-token window preserves accuracy up to CR 4×. Larger windows provide marginal improvements: at CR 4×, a 256-token window achieves slightly higher accuracy than a 16-token window on GSM8K, but the difference is small (several percentage points). Table 1 results use a 16-token window across all tasks, demonstrating that even very small windows suffice for effective information transfer before eviction. The reasoning models (Qwen-R1, Qwen3) use a default window of 256 tokens (Section 4), which provides a conservative margin.

Compression ratio annealing schedule (Section 4 and Appendix C): The linear schedule CR(t) = t/100 + 1 was determined empirically and produces a family of models at different compression ratios from a single training run. The paper does not ablate alternative schedules (e.g., geometric, step-wise), which is a minor limitation — it's possible that faster or slower annealing could improve final accuracy or training efficiency.

Logit distillation vs. standard LM loss (Section 4): The paper uses logit distillation for all retrofitting, citing prior work that it provides "greater robustness to shifts in training data" and is "especially beneficial for fragile LLMs with lower parameter counts." An explicit ablation comparing distillation to standard LM loss is not provided. However, the results in Table 5 (Llama 2 7B trained with standard LM loss) show that DMS still works with LM loss: at CR4, DMS achieves ARC-C 45.8 (vanilla 45.6), GSM8K 14.2 (14.9), HS 76.0 (75.5), MMLU 43.7 (45.4), NIAH 100.0 (100.0). The LM-loss DMS performs comparably to the distillation-trained models on these metrics, suggesting that distillation is beneficial but not strictly necessary.

Quest configuration details (Appendix H.1): The paper acknowledges that Quest is given an advantage over other methods: it uses full dense attention during prefilling (while DMS and TOVA/H2O use sparse prefilling), a separate top-k for each query head (potentially increasing memory transfers), and block size max(16, 2 × CR). This makes Quest's strong accuracy numbers even more noteworthy as a baseline, and DMS's dominance over Quest more significant — DMS outperforms a method that was given systematic advantages.

DMC extrapolation failure (Table 4, Appendix F): This is an important negative result. DMC models trained at 4K context length collapse to 0.0% on NIAH and VT when evaluated at 4K and 8K context lengths, even at low compression ratios (CR2). The paper notes this collapse is related to DMC's "more challenging training objective" and model capacity. It shows that learned merging (DMC's approach) is fragile with respect to context length extrapolation, while learned eviction (DMS) generalizes more robustly. This negative result strengthens the paper's argument that sparse attention (eviction) is practically preferable to merging-based compression.

Per-layer and per-head compression patterns (Figure 6): This is a descriptive analysis rather than an ablation, but it validates that DMS learns non-uniform compression. The measured compression ratio varies across sequence positions (Figure 6, left): the model sparsifies less aggressively early in sequences (below the target CR) and more aggressively later (above the target CR). This matches DMC's reported behavior and reflects the higher information density at the beginning of text. Per-head compression (Figure 6, right) shows that early layers retain more tokens (lower compression) than later layers, consistent with the intuition that early-layer representations are less redundant. Individual heads within each layer adopt different compression ratios, confirming the heterogeneity enabled by the one-sided auxiliary loss.

Effect of Grouped Query Attention on baseline performance (Section 4, Section 5.1): All evaluated models use GQA, which the paper notes "exacerbates the destructive effects of training-free token eviction" because evicting a KV token affects all query heads in the group simultaneously. This is not ablated directly (no non-GQA reasoning models are evaluated), but it explains why TOVA and H2O perform so poorly on these models compared to prior work that used non-GQA architectures.

Critical Assessment

Does the evidence support the central claim that KV cache compression improves inference-time scaling?

The paper's core claim is that by compressing the KV cache, models can generate more reasoning tokens within the same memory bandwidth and memory capacity budget, yielding higher accuracy at equivalent cost. The evidence for this claim is strong but conditional.

The Pareto frontier analysis (Figures 3 and 7) convincingly demonstrates that DMS configurations achieve higher accuracy than vanilla configurations at equivalent memory reads and peak memory across four reasoning benchmarks and three model scales. The averaging methodology in Tables 8 and 9 quantifies these gains, showing consistent positive improvements on AIME 24, GPQA Diamond, and LiveCodeBench, with smaller gains on the near-saturated MATH-500.

However, the claim is demonstrated specifically for DMS — a learned compression method — not for KV cache compression in general. The training-free baselines (TOVA, H2O) show substantially smaller or negative Pareto improvements (Tables 8 and 9), and Quest's improvements are inconsistent. This means the paper's title claim about "Inference-Time Hyper-Scaling with KV Cache Compression" is demonstrated only for the specific compression method the paper introduces. A reader might reasonably ask: does hyper-scaling work with any compression method, or only with DMS? The evidence suggests the latter: only compression that preserves reasoning accuracy at high ratios enables hyper-scaling. This is an important qualification that the paper acknowledges implicitly (Section 5.1: "KV cache compression methods generally yield superior Pareto frontiers compared to vanilla LLMs") but could state more directly.

A second condition: the gains are demonstrated on reasoning benchmarks with long generation traces. On short-context tasks like MMLU or HellaSwag, where the KV cache is small and attention is not memory-bound, hyper-scaling would provide no benefit — the memory bandwidth savings cannot be reinvested because there is no memory bandwidth bottleneck to begin with. The paper's throughput analysis (Figure 4) shows that latency benefits kick in at different context lengths depending on model scale and batch size, implying that hyper-scaling is most impactful for long-form reasoning, not for all inference workloads.

Are the efficiency metrics truly implementation-agnostic?

The paper introduces "KV-cache token reads" as an "implementation-agnostic proxy for attention compute" (Section 5.1). While this metric correctly identifies that memory bandwidth is the bottleneck, it has limitations as a proxy for real-world latency:

It assumes all token reads cost the same. In practice, GPU memory systems have caches (L1, L2), and repeated reads of the same KV cache entries (e.g., for tokens that remain after eviction) may hit in cache. DMS tokens that survive eviction are read at every subsequent step, while evicted tokens are never read again. The "token reads" metric treats each read as equally costly, but cache effects could make repeated reads cheaper than first-time reads.
It does not account for the overhead of computing eviction decisions. DMS adds a small computation at each step (extracting α_t from the repurposed query neuron and thresholding), but this overhead is negligible compared to attention computation. Training-free methods like TOVA and H2O have their own overhead for computing attention-based eviction scores. These overheads are not factored into the comparison.
It assumes optimal sparse attention kernel implementations. The paper mentions that DMS is "implementable with existing tools" (FlexAttention, FlashMask) but does not provide benchmarks with production-grade sparse attention kernels. The actual speedup relative to vanilla attention depends on how efficiently the sparse pattern can be executed on GPU hardware. The FlashAttention-based latency measurements (Figure 4) provide some grounding, but these use a simple Hugging Face implementation, not highly optimized kernels.

These limitations mean the reported Pareto improvements should be interpreted as upper bounds on achievable speedups — they represent what is possible with perfect sparse attention implementations, but real-world deployments may see smaller gains. The throughput measurements in Figure 3 (bottom) partially address this by using actual GPU measurements, but these are at the maximum batch size that fits in memory, which is a specific operating point that may not reflect all deployment scenarios.

Is the training cost truly negligible?

The paper emphasizes that DMS requires only ~1K training steps, which it contrasts favorably with DMC's ~44K steps. Table 3 shows that training one CR unit (e.g., 1× → 2×) requires 30 GPU hours for Qwen-R1 1.5B, 75 GPU hours for 7B, and 345 GPU hours for 32B. A full run to CR 8× (7 CR units) would require approximately 210, 525, and 2,415 GPU hours respectively.

This is cheap relative to pretraining but not negligible in absolute terms. For the 32B model, 2,415 H100 GPU hours is a significant computational investment — roughly $5,000–$ 10,000 at current cloud GPU pricing. Whether this is "cheap" depends on the deployment scale: for a model serving millions of queries, the one-time training cost is amortized quickly; for a research lab experimenting with compression, it's a meaningful barrier.

Moreover, the training requires access to the base model for logit distillation. For proprietary or API-only models (e.g., GPT-4, Claude), this is impossible — DMS can only be applied to models where the practitioner has full weight access and can run forward passes for distillation. This limits DMS to the open-weight model ecosystem (Llama, Qwen, DeepSeek), which the paper acknowledges implicitly by only evaluating on such models.

A missing comparison: the paper does not report the cost of deploying DMS compared to simply using a larger GPU. If DMS at 8× compression on a 32B model costs ~2,400 GPU hours to train and then enables fitting the model on a smaller GPU, the total cost of ownership calculation would need to include both the training cost and the hardware savings. This is a practical consideration that the paper does not address.

Does DMS genuinely "surpass" vanilla models on long-context tasks?

Table 1 shows DMS achieving higher accuracy than vanilla on NIAH (97.8% vs. 96.4% at CR2) and VT (63.2% vs. 55.8% at CR2; 69.2% vs. 55.8% at CR3). This is presented as evidence that DMS can improve over vanilla. However, the paper does not investigate why this happens.

Possible explanations include: (a) the additional training (even with distillation) serves as a form of continued pretraining that improves the model; (b) the compression acts as a regularizer (similar to dropout) that improves attention focus; (c) the specific eviction patterns learned by DMS happen to filter out distracting context. The paper speculates (in the context of VT) that "aggressive eviction of redundant tokens acts as an attention sharpening mechanism," but this is not tested. An ablation comparing DMS to a model that received the same amount of continued training without compression (i.e., just distillation fine-tuning for the same number of steps) would distinguish between the effects of additional training and the effects of compression. Without this ablation, it's unclear whether the improvements are due to DMS's compression mechanism or simply due to the extra training tokens.

This is a minor weakness — the fact that DMS can match or exceed vanilla is sufficient for the paper's claims — but it leaves open the question of whether the "surpassing" results on VT and NIAH reflect a genuine benefit of sparsification or an artifact of the experimental design.

How robust are the results to the specific prompt format and evaluation protocol?

The paper evaluates reasoning models using specific prompt formats (Appendix H.2) that include instructions to "think step by step" and output answers in specific formats (e.g., \boxed{ANSWER} for math, Answer: $LETTER for GPQA). The accuracy metrics depend on parsers (MathVerify) that extract final answers from model outputs. If DMS changes the surface form of model outputs (e.g., by causing the model to generate differently structured reasoning traces), this could affect parser accuracy independently of actual reasoning quality. The paper does not report qualitative analysis of DMS outputs vs. vanilla outputs, which would help assess whether observed accuracy differences reflect genuine reasoning improvements or parsing artifacts.

Additionally, the Qwen-R1 models are evaluated in a "thinking" mode (with thinking and response tags as shown in Appendix H.2), but the paper does not specify whether the efficiency metrics count only the final answer tokens or all tokens including the thinking trace. If thinking tokens are included in the KV cache reads (which they must be, since they are generated tokens), then DMS's compression during the thinking phase could affect the quality of the thinking trace itself. The paper's results aggregate final answer accuracy, but there is no analysis of whether DMS changes the nature of the thinking process (e.g., length, step-by-step structure, use of verification).

What's missing?

Several experiments would strengthen the paper's conclusions:

Comparison against a same-budget continued-training baseline without compression. This would isolate the effect of compression from the effect of additional training tokens. If DMS outperforms a model that was fine-tuned with distillation for the same number of steps but without compression, the case for compression specifically (rather than continued training generally) would be stronger.
Evaluation on non-reasoning long-generation tasks. The paper focuses on reasoning benchmarks, but many production use cases involve long-form generation that is not strictly "reasoning" (e.g., document summarization, story generation, dialogue). Testing whether hyper-scaling helps in these settings would broaden the claimed applicability.
Ablation of the specific neuron repurposing vs. adding a new parameter. The paper claims that borrowing a neuron from the query projection is equivalent to adding a separate learned projection, but this is not empirically verified. An explicit comparison would validate this design choice.
Evaluation with process reward models (PRMs). The paper notes (Section 5.4) that hyper-scaling could be extended to PRM-based strategies but does not test this. Given that PRMs are widely used in inference-time scaling (Snell et al., 2024; Lightman et al., 2024), demonstrating compatibility would be valuable.
Statistical significance of Pareto frontier improvements. The paper computes average improvements over Pareto frontiers (Tables 8 and 9) but does not provide confidence intervals or error bars for these metrics. Given that the Pareto frontiers are constructed from a finite set of W-L-CR configurations evaluated on a finite test set (e.g., 30 problems for AIME 24), the integrated difference metric has variance that is not characterized.
Comparison at equivalent training FLOPs, not just inference. All methods (DMS, DMC, vanilla) use different amounts of training compute. A FLOPs-matched comparison including retrofitting cost would provide a more complete picture of total cost of ownership, though this is admittedly a higher bar than most inference-efficiency papers meet.

Summary assessment

The experimental evidence strongly supports the claim that DMS enables better accuracy-efficiency Pareto frontiers for reasoning tasks compared to vanilla models and training-free compression baselines. The gains are consistent across model scales (1.5B–32B), datasets (math, science, coding), and efficiency metrics (memory reads, peak memory, throughput). The ablation of delayed vs. immediate eviction (Figure 5) provides convincing evidence for the central design choice.

The evidence partially supports the broader claim that KV cache compression in general enables hyper-scaling. Only DMS demonstrates consistent gains; training-free methods (TOVA, H2O) fail at the compression ratios needed for meaningful hyper-scaling, and Quest (which preserves accuracy but not memory savings) shows inconsistent benefits. The paper's title claim should therefore be understood as "Inference-Time Hyper-Scaling is enabled by DMS, a novel KV cache compression method" rather than "any KV cache compression method enables hyper-scaling."

The evidence is weakest regarding the practical deployability of the approach. The latency and throughput measurements use simple implementations, not production-grade sparse attention kernels. The training cost, while modest relative to pretraining, is not negligible in absolute terms and requires full model access. The difficulty estimation problem that the hyper-scaling framework introduces (how to choose the optimal W-L-CR configuration for a given query without evaluating all of them) is not addressed — the Pareto frontiers show what is possible, not how to achieve it automatically at deployment time. These are not fatal weaknesses for a research contribution, but they are important caveats for practitioners hoping to immediately deploy the technique.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation and Configuration Selection Are Not Addressed

The assumption or constraint: The paper's hyper-scaling framework establishes that for a given problem, some W-L-CR configuration will lie on an improved Pareto frontier, but it provides no mechanism for selecting which configuration to use for a specific query at deployment time. The entire evaluation is retrospective: the Pareto frontiers are constructed by sweeping all combinations of width, length, and compression ratio, then identifying which points are non-dominated. This is an analysis of what is possible, not a deployable system that can decide what to do.

The paper is transparent that the cost of sweeping is not accounted for:

"our experiments do not account for this cost largely for simplicity" (Section 3.2, regarding difficulty estimation cost, but the principle extends to configuration selection).

The consequence: A practitioner deploying a DMS-retrofitted model faces an exploration–exploitation problem that the paper does not solve: for each incoming query, should the system generate 1 long chain or 16 parallel shorter chains? At what compression ratio? There is no learned policy, no difficulty estimator, and no adaptive budget allocator. The Pareto frontiers in Figures 3 and 7 show upper bounds on achievable performance — to realize these gains in practice, one would need either (a) an oracle that knows the optimal configuration per query, (b) a meta-policy that selects configurations based on query features, or (c) a willingness to run all configurations and pick the best, which would consume the entire efficiency gain many times over.

This is not merely a missing engineering detail. It is analogous to the compute-optimal test-time scaling problem studied by Snell et al. (2024), where the optimal allocation of sequential vs. parallel compute depends on problem difficulty. The present paper adds a third dimension (compression ratio) to this already-open problem, making the configuration space larger and the need for an adaptive policy more acute.

What evidence exists in the paper: The paper provides no experiment on configuration selection. The Pareto frontier plots (Figures 3 and 7) implicitly demonstrate that the optimal W-L-CR tuple varies by dataset and model scale — the points lying on the frontier differ across plots — but there is no analysis of whether a simple heuristic (e.g., "always use 4× compression with balanced W/L") would capture most of the gains, or whether per-query adaptation is essential.

Mitigation status: Not addressed. The paper suggests future work on this in a different context (difficulty estimation for deciding how many reasoning tokens to generate), but does not propose or evaluate any configuration selection strategy. This is the single largest gap between the paper's analysis and practical deployment.

Limitation 2: Training Requires Full Model Access and Non-Trivial Compute

The assumption or constraint: DMS retrofitting requires running forward passes of the original (uncompressed) model as a teacher for logit distillation, and requires continued training of the full model with the Gumbel-sigmoid eviction mechanism. This assumes: (a) access to the full model weights, (b) access to sufficient training data (even if unlabeled, for distillation), and (c) sufficient GPU compute to perform the retrofitting.

The paper acknowledges the training cost in Appendix C (Table 3), reporting that reaching CR 8× from scratch requires 700 training steps: approximately 210 GPU hours for Qwen-R1 1.5B, 525 for 7B, and 2,415 for 32B on H100 GPUs. The authors also note that the training data composition matters — for Qwen-R1 models, they use the OpenR1-Math-220k dataset filtered for correctness, while for Llama models they use synthetic data. The paper does not claim this is zero-cost:

"The amount of required data depends directly on the context length of retrofitted models and the target compression ratio: higher ratios necessitate larger datasets." (Section 4)

The consequence: DMS is inapplicable to proprietary or API-only models (GPT-4, Claude, Gemini) where weights are not available for retrofitting. This bounds the method's impact to the open-weight model ecosystem (Llama, Qwen, DeepSeek, Mistral). For a practitioner with a 32B-parameter model, the ~2,400 H100-GPU-hour training cost for CR 8× represents a meaningful investment — roughly $5,000–$ 10,000 at current cloud pricing — that must be amortized over inference savings. If the model is only served infrequently, the training cost may exceed the lifetime inference savings. The paper provides no break-even analysis to guide this decision.

Additionally, the retrofitting process is specialized to the base model's architecture and tokenizer. A DMS-trained model cannot be easily transferred to a different base model (unlike training-free methods like TOVA or Quest, which can be applied to any model immediately). This means each model variant requires its own retrofitting run, multiplying the cost for practitioners who maintain multiple model sizes or versions.

What evidence exists in the paper: Table 3 provides training cost in GPU hours. The paper does not provide a total cost of ownership comparison (training + inference) against alternatives such as simply using a larger GPU or deploying the uncompressed model with lower batch sizes. Figure 5 (right) shows that DMS requires substantially less data than DMC, but this is a relative comparison within learned methods, not an absolute efficiency claim.

Mitigation status: The paper positions DMS as "inexpensive" relative to DMC (60× fewer training tokens for reasoning models) and relative to pretraining. It notes the alternative of using a separate zero-initialized projection rather than repurposing a query neuron, which "would effectively eliminate the need for continued training" (Appendix C) — but this alternative is not empirically evaluated, and it is unclear whether a frozen model with an added eviction head could achieve comparable accuracy without any training. This is explicitly flagged as future work.

Limitation 3: Verification Is Limited to a Narrow Set of Reasoning Benchmarks and Model Families

The assumption or constraint: The hyper-scaling evaluation (Figures 3 and 7) is conducted exclusively on four reasoning benchmarks (AIME 24, MATH-500, GPQA Diamond, LiveCodeBench) using Qwen-R1 distilled models at three scales (1.5B, 7B, 32B). The general-purpose evaluation (Tables 1 and 2) adds Llama 3.2 1B Instruct and Qwen3-8B, plus a broader set of tasks (GSM8K, MMLU, HellaSwag, NIAH, VT, MMLU-Pro, HumanEval, IFEval, ArenaHard), but these evaluations are at fixed generation lengths rather than in a hyper-scaling configuration.

The paper states this scope explicitly but does not claim broader applicability:

"In this work, we focus on models ranging from 1B to 32B parameters, context lengths up to 32K tokens, and compression ratios up to 8×" (Appendix A)

The consequence: The paper demonstrates that DMS + hyper-scaling works on math, science, and coding benchmarks with models that have been specifically fine-tuned for reasoning (R1 distillation). These are tasks where:

Long, structured reasoning chains (Chain-of-Thought) are both natural and effective.
Correctness is well-defined and can be automatically evaluated.
The base models already have non-trivial pass@1 rates, so additional tokens can plausibly improve answers.

For tasks where these conditions don't hold — factual recall (where additional tokens don't help), open-ended generation (where correctness is subjective), short-form tasks (where KV cache size is small and attention is not memory-bound), or models without strong base reasoning capabilities — the hyper-scaling framework may provide no benefit or even hurt. The paper's general-purpose results (Tables 1 and 2) show that DMS largely preserves accuracy at fixed generation length, but they do not demonstrate that adding more tokens via compression improves accuracy on these tasks. The hyper-scaling claim (better accuracy at equivalent cost) is only tested on reasoning benchmarks.

Additionally, all evaluated models use Grouped Query Attention (GQA). The paper argues that GQA "exacerbates the destructive effects of training-free token eviction" (Section 4), which makes DMS's advantage over TOVA/H2O larger on GQA models than it would be on standard multi-head attention models. This is a valid experimental choice, but it means the comparison against training-free baselines is evaluated in the regime most favorable to DMS. On non-GQA models, the gap between DMS and training-free methods may be narrower.

What evidence exists in the paper: The paper does not evaluate hyper-scaling on non-reasoning benchmarks or non-GQA models. The Llama 3.2 1B Instruct results (Table 1) use fixed-length evaluation, not hyper-scaling configurations, so they test accuracy preservation rather than accuracy improvement through expanded token budgets. The Llama 2 7B results (Table 5, Appendix F) are similarly at fixed length. There is no experiment showing that applying hyper-scaling to MMLU or HellaSwag would improve accuracy, and it is plausible that it would not (since these tasks don't benefit from extended reasoning chains).

Mitigation status: The paper does not claim applicability beyond the evaluated settings. The title and abstract refer to "inference-time scaling on reasoning tasks" specifically. However, a reader could easily overgeneralize from the impressive headline numbers to assume that hyper-scaling is universally beneficial. The paper could mitigate this by explicitly testing a case where hyper-scaling is not expected to help (e.g., a knowledge-intensive task with short answers) to establish the boundary conditions.

Limitation 4: The Sliding Window Size Is a Critical Hyperparameter with No Automated Selection Method

The assumption or constraint: DMS's central mechanism — delayed eviction — depends on a sliding window of size w that controls how long tokens remain accessible after being marked for eviction. The paper uses w = 256 as the default for reasoning models (Section 4) and shows that w = 16 suffices for Llama 3.2 1B Instruct on short-context tasks (Figure 5, left), but provides limited guidance on how to choose w for a new model or task.

The window size creates a fundamental tradeoff: larger w preserves more information before eviction (potentially improving accuracy) but reduces effective compression because more tokens are kept in the window at any time. For a sequence of length L with target compression ratio CR, the actual memory usage is approximately L/CR + w (the compressed portion plus the window). When L is small relative to w, the effective compression ratio is lower than the target; when L is large, the window overhead is negligible. This means the window size couples the compression ratio to the sequence length in a way that is not captured by the headline CR number.

The consequence: A practitioner applying DMS to a new use case must choose w without clear guidance. The paper shows that too-small windows hurt accuracy (Figure 5, left, shows accuracy declining as window size shrinks) and that DMC with effective window size 0 (immediate merging) collapses on long-context tasks (Table 4). But the paper does not provide a heuristic, a scaling relationship, or an automated method for selecting w. The choice of w = 256 for reasoning models appears to be a conservative default rather than an optimized value.

The window size also affects the maximum compression ratio achievable: if the model evicts all tokens (CR → ∞), the KV cache still contains w tokens from the sliding window. So the effective maximum compression ratio is bounded by L/w, meaning that for short sequences, very high compression ratios (>8×) may be practically unachievable regardless of what the model learns. This interacts with the context-length extrapolation behavior: when evaluating at context lengths longer than training (Table 4), the same window size represents a smaller fraction of the sequence, and the model must handle a higher effective compression ratio than it was trained for. This explains the collapse at CR3 with 8K evaluation (NIAH drops from 93.6% to 24.2% for Llama 3.2 1B going from 4K to 8K at CR3; Table 4).

What evidence exists in the paper: Figure 5 (left) provides a sweep of window sizes (16, 64, 256) for Llama 3.2 1B Instruct on GSM8K, showing that accuracy improves with larger windows but saturates quickly — the gap between 64 and 256 is smaller than between 16 and 64. The extrapolation results in Table 4 show that DMS with w = 16 trained at 4K context handles 8K evaluation at CR2 (NIAH 99.4%) but degrades at CR3 (NIAH 24.2%). The Qwen3-8B results (Table 2) use w = 512, suggesting that larger models benefit from or tolerate larger windows.

Mitigation status: Not addressed. The paper does not propose a method for selecting w automatically or provide scaling guidelines. The default of 256 is empirically justified for the evaluated models but is not theoretically motivated. The interaction between w, sequence length, and effective compression ratio is discussed qualitatively (Appendix F notes that DMS extrapolates "only up to a certain limit") but not formalized.

Limitation 5: The Latency and Throughput Measurements Use Non-Optimized Implementations

The assumption or constraint: The paper's throughput and latency measurements (Figures 3 bottom, 4, 8; Appendix I) are based on a "simple implementation based on the Hugging Face Transformers library and FlashAttention" (Section 5.1). The paper acknowledges that DMS relies on sparse attention patterns that "can be compactly passed as a vector of eviction decisions and is implementable with existing tools" (FlexAttention, FlashMask; Section 3.1), but the actual measurements do not use these optimized sparse kernels.

This means the measured speedups represent what is achievable with dense attention plus a compressed KV cache (fewer tokens to attend to, but the attention computation itself is still dense over those tokens), rather than what would be achievable with a sparse attention kernel that exploits the eviction pattern to skip computation entirely for evicted tokens.

The consequence: The reported throughput improvements (up to 5× for Qwen3-8B at CR 12×; Figure 8) may be either underestimates or overestimates of what a production implementation would achieve:

Underestimate: A sparse attention kernel that skips evicted tokens entirely (rather than attending to them with zero weight, as in the dense implementation) would reduce both memory reads and computation, potentially yielding larger speedups than the dense-attention-with-fewer-tokens approach.
Overestimate: The overhead of managing the sparse KV cache — tracking which tokens are evicted per head, compacting memory, handling the variable-length per-head caches in PagedAttention — is not measured. The paper states that evicted tokens "could be simply overwritten by incoming ones" (Section 3.2) and that PagedAttention facilitates sparse storage, but these claims about implementability are not validated with measurements. The actual overhead of sparse memory management on GPU could erode the theoretical gains.

Additionally, the throughput measurements in Figures 3 (bottom) and 8 are at the maximum batch size that fits in memory for each configuration. This is a valid measurement point but represents an extreme operating regime — maximum throughput, minimal latency per query. In latency-sensitive applications (where a user is waiting for a single response), the relevant metric is time-to-first-token or per-step latency at batch size 1, which the paper analyzes in Figure 4 but does not convert to end-to-end speedup numbers. DMS's sequential dependency (the sliding window means some tokens must be generated before eviction decisions are executed) could affect time-to-first-token differently from throughput.

What evidence exists in the paper: Figure 4 provides latency-per-step measurements showing that DMS delays the point at which attention becomes memory-bound. The throughput measurements in Figure 3 (bottom) and Figure 8 provide practical speedup numbers. The latency analysis in Appendix I formalizes the FLOPs and memory-read breakdown, confirming that KV cache reads dominate at scale. However, none of these measurements use production-grade sparse attention kernels — they rely on FlashAttention's dense implementation with fewer tokens.

Mitigation status: The paper is transparent about the implementation simplicity (Section 5.1) and claims implementability with existing tools, which is plausible given the existence of FlexAttention and FlashMask. But the gap between a Hugging Face + FlashAttention prototype and a production implementation is significant, and the paper does not quantify it. The "implementation-agnostic" metrics (KV-cache token reads, peak memory tokens) partially mitigate this by providing hardware-independent comparisons, but the throughput and latency numbers that practitioners care about are tied to the specific implementation.

Limitation 6: No Combination with Process Reward Models or Advanced Search Strategies

The assumption or constraint: The paper's hyper-scaling evaluation uses only verifier-free scaling strategies — specifically, majority voting (parallel scaling) and longer reasoning chains (sequential scaling). The paper explicitly acknowledges this scope limitation:

"While in this work we opt for verifier-free scaling strategies, adopting the recommendations of Wang et al. (2025b), inference-time scaling can rely on process reward models (PRMs) to verify intermediate reasoning steps. This allows effective self-critique loops and re-ranking candidate solutions" (Section 5.4)

The authors note that PRM-based strategies introduce their own computational challenges:

"the verifier's complexity is quadratic in the sequence length; to complement the benefits of KV cache compression of the LLM, the PRM would need to be accelerated by prefilling-time sparse attention methods, such as MInference" (Section 5.4)

The consequence: The results are bounded by what majority voting over independently sampled chains can achieve. Recent work on inference-time scaling (Snell et al., 2024; Lightman et al., 2024) has shown that verifier-guided strategies — such as best-of-N weighted by a process reward model, or tree search with PRM-guided pruning — can substantially outperform majority voting at equivalent compute budgets. If KV cache compression were combined with PRM-guided search, the expanded token budget could be allocated more efficiently (e.g., exploring more branches of a reasoning tree rather than independent chains), potentially yielding even larger gains than those reported.

However, the integration is not straightforward. The paper identifies a specific bottleneck: PRMs scale quadratically in sequence length, so while DMS compresses the generator's KV cache, the verifier's attention computation would still be dense over the full (or compressed) reasoning trace. Without also accelerating the verifier, the overall system might remain bottlenecked by PRM computation even if the generator is fast. This means the paper's current results represent a lower bound on what combined approaches might achieve, but also that the combination is a non-trivial research problem.

Additionally, the paper does not evaluate whether DMS preserves the step-by-step correctness signals that PRMs rely on. A PRM trained on vanilla model outputs might not transfer well to DMS-compressed outputs if the compression changes the structure or content of reasoning traces (e.g., by causing the model to omit intermediate steps that would normally be preserved). The paper's fixed-length evaluation (Tables 10–12) shows that DMS largely preserves final-answer accuracy, but there is no analysis of whether the intermediate reasoning steps remain similarly faithful.

What evidence exists in the paper: The paper provides no experiments combining DMS with PRMs. The discussion in Section 5.4 is speculative and identifies this as future work. The absence of PRM experiments is a deliberate scope choice, not an oversight, but it means the hyper-scaling results should be understood as specific to verifier-free majority voting, not as a general statement about inference-time scaling with any strategy.

Mitigation status: Explicitly deferred to future work (Section 5.4): "We leave this possible direction to future work." The paper does not provide preliminary results or a roadmap for the integration. A practitioner using PRM-based scaling strategies cannot directly apply the paper's findings without additional experimentation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a genuine conceptual reframing of inference-time scaling, rather than an incremental improvement. The dominant prior framing — embodied in work by Snell et al. (2024), Muennighoff et al. (2025), and Brown et al. (2024) — treats generated tokens as the unit of inference compute, leading to optimization strategies organized around token allocation (sequential vs. parallel, adaptive difficulty-based budgeting). This paper demonstrates that this framing is misaligned with hardware reality: in auto-regressive Transformers, the bottleneck is not token production FLOPs but the memory bandwidth consumed by reading the growing KV cache from HBM at each generation step. The latency analysis in Appendix I (Equations 2–6) makes this concrete, showing that KV cache reads can dominate per-step latency once sequence length and batch size cross modest thresholds (Figure 9).

The reframing has a specific prescriptive consequence: inference-time scaling research should optimize over memory operations (KV cache reads, peak memory tokens) rather than generated token counts. The paper's evaluation framework — Pareto frontiers of accuracy vs. memory reads and peak memory — provides the methodological template for this shift. This is not merely a different metric; it changes which methods look promising. For example, Quest (Tang et al., 2024) achieves reasonable accuracy at high compression-equivalent retrieval rates, but its Pareto frontier on the memory-reads axis is often equivalent to or worse than vanilla models (Table 8: Quest vs. Vanilla shows near-zero or negative average improvements on several model–task combinations). Under a token-count framing, Quest would appear effective; under the memory-bandwidth framing, its failure to reduce memory load makes it unsuitable for hyper-scaling. The reframing thus acts as a diagnostic tool that reveals hidden tradeoffs.

The paper also reconciles a tension in the efficient attention literature between training-free eviction methods and learned compression. Prior work presented a stark choice: training-free methods (TOVA, H2O) are cheap to apply but degrade accuracy at high compression ratios, while learned methods (DMC) preserve accuracy but require expensive training. DMS demonstrates that this tradeoff is not inherent to the problem but specific to design choices. By adopting binary eviction (simpler than DMC's merging) and delayed execution (the key temporal innovation), DMS achieves the best of both: training costs comparable to fine-tuning (700 steps to CR 8×), yet accuracy preservation competitive with or exceeding DMC. Crucially, Figure 5 (right) shows that DMS achieves higher accuracy than DMC while using 8× fewer training tokens — and on the reasoning models, the gap is ~60× (300 steps vs. thousands). This finding suggests that the field's prior investment in continuous merging (weighted averaging, learned compression functions) was solving a harder problem than necessary. Binary keep/evict decisions, when made well, suffice even at 8× compression.

A third shift concerns the role of the sliding window in efficient attention. Prior work (StreamingLLM, Xiao et al., 2024; TOVA, Oren et al., 2024) uses the window as the mechanism of compression — tokens outside the window are discarded. DMS repurposes the window as a scheduled information transfer buffer: it exists only to provide a grace period during which soon-to-be-evicted tokens remain accessible, while the actual compression ratio is determined by learned per-token decisions. This conceptual separation — what to keep vs. how long to keep it — is a small but genuine architectural insight. It means the window can be small (16–256 tokens) even at high compression ratios (8×), because its function is information extraction rather than retention.

The consequence for research directions: this paper increases the attractiveness of learned, per-token eviction policies trained with modest continued training, and decreases the attractiveness of both (a) training-free heuristics for eviction (which the paper shows fail at needed compression ratios on GQA models) and (b) complex merging-based compression (which the paper shows is unnecessarily expensive and fragile to context-length extrapolation, as DMC collapses in Table 4). For practitioners, the paper makes a specific cost–benefit argument: DMS retrofitting costs ~2,400 H100 GPU hours for a 32B model to reach CR 8× (Table 3), which is cheap relative to pretraining but non-trivial in absolute terms. Whether this is justified depends on inference volume — the paper does not provide a break-even analysis, but the throughput improvements (up to 5×; Figure 8) suggest rapid amortization at production scale.

Follow-Up Research This Work Enables

Adaptive configuration selection for hyper-scaling. The most immediate gap is that the paper's Pareto frontiers show what W-L-CR configurations are possible, but provide no mechanism for selecting the optimal configuration for a specific query at deployment time. This is the hyper-scaling analog of the compute-optimal test-time scaling problem studied by Snell et al. (2024), but with an added dimension (compression ratio). A strong follow-up would train a lightweight meta-policy — possibly a small classifier or regression model — that takes as input the prompt text and the available compute budget (in memory reads or peak memory) and outputs a (W, L, CR) tuple. The training signal could come from evaluating the DMS-retrofitted model across the configuration sweep on a training set, creating a dataset of (prompt, configuration, accuracy) tuples. The key evaluation would be whether the learned policy recovers most of the Pareto frontier gains without requiring per-query sweeps. The paper's difficulty estimation experiments (Section 3.2) provide a starting point for thinking about prompt features, and the observation that different datasets have different optimal configurations (compare the Pareto frontier point distributions across AIME 24 vs. GPQA Diamond in Figure 3) suggests that prompt-level adaptation is necessary rather than a one-size-fits-all default.

Combining DMS with process reward model (PRM) guided search. The paper explicitly defers this to future work (Section 5.4), but the combination is both natural and non-trivial. A PRM-guided search strategy — such as beam search over reasoning steps with a learned verifier scoring partial solutions — could benefit from DMS in two ways: (1) the generator's KV cache is compressed, allowing more beams to be explored within a fixed memory budget, and (2) if the PRM also uses sparse attention, the verifier's quadratic-in-length computation could be accelerated. The challenge the paper identifies is that PRMs do not naturally benefit from KV cache compression since they process complete (or partial) reasoning traces as input. A concrete experiment would be: take a Qwen-R1 32B model retrofitted with DMS at 8×, pair it with a PRM (e.g., Math-Shepherd; Wang et al., 2024a) that uses MInference-style sparse prefilling (Jiang et al., 2024), and evaluate on MATH-500 with a beam search budget of N beams, comparing against (a) vanilla model + PRM with N beams and (b) DMS model + majority voting with equivalent memory budget. The hypothesis is that DMS + PRM would achieve higher accuracy than either baseline because the expanded beam budget (from compression) is allocated more efficiently by the verifier than by independent sampling.

DMS for non-reasoning long-generation tasks with hyper-scaling evaluation. The paper's general-purpose results (Tables 1, 2, 5) show that DMS preserves accuracy at fixed generation lengths on tasks like MMLU, HellaSwag, and NIAH. However, the paper never tests whether hyper-scaling — reinvesting the memory savings into more generated tokens — improves accuracy on non-reasoning tasks. This is an important stress test because it would clarify the boundary conditions of the hyper-scaling framework. A concrete experiment: take a DMS-retrofitted instruction-tuned model (e.g., Qwen3-8B at CR 8× from Table 2) and evaluate on long-form generation tasks such as summarization (CNN/DailyMail, XSum), multi-document QA (NQ, TriviaQA with retrieved context), or dialogue (MT-Bench). Compare vanilla at 1-L-1× against DMS at W-L/CR-CR configurations (same memory footprint, more tokens) and 1-L×CR-CR configurations (longer output, same memory footprint). The key question: do these tasks benefit from expanded token budgets, or is the benefit specific to structured reasoning where Chain-of-Thought length correlates with accuracy? A negative result (no improvement) would not diminish the paper's contributions but would productively constrain the scope of hyper-scaling's applicability.

Context-length scaling laws for DMS window size. The paper uses a default window of 256 tokens for reasoning models and shows that 16 tokens suffices for Llama 3.2 1B at CR 4× (Figure 5, left), but provides no systematic study of how the required window size scales with context length, compression ratio, or model size. This is practically important because the window size determines the effective compression ratio for short sequences and affects extrapolation behavior. A scaling study would: for a fixed model (e.g., Qwen-R1 7B), train DMS variants with w ∈ {16, 64, 256, 512, 1024} at a fixed CR (e.g., 4×) and training context length (e.g., 8K), then evaluate on NIAH and VT at context lengths from 1K to 32K. The output would be a relationship between w, evaluation context length, and accuracy that could guide window size selection for new deployments. The paper's extrapolation results (Table 4) already hint at this: DMS with w = 16 succeeds at 8K NIAH at CR2 (99.4%) but collapses at CR3 (24.2%), suggesting the window-to-sequence-length ratio matters. A quantitative characterization of this relationship would transform w from an empirically chosen hyperparameter to a principled design choice.

Ablation isolating the effect of continued training from the effect of compression. The paper reports cases where DMS surpasses vanilla accuracy at equivalent generation length: DMS 8× on Qwen3-8B outscores vanilla on MATH-500 (95.5 vs. 95.1), HumanEval (89.6 vs. 87.8), and ArenaHard (89.7 vs. 88.4) in Table 2. Similarly, on Variable Tracking in Table 1, DMS at CR2 achieves 63.2% vs. vanilla 55.8%. The paper speculates this is due to "attention sharpening" from evicting redundant tokens, but an alternative explanation is that the continued training itself (700 steps of logit distillation, even without compression) improves the model. Distinguishing these hypotheses requires a simple control: train a model with the same distillation procedure, same data, and same number of steps, but with the compression auxiliary loss set to zero (i.e., α* = 0, so the model is never encouraged to evict). If this control model also shows accuracy improvements over vanilla, the gains are due to continued training. If only the compressed model improves, the gains are due to compression. This ablation would also establish a fairer baseline for DMC comparisons, since DMC's training cost includes both the merging mechanism and the continued training, and the paper's comparison (Figure 5, right) conflates the two.

Integration with quantization and decomposition methods. The paper notes (Appendix B) that KV cache quantization (KIVI, KVQuant) and SVD-based decomposition (LoRC, Eigen Attention) are orthogonal to sparsification and "can be potentially combined with [DMS] to further improve efficiency." A concrete combination experiment: apply 4-bit KV cache quantization (e.g., KIVI; Liu et al., 2024) to a DMS-retrofitted model at 4× compression. The combined memory reduction would be 4× (sparsification) × 4× (quantization) = 16× in terms of bytes per token (since sparsification reduces the number of tokens and quantization reduces the bytes per token). The question is whether the two forms of compression interact negatively — does sparsification change the distribution of retained KV values in a way that makes them harder to quantize accurately? Evaluating on a long-context retrieval task (NIAH, VT) at the combined 16× reduction would stress-test this interaction.

Practical Applications and Downstream Use Cases

High-throughput serving of reasoning models with long Chain-of-Thought. A cloud provider serving a reasoning model (e.g., Qwen-R1 32B) to multiple concurrent users faces a throughput bottleneck: each user's query generates thousands of reasoning tokens, and the KV caches for all concurrent requests must fit in GPU VRAM. At batch size 32 with 8K-token reasoning traces, the vanilla model would require storing 32 × 8K = 256K KV cache entries, which may exceed VRAM or push latency into the memory-bound regime (Figure 4). With DMS at 8× compression, the same VRAM accommodates 32 × 8K / 8 ≈ 32K entries worth of compressed KV cache, plus the sliding window overhead. This means the provider can either (a) serve 8× more concurrent users at equivalent latency, or (b) allow each user's query to generate 8× more reasoning tokens at the same concurrency. Figure 8 quantifies this: Qwen3-8B with DMS 12× achieves up to 5× higher throughput than vanilla at equivalent accuracy on LiveCodeBench. For a deployment processing millions of queries daily, this directly translates to reduced GPU rental costs or improved user experience through longer reasoning budgets.

Edge deployment of reasoning-capable LLMs on memory-constrained devices. A 7B-parameter reasoning model with 32K context length requires approximately 2 (KV) × 32 (layers) × 32K (tokens) × 128 (hidden dim per head) × 4 (KV heads for GQA) × 2 bytes ≈ 2 GB of KV cache memory at peak, in addition to the model weights (~14 GB in 16-bit). On an edge GPU with 16 GB VRAM, this leaves essentially no room for batch processing or longer sequences. DMS at 8× compression reduces the KV cache portion to ~250 MB, making the full 32K-context model comfortably deployable on 16 GB hardware. The general-purpose evaluation (Table 2) shows that DMS 8× on Qwen3-8B preserves accuracy within 2 points across seven diverse benchmarks, meaning the edge deployment sacrifices minimal capability. Additionally, the per-step latency improvement (Figure 4) matters for interactive applications where users expect real-time responses — DMS delays the onset of memory-bound latency to longer context lengths, keeping generation fast even as the reasoning trace grows.

Cost-efficient data generation for self-improvement pipelines. When using LLMs to generate reasoning traces for distillation or reinforcement learning (e.g., STaR, ReST, or the R1 training pipeline itself), the cost of generating millions of long Chain-of-Thought sequences can dominate the total training budget. A DMS-retrofitted model can generate the same number of tokens at lower latency (fewer memory reads per token) or generate more tokens within a fixed time budget. For a data generation run producing 1M reasoning traces averaging 4K tokens each on Qwen-R1 32B, the total KV cache reads for the vanilla model would be approximately 1M × 4K × (average reads per step). With DMS at 8× compression, the reads per step are reduced by ~8× for the compressed portion of the cache, translating to roughly proportional latency savings. The one-time retrofitting cost (~2,400 GPU hours; Table 3) would be amortized over the data generation run: if the vanilla model requires 10,000 GPU hours for generation, DMS might require ~1,250 GPU hours for generation + 2,400 for training = 3,650 total, a ~64% reduction. This calculation is approximate (the exact savings depend on sequence length distribution and the fraction of time spent in the memory-bound regime), but the order of magnitude suggests DMS retrofitting is likely cost-effective for large-scale data generation, not just for inference serving.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Transformers Meet In-Context Learning: A Universal Approximation Theory2025· inference-optimization← in-domain predecessor
- Rectified Sparse Attention2025· inference-optimization← in-domain predecessor
The Pitfalls of KV Cache Compression2025· inference-optimization← similar topic
- Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models2025· architecture← LLM-judged similar
- LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention2025· inference-optimization← LLM-judged similar
- Layer-Condensed KV Cache for Efficient Inference of Large Language Models2024· architecture← LLM-judged similar
- GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression2024· architecture← LLM-judged similar
- Transformers are Multi-State RNNs2024· inference-optimization← LLM-judged similar
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression2024· inference-optimization← LLM-judged similar
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference2024· serving← LLM-judged similar
- Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization2025· inference-optimization← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

The Pitfalls of KV Cache Compression2025· inference-optimization→ LLM-judged similar
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction2026· inference-optimization→ LLM-judged similar
- Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models2025· inference-optimization→ in-domain predecessor
- Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost2025· llm-systems→ topic thread
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression2024· inference-optimization→ similar topic
LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs2025· inference-optimization→ in-domain predecessor
- Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation2025· inference-optimization→ in-domain predecessor
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities2025· llm-systems→ topic thread
- Human-inspired Episodic Memory for Infinite Context LLMs2024· inference-optimization→ similar topic
- LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference2024· inference-optimization→ similar topic
- Writing in the Margins: Better Inference Pattern for Long Context Retrieval2024· inference-optimization→ similar topic

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: KV Cache Size Is the Real Bottleneck for Inference-Time Scaling

Why This Problem Matters Now

Where Prior Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Eviction Decisions: From Hidden States to Binary Flags

Parameter Implementation: Borrowing a Neuron

Constructing the Attention Mask from Eviction Decisions

Training Objective: Distillation Plus Compression Loss

Compression Ratio Annealing Schedule

Training Hyperparameters and Data

Delayed Eviction: The Central Mechanism

DMS During Inference

Sparse Attention Mask: Training vs. Inference Gap

Inference-Time Hyper-Scaling Evaluation Framework

4. Key Insights and Innovations

Innovation 1: Reframing Inference-Time Scaling as a Memory Bandwidth Problem, Not a Token Budget Problem

Innovation 2: Delayed Eviction as a Learned Anticipatory Mechanism

Innovation 3: Empirical Proof That Sparse Attention Alone Suffices for High-Ratio Compression

Innovation 4: The Sliding Window as a Learnable Buffer, Not a Fixed Architectural Constraint

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Inference-Time Hyper-Scaling Results (Accuracy vs. Memory Reads)

Inference-Time Hyper-Scaling Results (Accuracy vs. Peak Memory)

Throughput Results

DMS for General-Purpose Use (Non-Scaling Evaluation)

Ablation Studies and Robustness Checks

Critical Assessment

Does the evidence support the central claim that KV cache compression improves inference-time scaling?

Are the efficiency metrics truly implementation-agnostic?

Is the training cost truly negligible?

Does DMS genuinely "surpass" vanilla models on long-context tasks?

How robust are the results to the specific prompt format and evaluation protocol?

What's missing?

Summary assessment

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation and Configuration Selection Are Not Addressed

Limitation 2: Training Requires Full Model Access and Non-Trivial Compute

Limitation 3: Verification Is Limited to a Narrow Set of Reasoning Benchmarks and Model Families

Limitation 4: The Sliding Window Size Is a Critical Hyperparameter with No Automated Selection Method

Limitation 5: The Latency and Throughput Measurements Use Non-Optimized Implementations

Limitation 6: No Combination with Process Reward Models or Advanced Search Strategies

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Rectified Sparse Attention

Beyond the Buzz: A Pragmatic Take on Inference Disaggregation