A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

ArXiv: 2406.11430

🎯 Pitch

A key’s L2 norm strongly predicts how much attention it will receive—the smaller the norm, the higher the score. Exploiting this, the authors slash the KV cache by up to 90% without any retraining or even computing attention weights, while staying fully compatible with FlashAttention.

---

1. Executive Summary

This paper introduces a simple L2 norm-based strategy for KV Cache compression that requires no model fine-tuning or attention score computation, making it immediately applicable to any decoder-only Transformer. Analyzing attention distributions in Llama 2, Llama 3, and Gemma models (up to 8B parameters) on language modeling, needle-in-a-haystack, passkey retrieval, and LongBench tasks, the authors discover a strong correlation between the L2 norm of key embeddings and attention scores—a low L2 norm of a key embedding usually leads to a high attention score during decoding—enabling compression by retaining only keys with the lowest L2 norm and their corresponding values. This simple heuristic reduces KV Cache size by 50% on language modeling and needle-in-a-haystack tasks and by 90% on passkey retrieval without accuracy loss, while maintaining FlashAttention compatibility that attention-score-based methods lack. The effectiveness of this compression depends on which layers are compressed, establishing that skipping the first layer is critical for preserving performance while the correlation between L2 norm and attention scores remains consistent across most layers and a wide range of inputs.

2. Context and Motivation

The Core Problem: KV Cache Memory Bottleneck in Long-Context LLM Inference

The fundamental challenge this paper tackles is the memory bottleneck created by the KV Cache during long-context autoregressive inference. To understand why this matters, we need to walk through what the KV Cache does and why it becomes problematic as context length grows.

During autoregressive generation—where a language model produces tokens one at a time—every new token must attend to all previously generated tokens. Without any optimization, this means recomputing the key ($K$) and value ($V$) projections for every past token at every single generation step. For a sequence of length $n$ with an embedding dimension $d$, the attention computation alone scales as $O(n^2 d)$, and the redundant recomputation of $K$ and $V$ for past tokens compounds this cost dramatically.

The KV Cache solves this by storing the key and value projections of all past tokens in memory after they are first computed. When generating token $t+1$, the model simply retrieves $K_{1:t}$ and $V_{1:t}$ from the cache, computes $Q_{t+1}$ for the current token, and performs attention without recalculating anything for earlier positions. This turns a quadratic recomputation problem into a linear memory storage problem.

But that linear memory storage problem is exactly where the bottleneck shifts. For a model with $L$ layers, $H$ attention heads, and a sequence of length $n$, the total KV Cache memory requirement is:

$L \times H \times n \times d_k \times 2 \times \text{precision}$

where $d_k$ is the per-head key/value dimension, the factor of 2 accounts for storing both keys and values, and precision is the number of bytes per floating-point value (e.g., 2 bytes for FP16). As the paper notes in Section 2, this memory grows linearly with sequence length—and with modern models pushing to 32K, 80K, 128K, or even 1M+ token contexts, this linear growth becomes a severe practical constraint.

There are actually two distinct costs imposed by a large KV Cache, and the paper is careful to identify both:

1. Memory capacity: The raw number of bytes consumed. On memory-constrained hardware (edge devices, consumer GPUs), the KV Cache for a long sequence can exceed available high-bandwidth memory (HBM) entirely, making inference impossible.

2. Memory bandwidth during decoding: Even if the KV Cache fits in HBM, every single token generation step requires reading the entire KV Cache from HBM to the streaming multiprocessor (SM). The paper explicitly cites Fu [2024] on this point: the decoding phase becomes bandwidth-bound, not compute-bound. If the KV Cache is large, each token generation stalls on memory transfers, producing unacceptable latency for real-time applications.

This second point is subtle but critical. It means that compressing the KV Cache doesn't just save memory capacity—it directly reduces decoding latency by shrinking the amount of data that must be transferred from HBM to SM at each step. For interactive applications (chatbots, coding assistants, real-time translation), this latency reduction is arguably more important than the raw memory savings.

Why This Problem Is Pressing Now

The paper is situated at a moment when the tension between model capabilities and deployment feasibility has become acute. Commercial and open-source models (GPT-4, Claude-3, Gemini-Pro-1.5, Llama 3, etc.) now routinely support context windows of 32K to 1M+ tokens, enabling tasks that require synthesizing information from entire documents, multi-turn conversations, codebases, or video transcripts. But the hardware costs of these long contexts are prohibitive for widespread deployment—especially on consumer devices or in cost-sensitive production environments where latency and throughput directly translate to user experience and infrastructure bills.

The paper frames this tension explicitly in Section 1:

"The practical deployment of LLMs is frequently hindered by hardware limitations."

This isn't a theoretical concern. It affects concrete decisions about whether to deploy a 7B model on-device or require cloud inference, whether to support 128K context in a free-tier product, or whether to batch many user requests together (which multiplies the KV Cache memory requirement by batch size). Any technique that reduces the KV Cache footprint by 50% or more without degrading model quality directly expands the envelope of what's deployable.

The Prior Approaches Landscape

The paper organizes existing KV Cache compression methods into two broad categories (Section 1 and Section 6), and identifying the limitations of each is essential to understanding the paper's contribution:

Trainable Approaches (Require Model Modification)

These methods alter the model architecture or require additional training:

• Dynamic Memory Compression (DMC; Nawrot et al., 2024): Dynamically merges tokens in the KV Cache, reducing the effective sequence length. The critical limitation: it requires continual pre-training of the model, which is computationally expensive and may not be feasible for practitioners who only have access to model weights, not training infrastructure.

• Generalized Query Attention (GQA; Ainslie et al., 2023): Reduces the number of KV heads by sharing them across multiple query heads, thereby shrinking the cache size by a structural factor. The limitation: this is an architectural change that must be designed in at training time. It cannot be retrofitted to existing models.

The common weakness of trainable approaches is the barrier to adoption: if you have a pre-trained Llama 2 or Llama 3 model and want to reduce its KV Cache footprint, you cannot apply these methods without retraining. For the vast majority of practitioners who consume pre-trained models rather than train them from scratch, trainable methods are not actionable.

Non-Trainable Approaches (Post-Hoc Compression)

These methods apply compression at inference time without modifying the model weights. This is where the paper's method lives, and where it needs to establish a clear advantage:

• H2O (Zhang et al., 2024b): Identifies "Heavy Hitter" KV pairs by accumulating attention scores across all queries. KV pairs that consistently receive high attention are retained; others are evicted. The mechanism is effective but has a critical dependency: it requires computing and inspecting full attention scores to make eviction decisions.

• FastGen (Ge et al., 2023a): Exploits the observation that different attention heads exhibit different attention patterns (some attend locally, some globally, some to special tokens). It compresses each head's KV Cache according to its pattern type. Again, the limitation: requires attention score computation to determine which pattern each head exhibits and which tokens to retain.

• SnapKV (Li et al., 2024): Selects KV pairs based on attention scores from the user's query (the last token), identifying which past tokens are most relevant to generating the next token. Like H2O and FastGen: depends on attention scores to make compression decisions.

The key limitation that unifies these non-trainable methods is buried in a single crucial sentence in Section 1:

"post-hoc compression algorithms usually evict KV pairs based on attention scores, which is not compatible with FlashAttention [Dao et al., 2022] and thus prevents their applications in modern LLMs inference systems."

This is a practical showstopper that deserves careful explanation. FlashAttention is not just one of many attention implementations—it has become the de facto standard for efficient attention in modern LLM inference and training. It achieves its speed by fusing the attention computation into a single GPU kernel that reads and writes in carefully tiled blocks, never materializing the full $n \times n$ attention matrix in HBM. The catch: FlashAttention never exposes the computed attention scores to the user. The softmax-normalized attention weights are consumed internally within the kernel; by the time the attention output is produced, the intermediate scores are gone.

For compression methods that need to inspect attention scores to decide which KV pairs to evict (H2O, FastGen, SnapKV), this creates an irreconcilable conflict. You can either:

• Use FlashAttention (fast, memory-efficient, but no access to attention scores—so no compression).
• Use standard attention (access to attention scores for compression, but the slowdown and memory cost from materializing the full attention matrix largely defeats the purpose of compressing the KV Cache in the first place).

The paper's framing here is sharp: the very systems that need KV Cache compression the most (production inference systems running on modern GPU hardware with FlashAttention) are the ones where existing compression methods cannot be applied. This is the gap that the paper's L2 norm-based method fills. Because the compression decision is based solely on the L2 norm of key embeddings—which are computed and available before the attention operation—the method never needs access to attention scores. It is fully compatible with FlashAttention, as the paper emphasizes in both the abstract and Section 1.

Other Efficiency Approaches Not Directly Competing

The paper mentions related lines of work that address different aspects of the inference efficiency problem, to position itself clearly:

• Memory management systems like PageAttention (Kwon et al., 2023), Infinite-LLM (Lin et al., 2024), and vAttention (Prabhu et al., 2024) focus on reducing I/O overhead through better KV Cache memory allocation, paging, and scheduling. They treat the cache contents as fixed; they optimize how it's stored and accessed, not how much is stored. These are complementary—a paging system could manage a compressed KV Cache—but they don't reduce the fundamental memory footprint.

• StreamingLLM (Xiao et al., 2024) identifies "attention sink" tokens—specific tokens (like the initial BOS token) that receive disproportionately high attention scores regardless of their semantic relevance. The paper explicitly references this work because it's a key precursor to the L2 norm insight. Xiao et al. observed the phenomenon of peaked attention distributions, but used it for a different purpose (enabling infinite-length streaming by retaining sink tokens + recent tokens). The present paper builds on this observation but takes it in a different direction: using a structural property of the embeddings (L2 norm) to identify which tokens to keep, rather than using attention scores or fixed positions.

The Paper's Positioning: A Simpler, More Practical Approach

The paper positions itself through a specific contrast with prior work along three dimensions:

1. No training required. Unlike DMC and GQA, this method applies to any off-the-shelf decoder-only Transformer without additional training, fine-tuning, or architectural modification. This is a practical advantage that lowers the barrier to adoption substantially.

2. No attention score dependency. Unlike H2O, FastGen, and SnapKV, the compression decision is made based solely on the L2 norm of key embeddings—a quantity that is computed during the forward pass before attention and is available without any FlashAttention compatibility issues. The paper's title ("A Simple and Effective L2 Norm-Based Strategy") emphasizes this simplicity as a feature, not a bug.

3. Empirically grounded in an observed correlation. The method emerges from analysis, not invention. The paper doesn't propose a new objective function or optimization procedure; it observes a pattern in how models allocate attention (Section 3), verifies the pattern quantitatively through the attention loss ratio (ALR) metric, and then exploits that pattern for compression. This distinguishes it from methods that make assumptions about what constitutes an "important" token (e.g., recency bias, local attention patterns) without verifying those assumptions empirically.

The paper also positions itself relative to Darcet et al. (2024), which found that hidden states with high L2 norm aggregate important and global information in Vision Transformers. The present paper's finding goes in the opposite direction for attention keys: low L2 norm of key embeddings correlates with high attention scores. This is not a contradiction but a domain difference—Darcet et al. studied the norm of hidden states (which influence information content), while this paper studies the norm of key embeddings specifically (which influence how much a token is attended to). The paper explicitly notes this distinction:

"Previous work [Darcet et al., 2024] finds the hidden states with high L2 norm usually aggregate more important and global information. On the other hand, our findings indicate that a low L2 norm of key embedding generally results in a high attention score."

The Observation That Drives Everything

The paper's motivating observation is presented in Figure 1 and elaborated in Section 3. When visualizing attention distributions alongside the L2 norm of key embeddings for specific heads in Llama 2-7B, a clear and surprising pattern emerges:

Tokens with the highest attention scores—such as the beginning-of-sequence token <s> and punctuation marks like .—consistently have the lowest L2 norm values for their key embeddings.

This isn't a phenomenon the authors designed or engineered. They observed it, and then asked: can we exploit this to compress the KV Cache? The correlation is visually striking in Figure 1, where the attention score heatmap (top row) and L2 norm values (bottom row) show an almost inverse relationship across multiple heads at layer 9.

The paper introduces the Attention Loss Ratio (ALR) to quantify how closely L2 norm-based compression approximates the ideal (but practically unavailable) attention score-based compression. For a given layer $l$ and head $h$, dropping $m$ KV pairs produces an attention loss $L^m_{l,h}$—the sum of attention scores for the dropped tokens. The reference $L^{m,\text{ref}}_{l,h}$ is the attention loss from dropping the $m$ tokens with the lowest attention scores (the ideal compression). The difference $Y^m_{l,h} = L^m_{l,h} - L^{m,\text{ref}}_{l,h}$ measures how much worse L2 norm-based eviction is than the ideal. Summing over all $m$ gives the ALR $Y_{l,h}$. A lower ALR means L2 norm-based compression closely tracks the ideal.

Figure 2 shows the ALR heatmap across layers and heads for Llama 2-7B and Llama 2-7B-32K. The critical finding: most layers have very low ALR, meaning L2 norm-based eviction is nearly as good as attention-score-based eviction. The exceptions are layers 0–1 and some middle layers (around layer 12), which show higher ALR—indicating weaker correlation. This analysis directly motivates the paper's practice of skipping compression on the first two layers, since the L2 norm signal is less informative there.

Reconciling With Prior Observations About Attention Sinks

The connection to attention sink research (Xiao et al., 2024) is important for understanding why this method works. Xiao et al. observed that certain tokens act as "attention sinks"—they capture a large fraction of total attention mass regardless of their semantic relevance. These sink tokens (often the initial token or delimiter tokens) serve a structural role in the attention mechanism: they provide a location for the model to "dump" excess attention mass that would otherwise be distributed across many tokens, which helps stabilize the softmax computation.

What this paper adds is a mechanistic hypothesis for why certain tokens become attention sinks: their key embeddings have low L2 norm. Section 5 explores this connection by examining the embedding structure of low-L2-norm tokens. The authors find that these embeddings are sparse—only a few dimensions have high activation values while most are near zero. This sparsity means the embeddings occupy a narrow subspace, creating a direction in the embedding space that many queries align with, regardless of their specific content. The result: these tokens receive high attention scores from a wide range of queries.

This provides a satisfying conceptual bridge: the L2 norm is not just a heuristic that happens to work; it reflects an underlying geometric property of the key embedding space that causes certain tokens to become attention sinks. The paper supports this with a perturbation experiment (Figure 9): zeroing out the specific high-activation dimensions in low-norm key embeddings significantly alters the attention map, while zeroing random dimensions does not. This demonstrates a causal relationship, not just a correlation.

Cancedda (2024) offers a complementary perspective, suggesting that attention sinks operate through a "dark subspace" in the embedding space—dimensions that are specifically allocated for structural attention routing rather than content-based attention. The sparse activation patterns observed in this paper's analysis (Figure 8, Appendix D) are consistent with this "dark subspace" hypothesis: certain dimensions appear specialized for creating high-attention tokens, and the L2 norm serves as a proxy for whether a token's embedding substantially projects onto these dimensions.

Summary of the Motivating Gap

The paper addresses a specific, well-scoped gap in the KV Cache compression literature: there exists no compression method that is simultaneously (a) training-free, (b) attention-score-free (and thus FlashAttention-compatible), and (c) effective across multiple model families and task types.

Prior non-trainable methods (H2O, FastGen, SnapKV) satisfy (a) and (c) but fail (b). Prior trainable methods (DMC, GQA) satisfy (b) and (c) but fail (a). Memory management systems (PageAttention) don't compress the cache at all—they optimize its layout.

The paper's contribution is filling this gap with a method that satisfies all three criteria, grounded in an empirical observation about the relationship between key embedding geometry and attention allocation. Whether the method is sufficiently effective compared to attention-score-based alternatives is the question the experiments are designed to answer—and the paper positions the FlashAttention compatibility as a practical advantage that may justify slightly lower compression quality (if any exists) in many deployment scenarios.

3. Technical Approach

3.1 Reader Orientation

The paper builds a post-hoc KV Cache compression system for decoder-only Transformer-based LLMs that identifies and retains the most important key-value pairs during inference. The system solves the memory bottleneck problem by using a single, cheap-to-compute property—the L2 norm of key embeddings—as a proxy for token importance, enabling 50–90% cache size reduction without accessing attention scores or modifying the model, thereby maintaining compatibility with FlashAttention.

3.2 Big-Picture Architecture (Diagram in Words)

The compression system has three major components:

1. Key Embedding Extractor — during the pre-filling phase (processing the input prompt), for each layer and each attention head, the model computes key embeddings $K = XW_K$ as part of its normal forward pass. The extractor captures these key embeddings—which are already being computed anyway—and passes them to the scoring component.

2. L2 Norm Scorer — for each token's key embedding in each head of each layer, this component computes the Euclidean (L2) norm: $||\mathbf{k}||_2 = \sqrt{\sum_i k_i^2}$. This produces a single scalar per token per head per layer that serves as the compression signal. Tokens with higher L2 norm are candidates for eviction; tokens with lower L2 norm are retained.

3. Layer-Aware Eviction Policy — this component decides which layers to compress and how many tokens to evict per layer. Based on the empirical finding that the first two layers show weak correlation between L2 norm and attention importance (quantified by the Attention Loss Ratio in Figure 2), the policy skips compression entirely on those layers. For the remaining layers, when the KV Cache reaches a pre-defined maximum size, it discards the $m$ tokens with the highest key L2 norm and retains the corresponding values for the $n - m$ retained tokens (where $n$ is the current sequence length and $m$ is determined by the target compression ratio).

Information flows linearly: input prompt → forward pass through Transformer layers → at each layer, key embeddings are computed normally → L2 norms are calculated → when cache size exceeds threshold, highest-L2-norm pairs are evicted → attention proceeds using only retained KV pairs.

3.3 Roadmap for the Deep Dive

• First, the formal definition of attention and the KV Cache (from Section 2 of the paper), since understanding exactly what gets stored and how it is used is prerequisite to understanding what gets compressed.
• Second, the Attention Loss Ratio (ALR) metric that quantifies how well L2 norm-based compression approximates the ideal (but practically unavailable) attention-score-based compression. This is the diagnostic tool that validates the approach before any downstream task evaluation.
• Third, the compression algorithm itself—the step-by-step mechanism of selecting and evicting KV pairs based on L2 norm, including the critical design choice of which layers to compress and which to skip.
• Fourth, the hypothesized mechanistic explanation for why the L2 norm correlates with attention scores (sparse key embeddings, attention sink alignment, the "dark subspace" concept).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an observational and empirical paper whose core idea is that the L2 norm of key embeddings can substitute for attention scores as a token-importance signal for KV Cache compression, based on the discovery of a consistent inverse correlation between the L2 norm of a key embedding and the attention score it receives during decoding.

---

Background: Attention and the KV Cache

Before explaining the compression method, we need precision about what is being compressed and why it exists.

Token embeddings to projections. Given an input sequence represented as a tensor $X \in \mathbb{R}^{n \times d}$ (where $n$ is the sequence length and $d$ is the token embedding dimension), each Transformer layer computes three projections for multi-head attention:

$Q = XW_Q, \quad K = XW_K, \quad V = XW_V$

where $W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}$ are learned projection matrices, and $d_k$ is the dimensionality of the query and key vectors (typically $d_k = d/H$, where $H$ is the number of attention heads).

What each projection does. The query matrix $Q$ represents what each token position is "looking for" in the sequence. The key matrix $K$ represents what each token position "advertises" about itself—a signature that queries compare against. The value matrix $V$ represents the actual information content of each token that gets aggregated into the attention output. In the compression context, the paper's key insight is about $K$: the L2 norm of each key vector encodes information about how much attention that token will attract, independent of what any specific query is looking for.

Scaled dot-product attention. The attention output for a single head is:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

where $QK^T \in \mathbb{R}^{n \times n}$ computes pairwise similarity scores between every query and every key, $\sqrt{d_k}$ is a scaling factor to prevent the dot products from growing too large in magnitude (which would push the softmax into saturated regions where gradients vanish), and the softmax normalizes the scores into a probability distribution over keys for each query.

Why the scaling factor matters for the compression observation. The softmax normalization means that attention scores are a zero-sum game: for each query position, the scores over all keys must sum to 1. This implies that if some tokens receive disproportionately high attention (the low-L2-norm tokens), all other tokens receive proportionally less. The compression strategy exploits this: by retaining the tokens that attract high attention and discarding those that attract low attention, the distribution of total attention mass over retained tokens remains nearly unchanged, minimizing the impact on the model's output.

Multi-head attention. This process runs $H$ independent times in parallel, each with different learned projection matrices $W_Q^{(h)}, W_K^{(h)}, W_V^{(h)}$, producing $H$ separate attention outputs. These are concatenated and projected back to dimension $d$:

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_H)W_O$

where $\text{head}_h = \text{Attention}(Q^{(h)}, K^{(h)}, V^{(h)})$, and $W_O \in \mathbb{R}^{H d_k \times d}$ is a final learned projection.

Why multi-head attention matters for compression decisions. Different heads attend to different aspects of the input—some to local context, some to syntactic structure, some to semantic content, some to delimiter tokens. The L2 norm of key embeddings can vary across heads for the same token because each head has its own $W_K^{(h)}$ projection. This means compression decisions must be made per head: a token that has high L2 norm in head 3 might have low L2 norm in head 7, so it might be evicted from head 3's KV Cache but retained in head 7's. The paper compresses independently per head, per layer.

The KV Cache mechanism. During autoregressive generation, when producing token $t+1$:

$\text{Attention}(Q_{t+1}, [K_{1:t}; K_{t+1}], [V_{1:t}; V_{t+1}])$

where $[;]$ denotes concatenation along the sequence dimension. The key insight for efficiency: $K_{1:t}$ and $V_{1:t}$ for all previous tokens have already been computed and can be retrieved from the cache rather than recomputed. Only $Q_{t+1}$, $K_{t+1}$, and $V_{t+1}$ need to be freshly computed.

Memory cost. As established in the prior section, the total KV Cache memory for a model with $L$ layers, $H$ heads, sequence length $n$, and key/value dimension $d_k$ is:

$L \times H \times n \times d_k \times 2 \times \text{precision bytes}$

For a concrete example: Llama 2-7B has $L = 32$ layers, $H = 32$ heads, $d_k = 128$. With $n = 4096$ tokens and FP16 precision (2 bytes): $32 \times 32 \times 4096 \times 128 \times 2 \times 2 = 2.15$ GB for the KV Cache alone. At $n = 32768$, this grows to approximately 17.2 GB—exceeding the memory of many consumer GPUs. This is what compression addresses.

---

The Attention Loss Ratio (ALR): Quantifying the L2 Norm–Attention Correlation

Before describing the compression algorithm, the paper introduces a diagnostic metric to quantify how good a proxy the L2 norm is for attention scores. This is necessary because the method's validity rests entirely on this correlation; if the correlation were weak, the compression would be arbitrary and harmful.

Defining attention loss from compression. When $m$ KV pairs are evicted from the cache for a given layer $l$ and head $h$, the model can no longer attend to those positions. The attention loss $L^m_{l,h}$ is defined as the sum of attention scores that the current query would have assigned to the evicted tokens:

$L^m_{l,h} = \sum_{p \in D_{l,h}} a_{l,h,p}$

where $D_{l,h}$ is the set of $m$ evicted positions (with $|D_{l,h}| = m$), and $a_{l,h,p}$ is the attention score that the query token assigns to position $p$ in layer $l$, head $h$.

What this equation computes. For a given head and layer, when we decide to drop $m$ tokens from the KV Cache, we sum up the attention scores that the next generated token would have assigned to those dropped tokens. A perfect compression algorithm would make this sum as small as possible—it would drop tokens that the model is barely attending to anyway. An upper bound: if we drop $m$ tokens completely at random, the expected attention loss is $m/n$ (since total attention mass sums to 1). A good compression method should achieve attention loss substantially below this random baseline.

Why attention loss matters as a metric. The attention output for a head is a weighted sum of value vectors: $\sum_{p} a_p \cdot \mathbf{v}_p$. If tokens with very small $a_p$ are dropped, the weighted sum changes only slightly—the missing contribution is negligible. The attention loss directly measures the magnitude of the perturbation to the attention output. Low attention loss means the compression minimally disturbs what the head computes.

The reference compression (ideal but unavailable). To know how good L2 norm-based compression is, we need a baseline for the best possible compression. The paper defines the reference attention loss $L^{m,\text{ref}}_{l,h}$ as the attention loss achieved by an oracle that knows the attention scores in advance and evicts the $m$ tokens with the lowest attention scores:

$L^{m,\text{ref}}_{l,h} = \text{sum of the } m \text{ smallest attention scores in head } h, \text{ layer } l$

This oracle is unavailable in practice because it requires computing attention scores before deciding what to compress—defeating the purpose, since computing full attention is what we want to avoid, and is incompatible with FlashAttention.

The gap between L2 norm-based and ideal compression. The difference between the attention loss of L2 norm-based compression and the ideal reference is:

$Y^m_{l,h} = L^m_{l,h} - L^{m,\text{ref}}_{l,h}$

where $Y^m_{l,h}$ is a non-negative number (since $L^{m,\text{ref}}_{l,h}$ is the minimum achievable attention loss for $m$ evictions).

What this equation computes. For a specific number of evicted tokens $m$, it measures the excess attention loss incurred by using L2 norm for eviction decisions instead of using the actual attention scores. If L2 norm perfectly predicted attention rank order, $Y^m_{l,h}$ would be zero—the two methods would evict exactly the same tokens. The larger $Y^m_{l,h}$, the worse L2 norm is as a proxy for attention importance at that specific compression level.

Why this form, not correlation coefficient. A simple Pearson or Spearman correlation between L2 norm and attention scores would tell us whether the relationship is monotonic on average, but it would not tell us whether the ranking of tokens by L2 norm matches the ranking by attention score specifically for the tokens at the boundary between retention and eviction. A method that perfectly ranks the top-10 highest-attention tokens but scrambles the bottom 100 would have high correlation but poor compression performance, because eviction decisions depend on identifying the least important tokens. The attention loss gap directly measures what matters: when we drop the $m$ highest-L2-norm tokens, how much attention mass do we lose compared to the oracle?

The Attention Loss Ratio (ALR) aggregated across all eviction levels. To get a single summary statistic per head and layer, the paper sums $Y^m_{l,h}$ over all possible numbers of evicted tokens $m$ from 1 to $n$:

$Y_{l,h} = \sum_{m=1}^{n} Y^m_{l,h}$

where $Y_{l,h}$ is named the Attention Loss Ratio (ALR) for layer $l$, head $h$.

What this equation computes. It aggregates the cumulative excess attention loss when using L2 norm instead of ideal attention scores, integrated over all possible compression ratios from "evict one token" to "evict all tokens." A low ALR means that across the full range of possible cache sizes, L2 norm-based eviction tracks the oracle closely. A high ALR means that for some compression ratios, L2 norm makes substantially worse eviction decisions—it drops tokens that the oracle would have kept, or keeps tokens the oracle would have dropped.

Why sum over all $m$ rather than at a single compression ratio. The optimal compression ratio for deployment is not known in advance—it depends on hardware constraints, latency requirements, and the specific task. A method that works well at 10% compression but poorly at 50% compression is less useful than one that tracks the oracle consistently across all ratios. The summed ALR rewards methods that maintain rank correlation throughout the entire distribution, not just at the extremes.

Results of the ALR analysis (Figure 2). The ALR heatmap in Figure 2 reveals three distinct patterns:

• Most layers and heads (the bulk of the heatmap): Very low ALR (dark blue regions), indicating that L2 norm-based eviction nearly matches the oracle. These are layers where compression is safe and effective.
• Layers 0–1 (the first two layers): Consistently high ALR across most heads, indicating that L2 norm is a poor proxy for attention importance in the earliest layers. The paper hypothesizes that early-layer attention patterns are more content-dependent and less dominated by structural tokens (sinks), making the L2 norm signal less informative.
• Some middle layers (especially around layer 12 in Llama 2-7B): Sporadic high ALR in specific heads, suggesting that some mid-network heads also have weak L2 norm–attention correlation.

This analysis directly motivates the paper's practice of skipping compression on the first two layers (and optionally layer 12), since compressing layers with high ALR would evict the wrong tokens—dropping high-attention tokens and retaining low-attention ones—causing substantial perturbation to the model's computation.

The paper also compares models. Figure 2 shows ALR heatmaps for both Llama 2-7B (left) and Llama 2-7B-32K (right), a long-context fine-tuned variant. The patterns are "quite consistent across all the models" (Section 5), suggesting that the L2 norm–attention correlation is a structural property of the Transformer architecture rather than an artifact of a specific training run or context length.

---

The L2 Norm-Based Compression Algorithm

Overview. The compression algorithm operates during autoregressive generation. During the pre-filling phase (processing the input prompt), the KV Cache accumulates normally until it reaches a pre-defined maximum size (the max_kv hyperparameter). Once the cache size exceeds this threshold, at each subsequent generation step, the algorithm evicts the tokens with the highest L2 norm in their key embeddings, retaining only the max_kv tokens with the lowest L2 norm. The key insight: this eviction decision is made independently per head, per layer, based only on key embeddings that are already computed during the normal forward pass.

Step-by-step at inference time. For each new token generated:

1. Compute key embeddings normally. For each layer $l$ and each head $h$, compute $K^{(l,h)} = XW_K^{(l,h)}$ as part of the standard forward pass. This produces a key vector $\mathbf{k}_{t}^{(l,h)} \in \mathbb{R}^{d_k}$ for the new token at position $t$.

2. Compute the L2 norm of the new key embedding. For the newly generated token's key embedding in each head of each layer, compute:

$||\mathbf{k}_{t}^{(l,h)}||_2 = \sqrt{\sum_{i=1}^{d_k} (k_{t,i}^{(l,h)})^2}$

This produces a single non-negative scalar per head per layer. The computation is extremely cheap: $d_k$ multiply-add operations (for the squares), $d_k - 1$ additions (for the sum), and one square root—typically $d_k = 128$, so this is negligible compared to the attention computation itself.

3. Append to KV Cache with L2 norm metadata. Store the new key-value pair $(\mathbf{k}_{t}^{(l,h)}, \mathbf{v}_{t}^{(l,h)})$ in the cache for head $h$, layer $l$, along with its L2 norm $||\mathbf{k}_{t}^{(l,h)}||_2$ as metadata.

4. Check cache size against threshold. If the number of stored tokens $n$ exceeds the pre-defined max_kv for that layer, trigger eviction.

5. Select tokens to evict. Sort the stored tokens by their L2 norm metadata in descending order (highest norm first). Select the $n - \text{max\_kv}$ tokens with the highest L2 norm as eviction candidates. The retained set consists of the max_kv tokens with the lowest L2 norm.

6. Remove evicted pairs from the cache. Delete the key and value vectors for the evicted tokens. The cache now contains exactly max_kv pairs.

7. Proceed with attention using the compressed cache. The attention computation for the current token uses only the retained KV pairs: $\text{Attention}(Q_t, K_{\text{retained}}, V_{\text{retained}})$. The evicted tokens are permanently inaccessible to this head and layer for all future generation steps.

Per-head independence. Steps 2–6 execute independently for each attention head within each layer. This means a token could be retained in head 3 (because its key L2 norm in head 3's projection is low) but evicted from head 7 (because its key L2 norm in head 7's projection is high). The paper does not coordinate eviction decisions across heads—each head maintains its own compressed KV Cache based solely on its own key embeddings.

Why per-head independence is necessary and correct. Different heads learn different $W_K$ projection matrices, so they map the same token embedding to different key vectors. A token that contains information relevant to local syntactic structure might have a low-L2-norm key embedding in a head that specialises in local attention, but a high-L2-norm embedding in a head that specialises in global content-based attention. Making a global "evict this token from all heads" decision would discard information that some heads need. Per-head eviction respects the functional specialization of attention heads.

The crucial question: when does eviction start? The algorithm does not begin evicting tokens until the cache reaches max_kv size. During the initial pre-filling phase (when the prompt is being processed and the cache is filling for the first time), all tokens are retained. Only when the sequence length exceeds max_kv does compression activate. This means that for short sequences (shorter than max_kv), the model operates identically to an uncompressed model with no overhead from the compression logic.

What about tokens generated after eviction starts? Each newly generated token is added to the cache with its L2 norm computed. The sorting and eviction step runs after adding the new token. This means that a newly generated token could itself be evicted immediately if its L2 norm is higher than all currently retained tokens. Similarly, a previously retained token that has relatively low L2 norm might eventually be evicted as more tokens accumulate and the "lowest $k$" set changes. The retained set is a sliding window of the globally lowest-L2-norm tokens among all tokens seen so far, not a fixed set determined at the first eviction step.

Why not a fixed retained set from the start? An alternative design would be: at the moment the cache fills, compute which tokens to retain once and keep those fixed forever. This would be incorrect because (a) new tokens are continuously added and might be more important than some originally retained tokens, and (b) the compression ratio would erode as the sequence grows (retaining max_kv tokens from an ever-growing total means an ever-decreasing retention fraction, rather than a fixed budget). The sliding-window approach maintains a constant memory footprint regardless of total sequence length.

The layer-skipping policy. Based on the ALR analysis in Figure 2, the paper introduces a critical design choice: do not compress the first two layers (layers 0 and 1). For these layers, the KV Cache is allowed to grow without bound (or at least to whatever the hardware can support), because compressing them based on L2 norm would cause large attention loss—the L2 norm signal is weak in early layers, so eviction would drop important tokens.

How the skipping decision is operationalized. For each layer $l$:

• If $l \in \{0, 1\}$ (the first two layers): skip all compression. The KV Cache for all heads in these layers retains all tokens.
• Optionally, also skip layer 12 (and any other layers identified as having high ALR in the specific model being used).
• For all other layers: apply the full compression algorithm with the specified max_kv.

Evidence for the necessity of skipping. Appendix B.1 (Figures 16a–16d) shows ablation experiments comparing different skip configurations. The key finding: "only skipping the first layer (layer-0) decreases the performance on the needle-in-a-haystack task significantly" (Appendix B.1). Skipping layers 0 and 1 (or 0, 1, and 2) restores performance to near-uncompressed levels. On passkey retrieval, the effect is even more dramatic: skipping only layer 0 produces a U-shaped accuracy curve with respect to compression ratio (Figure 16b), indicating that some tokens critical for retrieving the passkey are being incorrectly evicted in layer 0. Adding layer 1 to the skip set eliminates this U-shape and yields consistently high accuracy across all compression ratios.

Why early layers are different. The paper does not fully explain this phenomenon but hypothesizes that early layers perform more content-dependent attention, where the L2 norm of key embeddings carries less information about attention importance. In early layers, the model may still be refining its representations from the raw token embeddings, and attention patterns have not yet settled into the stable sink-token-dominated structure observed in middle and late layers. An alternative (complementary) hypothesis: early-layer attention heads may attend based on positional proximity or other structural features that don't map cleanly onto key embedding norm.

The max_kv hyperparameter. This is the central tuning knob of the method. It directly controls the compression ratio: if the sequence length reaches $N$ and max_kv is set to $M$, the effective compression ratio is $(N - M) / N$ (for the compressed layers). The paper experiments with values like max_kv = 2000, max_kv = 1500, max_kv = 1000 for language modeling (Section 4, Figure 3), and sweeps compression ratios from 10% to 90% for the long-context tasks (Figures 4a–4b, 5). The choice of max_kv represents a direct tradeoff: smaller values save more memory and bandwidth but discard more tokens, potentially degrading model quality.

Computational overhead of the compression algorithm. For each token generated, the additional computation consists of:

• Computing the L2 norm for $H \times L_{\text{compressed}}$ key vectors (where $L_{\text{compressed}}$ is the number of layers being compressed). Each norm computation costs approximately $2 d_k$ floating-point operations.
• Maintaining a sorted order of tokens by L2 norm (or finding the top-$k$ highest norms). For a cache of size $M$, finding the $m$ tokens with highest L2 norm among $M + 1$ tokens (after adding the new token) costs $O(M \log M)$ with a full sort, or $O(M)$ with a selection algorithm if only the eviction candidates are needed.

The paper does not provide wall-clock timing comparisons, but the overhead is clearly much lower than computing full attention scores (which costs $O(n^2 d_k)$ for standard attention). The method's practical advantage is that it adds a small, constant-factor overhead to the attention computation rather than requiring a fundamentally more expensive operation.

Comparison with eviction baselines. To validate that low L2 norm specifically—not just any deterministic pattern—identifies important tokens, the paper tests three alternative eviction strategies on language modeling (Figure 3):

• Keep low L2 norm (the proposed method): Evict tokens with highest L2 norm, retain those with lowest L2 norm. This preserves performance closest to the uncompressed baseline.
• Keep random tokens: Evict tokens uniformly at random. Performance degrades substantially—the gap between "keep low norm" and "keep random" quantifies how much better the L2 norm signal is than chance.
• Keep high L2 norm (the inverse strategy): Evict tokens with lowest L2 norm, retain those with highest L2 norm. Performance collapses completely—perplexity spikes dramatically, confirming that low-L2-norm tokens are the important ones. The paper notes that this "impairs performance, even more so than random discarding" (Section 4), which is strong evidence that low-L2-norm tokens are specifically important, not just that deterministic eviction is better than random.

The "keep high norm" ablation is particularly informative: it demonstrates that the L2 norm is not just any signal that happens to correlate with something—it is specifically low L2 norm that marks important tokens. If importance were associated with high L2 norm (as Darcet et al., 2024 found for hidden states in Vision Transformers), the inverse strategy would work well. The fact that it performs worse than random confirms the directionality of the relationship.

---

Mechanistic Hypothesis: Why Low L2 Norm Correlates with High Attention Scores

Having established the empirical correlation and demonstrated its practical utility, the paper offers a mechanistic hypothesis for why this relationship exists. This moves the work beyond pure empiricism toward a structural understanding, though the paper is careful to label this as hypothesis rather than proven fact.

Sparse key embeddings in low-L2-norm tokens. Section 5 examines the key embeddings of tokens with low L2 norm and finds a consistent pattern: these embeddings are sparse—most dimensions have values near zero, while a small number of dimensions (typically 2–5 out of 128) have substantially larger absolute values. Figure 8 illustrates this by plotting the activation values across all 128 dimensions for several tokens. The BOS token <s> shows clear peaks at specific dimensions (~50, ~56, ~120), while content tokens like "political" and "philosophy" show more uniformly distributed activation patterns.

What sparsity means for attention. In the scaled dot-product attention mechanism, the attention score for a key $\mathbf{k}$ given a query $\mathbf{q}$ is proportional to $\mathbf{q}^T \mathbf{k}$ (before softmax normalization). If $\mathbf{k}$ is sparse—meaning it has large values in only a few dimensions—then:

$\mathbf{q}^T \mathbf{k} \approx \sum_{i \in \text{active dimensions}} q_i k_i$

The dot product is dominated by the query's projection onto those few active dimensions. If multiple different queries project positively onto the same few dimensions (the "common direction" hypothesis), then the sparse key will receive high attention scores from a wide range of queries—regardless of what those queries are "looking for" semantically.

The attention sink connection. This sparsity-based mechanism explains why certain tokens become attention sinks (Xiao et al., 2024): their key embeddings occupy a narrow subspace of the embedding space, creating a direction that captures alignment with many different query vectors. The result is that these tokens receive consistently high attention scores from diverse queries, serving as a structural component of the attention mechanism rather than carrying content-specific information. The paper explicitly connects this to Cancedda (2024)'s concept of a "dark subspace"—certain dimensions of the embedding space may be functionally dedicated to creating these high-attention tokens, separate from the dimensions used for content-based attention routing.

The perturbation experiment (Figure 9). To test the causal role of the sparse activation peaks, the paper performs a targeted intervention: zeroing out the specific high-magnitude dimensions in low-L2-norm key embeddings and observing the effect on the attention map. The results (Figure 9) show that zeroing these specific peaked activations dramatically alters the attention distribution—the previously peaked attention on sink tokens disperses. In contrast, zeroing an equal number of random dimensions produces minimal change in the attention map. This demonstrates that the sparse activation dimensions are causally responsible for the high attention scores, not merely correlated with them.

Why this implies low L2 norm is not an accident. If a key embedding uses only a few dimensions with large values, its L2 norm is dominated by those few dimensions: $||\mathbf{k}||_2 = \sqrt{\sum_{i \in \text{active}} k_i^2} \approx \sqrt{\sum_{i \in \text{active}} k_i^2}$. But the total "energy" of the embedding is concentrated rather than spread out. An alternative embedding with the same total squared sum but spread across all 128 dimensions would have the same L2 norm but might not create the sink effect—because no single dimension is large enough to dominate dot products with arbitrary queries. The L2 norm is a proxy for sparsity, not a direct measure of it, but in practice, low L2 norm in these models correlates with the sparse activation pattern that drives sink behavior.

Implications for the compression strategy. This mechanistic picture explains why compressing based on L2 norm works: low-L2-norm tokens are attention sinks that receive high attention regardless of query content. Retaining them preserves the structural attention patterns that the model relies on, while evicting high-L2-norm tokens (which have more uniformly distributed key embeddings and play content-specific roles that vary across queries) minimally disrupts the attention computation for most queries. It also explains why the first two layers are different: in early layers, representations may not have converged to the stable sink-token structure, so the L2 norm–attention correlation is weaker.

What the paper does not explain. The mechanistic hypothesis does not address how the model learns to create these sparse key embeddings. It is plausible that during pretraining, the model discovers that allocating specific embedding dimensions to high-attention structural tokens (BOS, punctuation, delimiters) is an efficient use of the attention budget—it stabilizes training by providing a consistent attention target regardless of input content. But this is speculation; the paper does not investigate the training dynamics.

4. Key Insights and Innovations

Innovation 1: Token Importance Can Be Assessed Before Attention, Not After

The dominant assumption in non-trainable KV Cache compression—codified in methods like H2O (Zhang et al., 2024b), FastGen (Ge et al., 2023a), and SnapKV (Li et al., 2024)—is that you must compute attention scores to know which tokens are important. This assumption has a natural logic: attention scores literally measure how much each past token contributes to the current token's representation, so surely they are the definitive importance signal. The entire pipeline of these methods follows from this assumption: compute full attention → inspect the scores → identify high-scoring tokens → retain those → evict the rest.

This paper makes a conceptual move that breaks this dependency. The key insight is that token importance—measured by eventual attention scores—is encoded in a structural property of the key embeddings themselves, accessible before any query is computed. The L2 norm of a key vector, an intrinsic geometric property of how the model represents each token in key space, predicts which tokens will attract high attention during decoding. This is not an incremental efficiency improvement on existing attention-based methods; it is a reframing of when importance information becomes available in the computation graph.

Why does this matter beyond the specific L2 norm heuristic? Because it opens a new category of compression signals: properties of the cached representations themselves, independent of the queries that will later access them. If the L2 norm works, perhaps other intrinsic properties—sparsity patterns, direction alignment with principal components, distance from the key embedding centroid—also carry importance information. The paper doesn't explore these alternatives, but by establishing the principle that KV pairs carry their own importance metadata in their geometric properties, it creates a template for an entire family of attention-free compression methods.

The evidence for this conceptual shift is not a single experiment but the entire empirical structure of the paper: the consistent inverse correlation between L2 norm and attention scores visible in Figure 1, the low Attention Loss Ratio across most layers in Figure 2, and the catastrophic failure of the "keep high norm" strategy in Figure 3 (where performance collapses below random eviction, confirming that low L2 norm specifically—not just any deterministic pattern—marks the important tokens). This is a fundamental finding about the geometry of learned key representations, not just a new compression trick.

The practical consequence—FlashAttention compatibility—is downstream of this conceptual move. Previous methods couldn't work with FlashAttention because they needed attention scores, which FlashAttention never exposes. By decoupling importance assessment from attention computation, the paper makes compression compatible with the fastest attention implementation available, removing the painful tradeoff between inference speed and memory efficiency that plagued prior work. But the intellectual contribution is upstream of this practical benefit: the demonstration that key embeddings carry a self-contained importance signal.

---

Innovation 2: The Attention Loss Ratio as a Diagnostic Framework for Compression Quality

Prior work on KV Cache compression has largely evaluated methods end-to-end: apply compression, run the model on downstream tasks, check if accuracy holds. This black-box evaluation tells you whether a method works but not where or why. A method that works well on average might fail catastrophically on specific heads or layers, and end-to-end metrics obscure this heterogeneity.

The paper introduces the Attention Loss Ratio (ALR) as a layer-and-head-resolved diagnostic that quantifies how closely any compression signal (here, L2 norm) approximates the ideal but unavailable attention-score-based oracle. The ALR computes the cumulative excess attention loss—the attention mass lost beyond what the oracle would lose—integrated across all possible compression ratios, producing a single scalar per head per layer that can be visualized as a heatmap (Figure 2).

What makes this a genuine innovation rather than just a new metric is that it enables layer-specific policy decisions. The ALR heatmap in Figure 2 reveals that layers 0–1 and some middle layers (around layer 12 in Llama 2-7B) have high ALR—the L2 norm signal is weak there. This directly motivates the paper's practice of skipping compression on those layers, a design choice that proves critical for performance (Appendix B.1, Figures 16a–16d: skipping only layer 0 causes significant accuracy degradation on needle-in-a-haystack; adding layer 1 restores near-baseline performance). Without the ALR analysis, the layer-skipping policy would be a hyperparameter to tune blindly; with it, the policy emerges from a principled measurement of where the compression signal is reliable.

The diagnostic also explains a subtle but important point: the correlation between L2 norm and attention scores varies across the model, and good compression requires respecting this variation. A uniform compression policy applied to all layers would over-compress early layers (where the signal is weak) and under-compress later layers (where the signal is strong and more tokens could be safely evicted). The ALR framework makes this heterogeneity visible and actionable.

This is fundamentally a meta-methodological contribution: it provides a language and a measurement tool for reasoning about why a compression method works, not just that it works. Future compression methods—whether based on alternative geometric properties, learned scorers, or hybrid signals—can use the same ALR framework to diagnose their per-layer reliability and optimize their layer-specific policies. The paper thus contributes not only a specific compression technique but a way of thinking about compression quality that generalizes beyond L2 norm.

---

Innovation 3: The Geometric Mechanism Connecting Key Embedding Sparsity to Attention Sink Behavior

The attention sink phenomenon—certain tokens receiving disproportionately high attention regardless of their semantic relevance—was identified by Xiao et al. (2024) as an empirical regularity in Transformer attention patterns. StreamingLLM exploited this by retaining sink tokens (plus recent tokens) to enable infinite-length generation. But the mechanism remained opaque: why do specific tokens become sinks? Is it a learned optimization artifact, an architectural necessity, or something else?

This paper advances a mechanistic hypothesis grounded in the geometry of key embeddings: attention sink tokens have sparse key embeddings—most dimensions are near zero, with a few dimensions carrying large activation values. When a key vector is sparse in this way, its dot product with any query is dominated by the query's projection onto those few active dimensions. If the active dimensions are chosen (through learning) to align with directions that many different queries project onto positively, the sparse key will receive high attention scores from a wide range of queries, regardless of their content-specific "intent."

The paper supports this hypothesis with a causal intervention (Figure 9): zeroing out the specific high-magnitude dimensions in low-L2-norm key embeddings dramatically alters the attention map, dispersing the previously peaked attention. Zeroing random dimensions of equal number produces minimal change. This demonstrates that the sparse dimensions are causally necessary for high attention scores, not merely correlated—a stronger claim than the observational correlation with L2 norm alone.

What elevates this beyond an interesting mechanistic observation is that it explains why the L2 norm works as a compression signal and provides a theoretical foundation for generalizing the method. The L2 norm is a proxy for sparsity: an embedding that concentrates its energy in a few dimensions will tend to have lower L2 norm than one that distributes the same total squared activation across many dimensions. By retaining low-L2-norm tokens, the compression algorithm is implicitly retaining the sparse, sink-like tokens that serve as structural attention anchors. By evicting high-L2-norm tokens, it is discarding tokens with more uniformly distributed key embeddings that play content-specific, query-dependent roles—and whose absence therefore minimally disrupts attention for most queries.

This connection to Cancedda (2024)'s "dark subspace" concept provides a satisfying conceptual framework: certain dimensions of the learned key embedding space appear specialized for creating structural attention sinks, separate from the dimensions used for content-based attention routing. The L2 norm serves as a cheap-to-compute indicator of whether a token's key embedding substantially projects onto the "dark" sink dimensions versus the content dimensions.

This is a fundamental contribution to understanding Transformer attention geometry, not just a practical trick. It explains why a simple heuristic works, which means practitioners can trust it in new settings (new models, new tasks, new context lengths) rather than treating it as an empirical curiosity that might break under distribution shift. It also suggests future directions: if sparsity is the underlying mechanism, perhaps directly measuring sparsity (e.g., via Gini coefficient or L1/L2 ratio) could provide an even better importance signal than L2 norm alone.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses multiple datasets across different task types:

  • Language modeling: Wikipedia (English) and a code dataset (unspecified, referenced in Appendix A). Results are averaged over 50 chunks from Wikipedia for language modeling experiments.
  • Needle-in-a-haystack: A synthetic task (Kamradt, 2023) where the model must retrieve a specific "needle" sentence embedded at various positions within a long "haystack" of filler text.
  • Passkey retrieval: A synthetic task (Mohtashami and Jaggi, 2023) where a random passkey number is inserted at a specific position in a long context, and the model must output it.
  • LongBench (Zhang et al., 2024a): A suite of long-context understanding tasks, from which the paper selects five subsets: NarrativeQA (Kociský et al., 2018), Qasper (Dasigi et al., 2021), HotpotQA (Yang et al., 2018), 2WikiMQA (Ho et al., 2020), and QMSum (Zhong et al., 2021).

• Base model(s). The paper tests three model families, all decoder-only Transformers:

  • Llama 2-7B (Touvron et al., 2023): The primary analysis model. Several long-context variants are additionally tested: Llama 2-7B-32K (a 32K context model) and Llama 2-7B-80K (an 80K context model from Fu et al., 2024), as well as Llama 2-7B-LongLoRA-32K-ft (Chen et al., 2023), a fine-tuned 32K variant.
  • Llama 3-8B (Dubey et al., 2024) and Llama 3.1-8B: Tested for language modeling, LongBench, and the FastGen comparison.
  • Gemma (Google): Tested for language modeling.

  The choice of models up to 8B parameters is deliberate: these are representative of commonly deployed open-weight models where KV Cache memory constraints are practically relevant. The paper acknowledges in Section 8 (Limitations) that testing is limited to models of this scale and flags larger-model evaluation as future work.

• Metrics.

  • Perplexity (log PPL): For language modeling, measured on held-out text. Lower perplexity indicates better prediction.
  • Next token accuracy: The fraction of tokens correctly predicted, also for language modeling (reported in Appendix A).
  • Overall accuracy: For needle-in-a-haystack and passkey retrieval, the fraction of trials where the model correctly retrieves the target information.
  • LongBench scores: Task-specific metrics as defined by Zhang et al. (2024a) for each subset (F1, ROUGE-L, exact match, etc., depending on the task). The paper reports both per-subset scores and an overall average.
  • Attention Loss (Equation 5): For diagnostic analysis only, defined as the sum of attention scores for evicted KV pairs. Not an evaluation metric but a mechanistic measurement.

• Baselines. The paper tests several eviction strategies to isolate the effect of keeping low L2 norm specifically:

  • No compression: The uncompressed model with full KV Cache. This is the upper bound.
  • Keep low L2 norm (proposed): Evict tokens with highest L2 norm, retain tokens with lowest L2 norm.
  • Keep high L2 norm (inverse): Evict tokens with lowest L2 norm, retain tokens with highest L2 norm. This tests directionality—if low L2 norm marks important tokens, this strategy should perform worse than random.
  • Keep random: Evict tokens uniformly at random. This provides a baseline for whether the L2 norm signal is better than chance.
  • FastGen (Ge et al., 2023a): A popular attention-pattern-based compression method. The paper implements FastGen without using attention scores (i.e., only considering local, punctuation, and special tokens) to enable a FlashAttention-compatible comparison (Figure 6). This is a weakened version of FastGen; the paper notes this explicitly in Section 4.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time. Instead, the "compute budget" is expressed as the compression ratio—the fraction of the KV Cache evicted—or equivalently the keep ratio (the fraction retained). This is reported as a percentage throughout (10%, 30%, 50%, 70%, 90% compression). For language modeling, the constraint is expressed as max_kv, the absolute number of KV pairs retained. The paper's claim is that the method's computational overhead is negligible relative to the attention computation itself, since L2 norm computation requires only $O(d_k)$ operations per key vector versus $O(n^2 d_k)$ for full attention. No wall-clock timing comparisons are provided.

• Cross-validation / statistical protocol. No cross-validation is reported. Language modeling results are averaged over 50 chunks from Wikipedia. The ALR measurements in Figures 2 and 7 are averaged over 1024 chunks of length 1024 from Wikipedia. For the needle-in-a-haystack and passkey retrieval tasks, the paper reports overall accuracy across varying context lengths and insertion positions (detailed heatmaps in Appendix B), but does not report confidence intervals or statistical significance tests.

---

Main Quantitative Results

Language Modeling: Perplexity Is Preserved Even at 50% Compression

The headline result for language modeling appears in Figure 3, which shows perplexity as a function of input sequence length for Llama 2-7B, Llama 3-8B, and Gemma on Wikipedia text. For each model, the experiment lets the KV Cache grow to a pre-defined max_kv limit (2000 tokens for the shown experiments), then begins evicting tokens with the highest L2 norm.

The key finding: evicting up to 50% of the KV Cache does not increase perplexity. Specifically, for Llama 2-7B (Figure 3, left), the "max kv 2000 (keep low norm)" curve tracks the "no_compression" curve almost exactly from input length 0 through approximately 4000 tokens (at which point the cache is holding 2000 retained tokens out of 4000 total, i.e., 50% compression). Only when the input length exceeds the pre-training context length (4096 for Llama 2-7B) does perplexity rise—and it does so for both the uncompressed and compressed models, indicating that the degradation is due to exceeding the model's trained context window, not the compression.

For Llama 3-8B (Figure 3, center) and Gemma (Figure 3, right), the pattern is identical: "keep low norm" tracks the uncompressed baseline closely at a cache size of 2000.

The comparison with alternative eviction strategies within Figure 3 is decisive:

• "Keep high norm" (evicting low-L2-norm tokens, i.e., the inverse of the proposed method) causes perplexity to spike dramatically—substantially worse than random eviction. For Llama 2-7B, log PPL for "keep high norm" reaches approximately 7 at input length 6000, compared to approximately 6 for "keep random" and approximately 5 for "keep low norm" / uncompressed. The paper states: "discarding tokens with low L2 impairs performance, even more so than random discarding, thus highlighting the importance of these low L2 norm keys" (Section 4).
• "Keep random" (random eviction) performs between the two extremes, consistently worse than keeping low norm and consistently better than keeping high norm. This confirms that the L2 norm signal is both directionally correct and more informative than chance.

Figure 6 extends this comparison to FastGen on Llama 3-8B, showing both perplexity and next token accuracy. The proposed method ("max kv 2000 (keep low norm)") matches the uncompressed baseline, while FastGen (using only local, special, and punctuation tokens, without attention scores) shows higher perplexity at the same cache size. The paper states: "Our method still outperforms FastGen with up to 50% KV Cache eviction" (Section 4). This is notable because FastGen normally leverages attention scores; the comparison here is against an attention-free version, making the result fair for FlashAttention-compatible settings.

Additional language modeling results in Appendix A (Figures 10, 11, 12, 13) explore layer-skipping configurations on Llama 2-7B, showing that skipping different combinations of layers (0 only; 0 and 1; 0, 1, and 12) produces similar perplexity and next-token accuracy curves, with differences emerging only at aggressive cache sizes (max_kv = 1000). This suggests that the layer-skipping policy is somewhat robust—exactly which layers are skipped matters less than that the early layers are skipped.

Needle-in-a-Haystack: 99% Accuracy at 50% Compression, Near-Zero When Keeping High Norm

Figure 4a shows overall accuracy on the needle-in-a-haystack task for Llama 2-7B-80K (Fu et al., 2024) across compression ratios from 10% to 90%. The results:

• No compression: Approximately 100% accuracy.
• Keep low norm (proposed): At 10% compression, accuracy remains at ~100%. At 30% compression, accuracy remains "preserved" (per the paper's text; the exact number is not stated but the curve in Figure 4a shows approximately 100%). At 50% compression, accuracy is 99% per the paper's explicit statement in the abstract and Section 4: "maintain 99% accuracy when compressing 50% of the KV Cache." At 70% compression, accuracy drops to approximately 87–88% (read from Figure 4a). At 90% compression, accuracy drops further to approximately 78–80%.
• Keep high norm (inverse): Accuracy collapses to near zero at all compression ratios above 10%. The paper states: "the model cannot answer correctly when keeping only high L2 norm KV pairs, obtaining near zero accuracy" (Section 4).
• Keep random: Accuracy degrades significantly faster than keep low norm, falling to roughly 60% at 50% compression and below 20% at 90% compression (read from Figure 4a).

The sharp separation between "keep low norm" and "keep random" demonstrates that the L2 norm signal is specifically identifying tokens critical for retrieval. The "keep high norm" collapse confirms directionality: the tokens with low L2 norm are the ones carrying the information needed to locate and extract the needle.

Detailed heatmaps in Appendix B (Figures 17, 18) break down needle-in-a-haystack accuracy by both context length (1000 to 40077 tokens) and insertion depth (0% to 100% of the context). Figure 17a (skip layer 0, keep ratio 0.7, i.e., 30% compression) shows an overall score of 0.876, with the primary failure region being when the needle is placed in the first 11% of the context at long context lengths (>20000 tokens). Figure 17b (skip layers 0 and 1, same keep ratio of 0.7) shows an overall score of 0.997—essentially perfect—with no visible failure region. Figure 17c (skip layers 0 and 1, keep ratio 0.8, i.e., 20% compression) achieves a perfect overall score of 1.000.

This confirms that skipping both of the first two layers, not just layer 0, is critical for needle-in-a-haystack performance. The paper's analysis in Appendix B.1 (Figure 16a) quantifies this: "only skipping the first layer (layer-0) decreases the performance on the needle-in-a-haystack task significantly."

Passkey Retrieval: 100% Accuracy Even at 90% Compression

Figure 4b shows overall accuracy on the passkey retrieval task for Llama 2-7B-80K:

• No compression: 100% accuracy.
• Keep low norm (proposed): At 10%, 30%, 50%, 70%, and 90% compression, accuracy remains at 100%. The paper states: "the model can achieve 100% accuracy on the passkey retrieval task even when compressing 90% of the KV Cache" (Section 4).
• Keep high norm: Zero accuracy at all compression ratios.
• Keep random: Accuracy degrades gradually, falling to approximately 85% at 70% compression and approximately 50% at 90% compression (read from Figure 4b).

The passkey task is particularly revealing because the model needs to retain a single number embedded somewhere in a long context. If compression evicts the passkey token or its surrounding context, the model fails. The 100% accuracy at 90% compression with "keep low norm" means that the passkey token consistently has low enough L2 norm to survive eviction—it is consistently classified as "important" by the L2 norm heuristic.

Appendix B.1 (Figures 16b, 16d) explores the effect of layer skipping on passkey retrieval. The key finding: when only layer 0 is skipped, accuracy exhibits a U-shaped curve with respect to compression ratio—accuracy drops at intermediate ratios and recovers at high ratios. The paper notes this is counterintuitive behavior suggesting that "the compression ratio is not proportional to the overall accuracy of models in the passkey retrieval task when we compress the first layer" (Appendix B.1). Skipping layers 0 and 1 eliminates this U-shape and yields consistently high accuracy. Detailed position-dependent accuracy plots (Figures 19, 20) show that with 90% compression and skipping layers 0 and 1, accuracy remains at 100% regardless of where the passkey is inserted (positions from 0 to 30000), while skipping only layer 0 produces visible dips in accuracy for passkeys inserted at early positions.

LongBench: Small Accuracy Degradation at 50% Compression, Collapse for High-Norm Retention

Figure 5 (left) shows overall scores on LongBench subsets for Llama 3.1-8B, while Figure 5 (right) shows results for Llama 2-7B-80K. Both figures display per-subset breakdowns in Appendix B (Figures 21, 22). The pattern is consistent across models and subsets:

• Keep low norm (proposed): At 10% compression, scores track the uncompressed baseline closely. At 30% compression, a small decrease is visible (for Llama 3.1-8B, average scores drop from approximately 48 to approximately 46; read from Figure 5 left). At 50% compression, scores degrade moderately—for Llama 3.1-8B, average scores drop to approximately 42 versus approximately 48 for uncompressed. At 70% and 90% compression, scores decrease more substantially but remain far above the alternative strategies.
• Keep high norm: "Results in almost zero accuracy" (Section 4) at compression ratios above 10% for Llama 2-7B-80K. For Llama 3.1-8B, scores collapse to near zero by 30% compression.
• Keep random: Degrades faster than keep low norm at all compression ratios, falling roughly halfway between keep low norm and keep high norm.

The per-subset breakdowns (Figures 21 and 22) show that the pattern holds across all five LongBench tasks (NarrativeQA, Qasper, HotpotQA, 2WikiMQA, QMSum), though the absolute scores and degradation rates vary by task. For Llama 2-7B-80K on Qasper (Figure 21b), "keep low norm" at 30% compression actually slightly exceeds the no-compression baseline (approximately 33 vs. 31), though the paper does not comment on this. On HotpotQA (Figure 21c), the degradation is steeper: "keep low norm" at 50% drops to approximately 28 from approximately 38 uncompressed.

The key takeaway from LongBench is that the L2 norm heuristic generalizes beyond synthetic retrieval tasks to diverse long-context understanding tasks requiring summarization, multi-hop reasoning, and question answering over long documents. The 50% compression point—where the method works well on language modeling and needle-in-a-haystack—shows modest degradation on LongBench, suggesting that for tasks requiring synthesis of information across the entire context (rather than retrieval of a single fact), more aggressive compression begins to discard useful information.

---

Ablation Studies and Robustness Checks

Layer-skipping configurations for language modeling (Appendix A, Figures 10–13): The paper tests four configurations on Llama 2-7B: skip layer 0 only (Figure 10); skip layers 0 and 1 (Figure 11); skip layers 0, 1, and 12 (Figure 12); and a comparison of all three (Figure 13). The finding: at max_kv values of 2000, 3000, and 4000, all skipping configurations produce nearly identical perplexity and next-token accuracy curves. Differences emerge only at the most aggressive compression (max_kv = 1000), where skipping layers 0, 1, and 12 performs slightly better than skipping only layer 0 or layers 0 and 1. This suggests that for moderate compression ratios, the exact skipping configuration is not highly sensitive—the critical factor is that early layers (especially layer 0) are skipped.

Layer-skipping configurations for needle-in-a-haystack and passkey retrieval (Appendix B.1, Figures 16a–16d): For both Llama 2-7B-80K and Llama 2-7B-LongLoRA-32K-ft, skipping configurations are ablated across skip sets: {0}, {0,1}, {0,1,2}, and {0,1,12}. The finding: "only skipping the first layer (layer-0) decreases the performance on the needle-in-a-haystack task significantly" (Appendix B.1). Skipping layers 0 and 1 restores performance to near-uncompressed levels, and adding layer 2 or layer 12 produces similar results. On passkey retrieval (Figures 16b, 16d), the "skip layer 0 only" configuration produces a U-shaped accuracy curve, while all configurations that include layer 1 in the skip set produce consistently high accuracy. The paper concludes that skipping the first two layers is critical for long-context tasks.

Alternative eviction strategies (Figures 3, 4a, 4b, 5): Across all tasks, "keep high norm" (evicting low-L2-norm tokens) consistently performs worse than random eviction, confirming the directionality of the L2 norm importance signal. "Keep random" consistently underperforms "keep low norm," confirming that the L2 norm provides a genuine improvement over chance. The consistent rank order—keep low norm > keep random > keep high norm—across all models (Llama 2, Llama 3, Gemma) and all task types (language modeling, needle-in-a-haystack, passkey retrieval, LongBench) demonstrates that the finding is not task-specific or model-specific.

FastGen comparison (Figure 6): Comparing against an attention-free version of FastGen on Llama 3-8B language modeling, the proposed method ("max kv 2000 (keep low norm)") matches the uncompressed baseline in both perplexity and next-token accuracy, while FastGen shows visible degradation at the same cache size. The paper explicitly notes the limitation of this comparison: FastGen normally uses attention scores, and the version tested here is weakened to maintain FlashAttention compatibility. This makes it a fair comparison for the FlashAttention-compatible setting but not a comparison against the strongest version of FastGen.

Model architecture variation: The method is tested on three distinct model families (Llama 2, Llama 3, Gemma) with different architectures, training recipes, and tokenizers. The consistency of results across these families suggests robustness to architectural differences. Additionally, both base models (Llama 2-7B) and long-context fine-tuned variants (Llama 2-7B-32K, Llama 2-7B-80K, Llama 2-7B-LongLoRA-32K-ft) are tested, showing that the L2 norm signal persists through long-context adaptation.

Perturbation experiment for mechanistic validation (Figure 9): Zeroing out the specific high-magnitude dimensions in low-L2-norm key embeddings dramatically alters the attention map (dispersing peaked attention), while zeroing equal numbers of random dimensions produces minimal change. This demonstrates a causal relationship between the sparse activation dimensions and attention scores, not merely a correlation.

ALR consistency across model variants (Figure 2, Section 5): The ALR heatmaps for Llama 2-7B and Llama 2-7B-32K show "quite consistent" patterns across models. The paper states: "We can see that patterns are quite consistent across all the models." This suggests that the L2 norm–attention correlation is not a quirk of a specific training run but a structural property.

Skipping compression at different max_kv thresholds (Figures 10–13): For language modeling, the paper sweeps max_kv values of 1000, 1500, 2000, 3000, and 4000. The finding: at moderate cache sizes (2000–4000 for a 4096-context model), the compression method is nearly lossless. Degradation becomes visible at the most aggressive setting (max_kv = 1000, corresponding to ~75% compression for sequences of length 4000+), where even the best layer-skipping configuration shows elevated perplexity.

---

Critical Assessment

Claim 1: "This simple strategy can reduce the KV Cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy" (Abstract)

The language modeling part of this claim is well-supported for the specific condition tested. Figure 3 shows that at max_kv = 2000 (which represents 50% compression only when the sequence length reaches 4000), perplexity tracks the uncompressed baseline for Llama 2-7B, Llama 3-8B, and Gemma. However, the paper's language modeling experiments test only sequences up to ~8000 tokens (at which point max_kv = 2000 represents 75% compression, and degradation is visible). The claim of "50% without losing accuracy" holds specifically for sequences approximately twice the cache size—for much longer sequences relative to the cache budget, the effective compression ratio increases and so does perplexity. This is an inherent property of the fixed-budget approach: "50%" is not a fixed compression ratio but a description of the regime where accuracy is preserved given a specific max_kv relative to sequence length.

The needle-in-a-haystack claim is strongly supported: Figure 4a shows the "keep low norm" curve at approximately 99% accuracy at 50% compression for Llama 2-7B-80K, and the detailed heatmaps (Figures 17b, 17c) show overall scores of 0.997 and 1.000 at 30% and 20% compression respectively (equivalent to 70% and 80% keep ratios). However, this result is specifically for the configuration that skips layers 0 and 1; skipping only layer 0 produces substantially worse performance (overall score 0.876 at 30% compression, Figure 17a). The claim implicitly assumes the layer-skipping policy is applied correctly.

The passkey retrieval claim is supported with a notable qualification: 100% accuracy at 90% compression is shown for Llama 2-7B-80K (Figure 4b), but only when skipping layers 0 and 1. When skipping only layer 0, passkey accuracy exhibits the U-shaped phenomenon (Figure 16b) that the paper itself flags as problematic. So 90% compression works, but is fragile to the layer-skipping configuration.

A limitation not discussed: the passkey task may be an unusually favorable case for this method. The passkey is a single, semantically distinctive token (a number) embedded in filler text. Its key embedding may naturally have properties that the L2 norm captures well. Whether the method achieves 90% compression on tasks requiring retention of multiple dispersed facts across the context—more realistic for many applications—is not tested directly, though the LongBench results (50% compression showing modest degradation) are suggestive that more complex tasks lose more information at aggressive compression ratios.

Claim 2: "This approach remains compatible with FlashAttention, enabling broader applicability" (Abstract)

This claim is about a structural property of the method, not an experimental result, and it is correct by construction. The compression decision uses only key embeddings (computed before attention) and never accesses attention scores. Therefore, FlashAttention's kernel fusion—which hides intermediate attention scores—does not conflict with the method. The paper does not demonstrate FlashAttention compatibility experimentally (no wall-clock benchmarks, no integration with a FlashAttention-based inference engine), but the claim is logically sound: there is no architectural reason the two cannot coexist.

However, the paper does not address a practical integration challenge: FlashAttention operates on contiguous memory blocks for efficiency. Evicting arbitrary tokens from the middle of the cache would fragment the memory, potentially requiring a reorganization step that could negate some of FlashAttention's speed benefits. The paper does not discuss how token eviction would be implemented in a FlashAttention-compatible inference system, which leaves an engineering gap between the conceptual compatibility and practical deployment.

Claim 3: "A low L2 norm of a key embedding usually leads to a high attention score during decoding" (Abstract), and "the influence of a KV pair is potentially determined by the key embedding itself before being queried" (Abstract)

The empirical evidence for the correlation is strong and multi-faceted. Figure 1 provides qualitative visualization. Figure 2 provides quantitative ALR measurements showing low excess attention loss across most layers. Figure 7 provides per-head attention loss curves for specific heads, demonstrating that L2 norm-based eviction closely tracks the oracle for heads with high correlation (layer 7, head 10) and diverges for heads with low correlation (layer 0, head 0). The "keep high norm" ablation (collapsing performance below random) confirms the inverse direction of the relationship.

However, "usually" is doing significant work here. The correlation is strong in most layers but weak in the first two layers and some middle layers (Figure 2). The method works because it skips the layers where the correlation is weak. The claim is accurate as an aggregate statement but obscures the layer-dependent heterogeneity that the paper's own ALR analysis reveals.

More importantly, the paper demonstrates correlation, not that the key embedding determines attention scores independently of the query. The attention score is a function of both key and query: $a \propto \exp(\mathbf{q}^T \mathbf{k} / \sqrt{d_k})$. A low-L2-norm key might receive high attention from most queries (making it a reliable "important" token to retain) but could still receive low attention from a specific query that is orthogonal to its active dimensions. The paper's implicit claim is that in practice, the query-independent component (the key's geometric properties) dominates enough to make the L2 norm a useful importance signal. The perturbation experiment (Figure 9) supports this by showing that modifying the key's sparse dimensions alters attention patterns, but it does not fully isolate the query-independent contribution from the query-dependent contribution.

Additionally, the paper's investigation is limited to the decoding phase: the correlation is observed "during decoding" (Section 1, Section 3). The pre-filling phase—where attention is computed over all prompt tokens simultaneously—may have different attention dynamics, and the L2 norm correlation might not hold in the same way. The paper compresses during both pre-filling (once the cache exceeds max_kv) and decoding but only analyzes attention scores during decoding.

What experiments would strengthen the paper?

1. Wall-clock latency and throughput benchmarks. The paper claims FlashAttention compatibility as a key advantage but provides no timing measurements. A comparison of the proposed method versus an attention-free FastGen versus uncompressed (all with FlashAttention) on tokens-per-second and memory usage would substantiate the practical benefit.

2. Larger model evaluation. All experiments use models of 7–8B parameters. The paper acknowledges this limitation (Section 8). Testing on models in the 13B–70B range would determine whether the L2 norm–attention correlation persists at scale or is specific to smaller models where attention patterns might be less specialized.

3. Comparison against the full (attention-score-using) versions of H2O, FastGen, and SnapKV. The paper compares only against an attention-free FastGen and does not compare against H2O or SnapKV at all. While the claimed advantage is FlashAttention compatibility, a direct comparison with the strongest versions of these methods (accepting the FlashAttention incompatibility as a tradeoff) would help practitioners decide whether the compatibility advantage justifies any potential quality gap. As it stands, we cannot assess whether the L2 norm method's compression quality is comparable to or worse than attention-score-based methods at the same compression ratio.

4. Robustness to distribution shift. All language modeling is on Wikipedia and a code dataset. Would the L2 norm–attention correlation change for domain-specific text (legal, medical, multilingual) or for chat/instruction-formatted text where the token distribution differs substantially from pretraining data? The needle-in-a-haystack and passkey tasks use synthetic filler text whose token distribution may not stress-test this.

5. Ablation on the L2 norm specifically versus other norms. The paper uses L2 norm. Would L1 norm, L∞ norm, or direct sparsity measures (e.g., the fraction of dimensions with activation above a threshold) work equally well or better? The mechanistic hypothesis (Sections 3.4 and 5) suggests sparsity drives the correlation, but the paper only tests L2 norm as the proxy. A brief ablation comparing L1, L2, and a sparsity metric would clarify whether L2 is the right measure or just an adequate proxy.

Missing negative results and potential failure modes

The paper does not explore scenarios where the method might fail:

• Extremely long sequences (100K+ tokens) where even a small max_kv might be insufficient to retain all sink tokens plus content-relevant tokens. The compression strategy relies on retaining the globally lowest-L2-norm tokens, but as the sequence grows, the number of tokens competing for the fixed budget increases. At some sequence length, important content tokens might be evicted in favor of additional low-L2-norm structural tokens.
• Tasks requiring dense retrieval of multiple facts. The LongBench results at 70–90% compression (Figure 5) show significant degradation, but the paper does not analyze which information is lost—whether the method fails by evicting critical factual tokens or by losing the structural context needed to connect them.
• Interaction with Grouped Query Attention (GQA). Llama 2 uses standard multi-head attention, but Llama 3 uses GQA where multiple query heads share a single KV head. The paper tests Llama 3 but does not analyze whether the L2 norm–attention correlation differs under GQA.

Overall, the experimental evidence strongly supports the paper's central empirical claim—that L2 norm of key embeddings is an effective importance signal for KV Cache compression—under the specific conditions tested (7–8B models, the task suite described, compression ratios up to 50–90% depending on the task, with early layers skipped). The paper is transparent about its limitations (model scale, theoretical understanding, per-head variation) and does not overclaim. The primary gap is the absence of comparisons against full attention-score-based methods, which makes it difficult to assess the quality tradeoff for the claimed compatibility advantage.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Compression Gains

The assumption or constraint. The paper's compression algorithm operates online during inference and requires no advance knowledge about the input. However, the diagnostic framework that validates the method—the Attention Loss Ratio (ALR) analysis in Section 3 and Figure 2—and the per-layer skipping policy derived from it rely on offline analysis that the paper does not include in any cost assessment. Crucially, the ALR is computed by comparing L2 norm-based eviction against an oracle that requires full attention scores, averaged over 1024 chunks of 1024 tokens from Wikipedia (Section 5, Appendix E). The paper acknowledges that its analysis of correlation patterns is empirical and does not provide a method for determining the optimal layer-skipping policy without such analysis:

"While our research offers valuable insights, we tested only on relatively small models... Understanding the underlying reasons behind the importance of the L2 norm would require further theoretical exploration and empirical validation." (Section 8, Limitations)

The consequence. The practical deployment path is underspecified. A practitioner with a new model (different architecture, different scale, different training data) does not know which layers to skip without conducting their own ALR analysis—which requires computing full attention scores (incompatible with FlashAttention, defeating the method's primary advantage) and access to a representative corpus for averaging. The paper provides a recipe for Llama 2 (skip layers 0 and 1, optionally skip layer 12) but no principle for generalizing this to arbitrary models. The ALR heatmaps in Figure 2 show that the specific layers with poor correlation vary somewhat even between Llama 2-7B and Llama 2-7B-32K—the pattern is "quite consistent" but not identical. For a model with a different number of layers or a different attention architecture (e.g., Grouped Query Attention), the optimal skip set might be different, and the cost of determining it could be substantial.

What evidence exists in the paper. The paper demonstrates that the layer-skipping policy matters significantly for long-context tasks. Figure 16a shows that skipping only layer 0 versus skipping layers 0 and 1 on needle-in-a-haystack changes the overall score from 0.876 to 0.997 at 30% compression. Figure 16b shows that skipping only layer 0 produces a U-shaped accuracy curve on passkey retrieval—a qualitatively different behavior that would be difficult to diagnose without full attention-based evaluation. Appendix A (Figures 10–13) shows that for language modeling at moderate compression, the exact skipping set is less sensitive, but this result is limited to one model family and one task type.

Mitigation status. The paper does not address this problem. It suggests in Section 8 that future work should "investigate per-head compression ratios to leverage this observation" but does not propose a method for automatically determining which layers to skip or what per-layer compression ratio to use. The ALR framework itself, while valuable as a diagnostic, requires the very attention computation that the method is designed to avoid. This creates a chicken-and-egg problem: to deploy the method optimally on a new model, you need a diagnostic that undermines the method's deployment advantages.

---

The Method Has Not Been Tested on Models Larger Than 8B Parameters

The assumption or constraint. All experiments use models from the 7–8B parameter range: Llama 2-7B and its long-context variants (32K, 80K, LongLoRA-32K-ft), Llama 3-8B, Llama 3.1-8B, and Gemma (which at the time was available up to 7B). The paper explicitly acknowledges this boundary:

"While our research offers valuable insights, we tested only on relatively small models (Llama family and Gemma up to 8 billion parameters). In future work, we will assess our method on larger-scale models to ensure our findings generalize." (Section 8, Limitations)

The consequence. The attention dynamics of larger models may differ in ways that weaken or invalidate the L2 norm–attention correlation. Larger models have more attention heads, deeper layers, and may develop more specialized attention patterns that distribute importance across tokens differently. The sparse key embedding phenomenon that the paper hypothesizes drives the correlation (Section 5, Figure 8) might be an artifact of the embedding dimension being "large enough" relative to the model's representational needs—in much larger models with proportionally wider key dimensions, key embeddings might be less sparse, and the L2 norm signal might weaken. Conversely, larger models might exhibit the phenomenon even more strongly. Without evidence, practitioners cannot confidently deploy this method on models in the 13B–70B+ range, which is precisely where KV Cache memory constraints are most severe and where compression would be most valuable.

What evidence exists in the paper. None beyond the 7–8B scale. The paper tests three model families at this scale and finds consistent results, which is encouraging but insufficient to establish scale-invariance. The ALR analysis in Figure 2 compares two Llama 2 variants at the same parameter count (7B) but different context lengths, not different scales. The finding that patterns are "quite consistent across all the models" (Section 5) refers to models of the same size.

Mitigation status. Acknowledged as future work. The paper does not provide any theoretical argument for why the phenomenon should be scale-invariant, nor does it test on even a single larger model (e.g., Llama 2-13B) as a preliminary probe. The limitation is particularly significant because the strongest use case for KV Cache compression—reducing the memory footprint of large models with long contexts on constrained hardware—is exactly the regime where the method is untested.

---

No Comparison Against Full Attention-Score-Based Compression Methods

The assumption or constraint. The paper's central claimed advantage is FlashAttention compatibility: because the method uses only key embedding L2 norms and never accesses attention scores, it can be deployed alongside FlashAttention's fused kernels. However, the paper evaluates this advantage in absentia—it compares against an intentionally weakened version of FastGen (stripped of its attention-score-dependent components, keeping only local, punctuation, and special tokens) but never compares against the full versions of H2O (Zhang et al., 2024b), FastGen (Ge et al., 2023a), or SnapKV (Li et al., 2024) operating with their standard attention-score-based policies.

The consequence. The paper cannot quantify the compression quality tradeoff that practitioners face. If a practitioner is willing to forgo FlashAttention compatibility (accepting slower inference but still wanting memory savings), they need to know: does the L2 norm method compress as well as attention-score-based methods at the same compression ratio, or does it sacrifice some quality for FlashAttention compatibility? The paper shows that L2 norm-based compression works, but does not show whether it works as well as the best available alternative. It is possible that attention-score-based methods preserve accuracy at higher compression ratios (e.g., 70% on needle-in-a-haystack where the L2 norm method drops to ~88%), or that they degrade more gracefully on complex tasks like LongBench. Without this comparison, the paper's claim of "broader applicability" (Abstract) is a claim about deployment flexibility, not about compression quality relative to the state of the art.

What evidence exists in the paper. Only the FastGen comparison in Figure 6, which the paper explicitly notes is a weakened version: "For a fair comparison, we implement FastGen without using the attention scores, i.e., we only consider local, punctuation and special tokens" (Section 4). This comparison is fair for the FlashAttention-compatible setting but does not represent the strongest version of FastGen. The paper does not compare against H2O or SnapKV at all, despite citing them as key prior work in Sections 1 and 6. The ALR metric in Section 3 quantifies how closely L2 norm tracks the ideal attention-based oracle, which is informative about how much attention mass is lost, but this is not the same as an end-to-end task performance comparison.

Mitigation status. The paper does not address this gap. The abstract and introduction emphasize FlashAttention compatibility as a key advantage, which frames the method as an alternative for a specific deployment scenario rather than a replacement for attention-based methods. But the absence of a head-to-head comparison leaves the quality tradeoff unspecified. A practitioner choosing between methods cannot determine from this paper alone whether FlashAttention compatibility is "worth it" in terms of compression quality.

---

The Method Lacks a Theoretical Explanation, Making Generalization Unpredictable

The assumption or constraint. The paper's method is grounded entirely in empirical observation: the authors noticed a correlation between L2 norm and attention scores in specific models on specific data, and built a compression strategy around it. The mechanistic hypothesis in Section 5 (sparse key embeddings creating attention sinks via a "dark subspace") is presented as hypothesis, not established fact. The paper explicitly acknowledges this gap:

"While we show that the L2 norm played a significant role in our experiments, we do not have a comprehensive theoretical explanation for why this is the case. Understanding the underlying reasons behind the importance of the L2 norm would require further theoretical exploration and empirical validation." (Section 8, Limitations)

The consequence. Without a theoretical grounding, practitioners cannot predict when the method will fail or succeed in new settings. The method works on Llama 2, Llama 3, and Gemma at 7–8B parameters. But will it work on:

• Encoder-decoder architectures (T5, BART) where attention patterns differ from decoder-only models?
• Models trained with different objectives (e.g., contrastive objectives, RLHF fine-tuning that might alter attention distributions)?
• Non-English text where token distributions, and thus key embedding geometry, might differ?
• Multimodal models (LLaVA, Flamingo) where visual tokens and text tokens interact in attention?
• Mixture-of-Experts models where different tokens are routed through different feed-forward pathways, potentially altering key embedding properties?

The paper provides no framework for answering these questions. The ALR diagnostic could, in principle, be computed for any new setting to assess whether the correlation holds—but as noted in Limitation 1, this requires the very attention computation the method avoids, and the paper provides no guidance on how to do this efficiently.

What evidence exists in the paper. The paper demonstrates consistency across three model families and multiple task types, which suggests some architectural generality. The perturbation experiment in Figure 9 provides causal evidence that the sparse dimensions are important for attention, but does not explain why the model learns to create these sparse embeddings or whether this is an inevitable consequence of Transformer training dynamics. The ALR consistency between Llama 2-7B and Llama 2-7B-32K (Figure 2) suggests the correlation survives long-context fine-tuning, but this is still within a single model family.

Mitigation status. Acknowledged as future work. The paper frames the mechanistic hypothesis (Section 5) as a starting point for investigation rather than a resolution. The connection to Cancedda (2024)'s "dark subspace" concept is suggestive but does not constitute a theory that yields falsifiable predictions about when the L2 norm signal will or will not work.

---

The Method Assumes a Fixed-Size KV Cache Budget Without Dynamic Adaptation

The assumption or constraint. The compression algorithm operates with a single global hyperparameter: max_kv, the fixed number of KV pairs to retain per head per layer (on compressed layers). Once set, this budget is enforced uniformly regardless of the input content or the model's apparent need for information. The eviction policy is also monotonic and irreversible: once a token is evicted from a head's KV Cache, it is permanently inaccessible to that head for all future generation steps, even if later tokens might have benefited from attending to it.

The consequence. The method cannot adapt to input heterogeneity. Consider two scenarios:

• A document with a single critical fact embedded in largely redundant filler text (like the needle-in-a-haystack task). The model can afford very aggressive compression because most tokens are not needed for retrieval.
• A dense reasoning task requiring synthesis of information distributed across many sentences (like HotpotQA or NarrativeQA). The model needs to retain a much larger fraction of tokens to preserve the relationships between facts.

A fixed max_kv treats these identically. The paper's results reflect this: on passkey retrieval (one isolated fact), 90% compression is lossless (Figure 4b). On LongBench tasks requiring multi-hop reasoning, 50% compression already shows visible degradation (Figure 5, with average scores dropping from ~48 to ~42 for Llama 3.1-8B). A content-aware method might allocate more cache budget to dense reasoning tasks and compress more aggressively on simple retrieval tasks, but the L2 norm method has no mechanism for such adaptation—the L2 norm reflects token-level geometric properties, not task-level information density.

Additionally, the irreversible eviction means that if the model changes its "focus" during generation (e.g., when answering a multi-part question that shifts topics midway), tokens relevant to the new focus that were previously evicted as high-L2-norm cannot be recovered. The model is stuck with whatever was retained at the time of eviction, which may have been optimized for the earlier focus.

What evidence exists in the paper. The paper does not experiment with adaptive budgets per input. All experiments use fixed max_kv or fixed compression ratios applied uniformly across all test examples. The LongBench per-subset breakdowns (Figures 21, 22) show that different tasks degrade at different rates under the same compression ratio (e.g., Qasper degrades less than HotpotQA at 50% compression for Llama 2-7B-80K), which indirectly demonstrates the heterogeneity that an adaptive method could exploit, but the paper does not explore this direction.

Mitigation status. The paper does not address this limitation. It does not propose any mechanism for input-dependent budget allocation, dynamic cache resizing, or token recovery. The method's simplicity—a single max_kv parameter, a uniform eviction rule—is simultaneously its strength (easy to implement, predictable memory footprint) and its weakness (no adaptation to input difficulty). For deployment scenarios where input characteristics are predictable and stable, the fixed budget is acceptable. For scenarios with highly variable inputs, the lack of adaptivity may lead to either wasteful memory usage on easy inputs or information loss on hard inputs.

---

The Method Has Not Been Evaluated on Wall-Clock Latency or Throughput

The assumption or constraint. The paper's primary claimed advantage over prior non-trainable methods is FlashAttention compatibility, which should translate to faster inference. However, the paper provides no timing measurements—no tokens-per-second, no latency percentiles, no throughput comparisons, and no memory bandwidth utilization analysis. The compression savings are measured exclusively in terms of the number of KV pairs retained (a proxy for memory capacity) and their effect on task accuracy, but not in terms of actual speed improvements.

The consequence. Practitioners cannot determine whether the method actually delivers faster inference in practice. Several factors could reduce or eliminate the theoretical speed advantage:

• Eviction overhead. Sorting the KV Cache by L2 norm at each generation step (or maintaining a sorted data structure) adds computational work that the paper does not benchmark. For large cache sizes and high generation counts, this overhead might be non-trivial.
• Memory fragmentation. Evicting arbitrary tokens from the middle of the KV Cache creates gaps in memory. FlashAttention and other optimized attention kernels typically assume contiguous memory layouts for efficient memory access. Reorganizing the cache after eviction (compacting the retained tokens into contiguous memory) would add overhead that the paper does not account for. The paper mentions FlashAttention compatibility in principle (Section 1: "without relying on the attention scores, this approach remains compatible with FlashAttention") but does not describe how the evicted tokens would be removed from FlashAttention's memory layout without incurring reorganization costs.
• The layer-skipping policy. The first two layers (and potentially layer 12) are not compressed at all. For these layers, the full KV Cache must still be stored and transferred from HBM to SM at each decoding step. For short-to-medium sequences, the uncompressed early layers might dominate the memory transfer time, reducing the practical benefit of compression in later layers.
• Comparison with attention-free baselines. The paper's only comparison against another method (FastGen, Figure 6) is purely in terms of perplexity, not speed. Without timing data, it is unclear whether the L2 norm method is actually faster than an uncompressed FlashAttention baseline (given the eviction overhead) or how it compares to other FlashAttention-compatible compression approaches on latency.

What evidence exists in the paper. None. No wall-clock measurements, no FLOP counts, no memory transfer analysis. Figure 6 shows perplexity and next-token accuracy for the FastGen comparison but no timing. The paper's argument for efficiency is entirely conceptual: L2 norm computation is cheap ($O(d_k)$ per key vector), eviction decisions don't require attention scores, therefore the method should be fast. But "should be fast" is not the same as "measured to be fast," and the paper provides no experimental validation of this central practical claim.

Mitigation status. Not addressed. The paper acknowledges broader limitations about model scale and theoretical understanding (Section 8) but does not mention the absence of latency benchmarking as a limitation. Given that the primary motivation for KV Cache compression is practical deployment efficiency (Section 1: "processing long-context inputs often results in a high decoding latency"), the omission of latency measurements is a significant gap in the evaluation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a conceptual reframing rather than an incremental improvement: it demonstrates that the importance of a KV pair for attention can be assessed before attention is computed, using only intrinsic geometric properties of the key embeddings themselves. This breaks the assumption—embedded in all prior non-trainable KV Cache compression methods (H2O, FastGen, SnapKV)—that attention scores are the necessary and only reliable importance signal. The consequence is not just a new compression technique but a new category of techniques: attention-free cache eviction policies that operate on the geometry of stored representations.

The magnitude of this shift should be understood in practical, not theoretical, terms. The paper does not introduce a new mathematical framework, prove optimality guarantees, or derive scaling laws. Instead, it provides:

• A diagnostic tool (the Attention Loss Ratio, ALR) for quantifying how well any attention-free importance signal approximates the ideal attention-based oracle, at per-head and per-layer resolution.
• A validated instance of this category (L2 norm of key embeddings) that works across three model families and multiple task types at compression ratios of 50–90% depending on task difficulty.
• A mechanistic hypothesis (sparse key embeddings create attention sinks) that explains why the instance works and suggests how to find others.

The research directions this reframing makes more attractive:

• Geometry-based importance signals. If L2 norm works, other intrinsic properties—sparsity (Gini coefficient, L1/L2 ratio), distance from centroid, principal component alignment, or the fraction of variance explained by the top-$k$ dimensions—may work equally well or better. The ALR framework provides a principled way to compare them.
• Per-head adaptive compression. The ALR heatmap (Figure 2) reveals that compression quality varies dramatically across heads and layers. This makes uniform compression ratios clearly suboptimal and motivates head-and-layer-specific policies: compress aggressively where the signal is strong, lightly or not at all where it is weak.
• FlashAttention-native compression. By decoupling importance assessment from attention computation, this work makes compression compatible with fused attention kernels. This removes the painful tradeoff that previously forced practitioners to choose between fast attention (FlashAttention) and memory-efficient attention (H2O-style compression). Future compression methods can be designed from the ground up with this compatibility constraint, rather than being retrofitted.

The research directions this work makes less attractive:

• Attention-score-based compression without FlashAttention compatibility. If geometry-based signals can achieve comparable compression quality at a fraction of the computational cost (no attention score materialization), methods that require full attention computation for eviction decisions become harder to justify, at least for deployment scenarios where FlashAttention is the standard.
• Uniform compression ratios across all layers and heads. The ALR analysis (Figure 2) makes the heterogeneity of compression quality visible. Methods that apply the same compression ratio everywhere leave efficiency on the table in layers with strong signals and risk damaging performance in layers with weak signals. The paper's layer-skipping policy is a first step; more fine-grained per-head policies are the natural next.

The paper also reconciles a tension in the attention sink literature. Xiao et al. (2024) identified that certain tokens act as attention sinks but did not explain which tokens become sinks or why. Darcet et al. (2024) found that high-L2-norm hidden states carry global information in Vision Transformers, which might lead one to expect high-L2-norm key embeddings to be important. This paper resolves the apparent contradiction by showing that the relationship is domain-and-role-specific: in the key embedding space of language Transformers, low L2 norm correlates with attention importance, likely because sparse key embeddings create geometric attractors for diverse queries. The direction of the correlation depends on which projection (K, Q, V, or hidden state) is being examined and what "importance" means in that context.

Follow-Up Research This Work Enables

A systematic comparison of geometric importance signals against attention-based methods on end-to-end task performance. The paper establishes that L2 norm works as a proxy relative to random eviction and relative to an attention-free FastGen. But it never measures the gap to full H2O, FastGen (with attention scores), or SnapKV at matched compression ratios. A direct follow-up would run all methods on the same models (Llama 2-7B, Llama 3-8B) across language modeling, needle-in-a-haystack, passkey retrieval, and LongBench, reporting accuracy as a function of compression ratio. The key measurement is: at what compression ratio does L2 norm-based eviction cross below the accuracy of H2O? If L2 norm matches attention-based methods at 50% but falls behind at 70%, then FlashAttention compatibility is "costing" approximately 20 percentage points of compression headroom—a concrete number practitioners can use to decide between the two approaches. If L2 norm matches or exceeds attention-based methods at all ratios, the case for attention-free compression becomes much stronger.

Per-head compression ratio optimization using the ALR diagnostic. The paper's Figure 2 shows that ALR varies across heads by a factor of roughly 5–10× (comparing the darkest blue regions to the bright spots). This suggests large efficiency gains from per-head compression ratios. A follow-up would: (1) compute ALR for each head in a target model, (2) define a budget allocation that assigns higher keep ratios to high-ALR heads and lower keep ratios to low-ALR heads, (3) constrain the total memory to match a uniform-compression baseline, (4) measure end-to-end accuracy on needle-in-a-haystack and LongBench. The hypothesis: per-head allocation recovers a significant fraction of the accuracy lost by uniform compression at the same total memory budget. A negative result (per-head allocation doesn't help beyond layer-level skipping) would suggest that the important variation is coarse (layer-level) rather than fine (head-level).

Scale invariance of the L2 norm–attention correlation. The paper tests only 7–8B parameter models. A direct scale-up experiment would measure the ALR heatmap (Figure 2 analog) for Llama 2-13B, Llama 2-70B, and if available, Llama 3-70B, using the same protocol (1024 chunks of 1024 tokens from Wikipedia). The specific question: does the mean ALR decrease, increase, or stay constant with model scale? Does the pattern of which layers show poor correlation (layers 0–1, middle layers around the 12th in a 32-layer model) generalize to deeper models proportionally (e.g., layers 0–3 and layers around 24–28 in a 80-layer model)? If the correlation strengthens with scale (sparser key embeddings in larger models?), the method becomes more attractive for the models where compression matters most. If it weakens, the method may be limited to smaller-scale deployment.

Does the L2 norm signal survive supervised fine-tuning and RLHF? All tested models are base pretrained models or long-context fine-tuned variants. Chat models (Llama 2-Chat, Llama 3-Instruct) undergo supervised fine-tuning and RLHF, which explicitly modify the model's output distribution and could alter attention patterns. A follow-up would replicate the ALR analysis and key eviction experiments (language modeling perplexity, needle-in-a-haystack accuracy at 50% compression) on Llama 2-7B-Chat vs. Llama 2-7B base. The specific concern: RLHF trains the model to produce helpful, harmless responses, which might shift attention away from structural sink tokens and toward content-relevant tokens, weakening the L2 norm–attention correlation. A negative result here would bound the method's applicability to base models and long-context fine-tuned variants, excluding the most commonly deployed model type (chat/instruct models).

Ablation of alternative norms and sparsity measures as compression signals. The paper uses L2 norm and hypothesizes that it works because it proxies for sparsity. A direct test: on Llama 2-7B, compare L2 norm, L1 norm, L∞ norm, and a direct sparsity measure (e.g., the fraction of dimensions needed to explain 90% of the total squared activation, or the Gini coefficient of the squared activations) as eviction signals. Measure the ALR for each signal and end-to-end accuracy on needle-in-a-haystack at 50% compression. If a sparsity measure outperforms L2 norm, it both validates the mechanistic hypothesis and provides a stronger practical signal. If L2 norm outperforms explicit sparsity measures, the hypothesis needs refinement—perhaps the total energy (which L2 norm measures) matters alongside how that energy is distributed.

The role of the first two layers and whether their behavior can be predicted without full attention analysis. The paper's layer-skipping policy is critical for performance but currently requires ALR computation to determine. A follow-up would investigate: can we predict which layers will have high ALR from properties of their weight matrices or their key embedding statistics, without ever computing attention scores? Candidates include: the condition number of $W_K$, the average L2 norm of key embeddings at that layer, the variance of L2 norms across tokens, or the effective rank of the key embedding matrix. If any of these correlates with ALR across layers and models, it provides a cheap way to set the skipping policy for arbitrary models, closing the deployment gap identified in the Limitations.

Practical Applications and Downstream Use Cases

On-device deployment of long-context LLMs with constrained memory. A Llama 3-8B model processing 32K-token contexts requires approximately 8.5 GB just for the KV Cache (at FP16, assuming $L=32$, $H=32$, $d_k=128$, $n=32768$, and GQA with 8 KV heads). On a device with 16 GB of total RAM (e.g., a high-end laptop or edge server), this leaves insufficient memory for model weights and activations. Applying 50% compression with the L2 norm method reduces the KV Cache to ~4.25 GB, making deployment feasible. The paper's Figure 5 (left) shows that on Llama 3.1-8B, 50% compression preserves LongBench average scores within approximately 6 points of the uncompressed baseline (roughly 42 vs. 48), which is a meaningful but manageable degradation for many on-device applications where the alternative is no long-context capability at all. The FlashAttention compatibility means this compression can be deployed alongside the fastest available attention kernels, avoiding the latency penalty that attention-score-based methods would impose.

Cost reduction for batch inference pipelines processing heterogeneous document lengths. Consider a document QA service that processes user-uploaded documents ranging from 1K to 100K tokens, running Llama 2-7B-80K. Without compression, the KV Cache must be sized for the maximum possible length for every request (since memory allocation is typically static per batch), wasting memory and limiting batch size. With L2 norm-based compression set to max_kv = 2000 (the setting from Figure 3), the KV Cache per sequence is capped at ~2000 tokens in compressed layers regardless of document length. For the short documents (1K tokens), no compression occurs. For the 80K-token documents, compression removes ~97.5% of tokens from compressed layers, reducing their KV Cache footprint from the equivalent of 80K tokens to 2000 tokens. The paper's passkey retrieval results (Figure 4b: 100% accuracy at 90% compression) suggest that information retrieval from very long documents remains viable at extreme compression ratios. This enables the service to batch many more requests together (since per-sequence memory is bounded), directly increasing throughput and reducing per-query cost.

Enabling long-context inference on previous-generation GPUs. GPUs with limited HBM (e.g., NVIDIA T4 with 16 GB, or consumer GPUs with 8–12 GB) cannot serve long-context models without aggressive KV Cache management. The L2 norm method provides a drop-in solution that requires no model modification, no fine-tuning, and no changes to the inference framework beyond adding a norm computation and eviction step. A practitioner with a pre-trained Llama 2-7B-80K model and a T4 GPU could, using this method, serve 32K-token contexts at 50% compression with approximately 4.25 GB of KV Cache memory—leaving room for model weights (~14 GB in FP16) within the 16 GB budget. The paper's needle-in-a-haystack results (Figure 4a: 99% accuracy at 50% compression, Figure 17c: perfect score of 1.000 at 20% compression with layers 0 and 1 skipped) suggest that even aggressive compression preserves the model's ability to locate and retrieve specific information from long documents, which is the core requirement for many retrieval-augmented generation and document QA applications.