Serving2024~114 minarXiv:2410.21465

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

score 9Hanshi Sun, Li-Wen Chang, Wenlei Bao +6ByteDance Seed↑ 11★ 283Oct 28, 2024cs.LGhttps://github.com/bytedance/ShadowKV

🎯 Pitch

ShadowKV makes long-context inference 3× faster by using a strikingly low-rank pre-RoPE key cache: storing just 1/60 of the keys on GPU and offloading values to CPU suffices to reconstruct accurate sparse attention on-the-fly. On an A100, it supports 6× larger batch sizes than full-attention systems and even outperforms the theoretical ceiling of unlimited GPU memory.

1. Executive Summary

This paper introduces ShadowKV, a high-throughput long-context LLM inference system that reduces GPU memory footprint by storing a low-rank pre-RoPE key cache and offloading the value cache to the CPU, then reconstructs minimal sparse KV pairs on-the-fly during decoding using an accurate chunk-based KV selection strategy (landmark-guided top-k chunk selection with outlier preservation). Evaluated on RULER, LongBench, and Needle In A Haystack across Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, and Yi-9B-200K, ShadowKV supports up to 6× larger batch sizes and boosts throughput by up to 3.04× on an A100 without accuracy degradation—even surpassing the performance achievable with infinite batch size under infinite GPU memory assumptions—while establishing a theoretical equivalent bandwidth exceeding 7 TB/s, though the memory savings materialize only when long-context sequences permit the pre-filling SVD overhead to amortize to negligible fractions (under 1% at 512K tokens).

2. Context and Motivation

The Core Problem: Long-Context LLMs Are Memory-Bound, Not Compute-Bound

The fundamental tension this paper tackles is that serving long-context LLMs at high throughput is bottlenecked by GPU memory capacity, not by computational speed. When a model processes a sequence of length $S$ , the KV cache stores two vectors (key and value) of dimension $d$ for every token in every layer. For a model with $L$ layers and $h$ KV heads, this means $2 \times L \times h \times S \times d$ floating-point numbers must reside in GPU memory simultaneously if we want to avoid recomputation. As sequence lengths push toward 128K, 256K, or even 1M tokens—a capability now common in models like Llama-3.1-8B (128K), GLM-4-9B-1M (1M), and Llama-3-8B-1M (1M+), the KV cache balloons to dominate the memory footprint.

The paper quantifies this concretely through its throughput experiments (Table 3, Table 4). For Llama-3-8B-1M with full attention at 60K context length, the maximum batch size is 8, yielding 160.62 tokens/s. At 122K, the maximum batch size crashes to 4, and at 244K it falls to 2. At 488K, the model is entirely out of memory (OOM)—you cannot run even a single sequence. The throughput follows this collapse: from 160.62 → 80.77 → 40.37 → 0 tokens/s. This is not a computation problem—the A100 has ample FLOPs for these operations—it's purely a memory capacity wall. The GPU simply cannot hold the KV cache for more than 2 sequences at 244K context.

This matters deeply for real-world deployment because:

Economic pressure toward higher throughput. Serving LLMs commercially requires maximizing tokens-per-second-per-dollar. If a GPU can only fit 2 sequences at 244K context, the hardware is severely underutilized—the compute units idle while waiting for memory operations, and the cost per token skyrockets. Organizations running batch inference (evaluating benchmarks, generating training data, powering retrieval pipelines) need high throughput, not just low latency for a single sequence.

The shift toward long-context applications. As models like GPT-4 (128K), Gemini (1M), and Llama-3 (1M+) expand context windows, applications increasingly exploit this capability: multi-document question answering across hundreds of pages, retrieval from extensive codebases, analysis of legal documents or scientific papers, multi-turn conversations that accumulate history over thousands of interactions. These workloads are inherently long-context, and they demand serving infrastructure that doesn't collapse under the memory pressure.

The batch size-throughput relationship is non-negotiable. Throughput in transformer inference scales nearly linearly with batch size—until you hit the memory wall. Doubling the batch size approximately doubles throughput. So when full attention at 244K caps batch size at 2, it's leaving perhaps 10-20× of potential throughput on the table compared to what the GPU compute units could handle. The paper's claim of 6× larger batch sizes (from 8 to 48 at 60K context for Llama-3.1-8B, Table 3) directly translates to 2.94-3.04× throughput improvements because it unlocks more parallelism that the GPU was designed to exploit.

The Prior Approach Landscape: Three Families, Three Failures

The paper identifies three lines of prior work, each of which fails to simultaneously satisfy the trifecta of requirements for practical long-context serving: reduce GPU memory, minimize latency, and maintain accuracy.

Family 1: KV Eviction Strategies

Methods like StreamingLLM, H2O, LESS, and SnapKV permanently discard KV pairs based on heuristics—keeping attention sinks and recent tokens (StreamingLLM), retaining high cumulative-attention tokens (H2O), or using the prompt's local window to select important tokens (SnapKV). The paper acknowledges this line of work reduces memory footprint by maintaining a fixed-capacity KV cache, but identifies a fatal flaw: evicted tokens are never recovered. This causes accuracy degradation in any scenario where future queries need information that was discarded.

The paper demonstrates this concretely through Figure 7's multi-turn conversation experiment. In a multi-turn needle retrieval benchmark (Multi-turn NIAH), the model must answer queries about different "needles" (pieces of information) embedded in the conversation context across multiple turns. SnapKV achieves reasonable accuracy on turn 1, but its performance collapses from turn 2 onward. Why? Because SnapKV selected which KV pairs to retain based on the first-turn query's attention patterns. When the second-turn query requires different contextual information that was evicted, it is permanently unavailable. StreamingLLM shows similar degradation. Full attention (which retains everything) and ShadowKV (which retains everything in compressed form) maintain accuracy across all turns. This is a fundamental limitation: eviction methods make irreversible decisions based on local information, and those decisions can be wrong for future queries.

The paper categorizes these methods under the "reduce GPU memory" axis of the problem space—they succeed at that goal—but at unacceptable accuracy cost for tasks requiring flexible access to distributed contextual information.

Family 2: Dynamic Sparse Attention

Methods like SparQ, Quest, Loki, and TriForce keep the full KV cache on the GPU but only compute attention over a selected subset of KV pairs during decoding. The KV cache is never discarded; it just isn't fully accessed. The paper highlights Quest (Tang et al., 2024) as a representative approach: it segments tokens into pages, approximates the highest attention within each page using efficient operations, and then computes exact attention only on the top-scoring pages.

The critical limitation the paper identifies: these methods do not reduce GPU memory footprint. All KV pairs remain resident on the GPU. This caps the maximum batch size at exactly the same level as full attention—the KV cache still occupies the same memory. The paper's Table 3 starkly illustrates this: at 244K context on Llama-3-8B-1M, full attention can fit batch size 2, and dynamic sparse attention methods like Quest can also only fit batch size 2. They may compute attention faster for those 2 sequences (because they compute sparse attention over fewer KV pairs), but they cannot serve more sequences simultaneously. Throughput remains bottlenecked by memory capacity, not computation speed.

The paper explicitly notes this as the defining shortcoming: "this line of work does not mitigate the memory footprint, thereby limiting the batch size and preventing accommodation of extremely long contexts (e.g., 1M tokens)" (Section 1). For high-throughput serving, reducing per-sequence computation without enabling larger batch sizes provides diminishing returns because the ceiling is already low.

Family 3: CPU Offloading with Sparse Attention

The "naive solution" the paper describes: take a dynamic sparse attention method (like Quest) and offload the entire KV cache to the CPU to free GPU memory. During decoding, identify the selected sparse KV pairs, fetch them from CPU to GPU over PCIe, and then compute attention. This approach does reduce GPU memory footprint—the KV cache now lives in CPU memory—potentially enabling larger batch sizes.

The paper identifies two fundamental problems with this approach, illustrated in Figure 4's timeline diagrams:

1. Excessive data movement latency. The KV cache must be fetched from CPU to GPU for every decoding step. Even with sparse attention selecting only a fraction of tokens, the raw PCIe bandwidth (31.5 GB/s on A100) is vastly lower than GPU memory bandwidth (2 TB/s on A100). The paper calculates this theoretical gap in Section 4.2: fetching K × C value vectors over PCIe dominates the decoding timeline. For the paper's example parameters (128K sequence, 256 selected chunks of size 8, 48 outliers), the value fetching alone requires transferring approximately 256 × 8 × 1024 bytes = 2.1 MB per head per layer over a 31.5 GB/s bus, which takes roughly 67 μs—compared to GPU memory access latency perhaps 1-2 orders of magnitude lower for the same data size.

2. Poor scaling to larger KV caches. As the KV cache grows (longer sequences), the amount of data to fetch for each decoding step grows proportionally if you maintain the same sparse budget percentage. Figure 4 shows that the "Fetch Sparse KV" timeline for methods like Quest with CPU offloading grows with the KV cache size, creating an ever-larger latency penalty. The paper notes that "overlapping KV fetching and computation becomes challenging with larger KV caches" (Section 2, Figure 4 caption). The operations are inherently serial: you must identify which chunks to fetch (using some approximate attention computation), then fetch those chunks, then compute exact attention. The fetch dominates the pipeline.

InfiniGen (Lee et al., 2024) represents the most sophisticated attempt in this family. It offloads the entire KV cache to the CPU and uses predefined SVD-based projections to identify which KV entries to prefetch before attention computation. However, the paper identifies two weaknesses through its experiments: (1) InfiniGen suffers from inaccurate prefetching—the SVD projections used for KV selection are fixed and offline (computed once on calibration data), meaning they cannot adapt to the specific query or context; and (2) it shows performance drops on complex tasks (Table 1 shows InfiniGen achieving far lower scores than full attention on tasks like multi-key needle retrieval and variable tracking). The paper's empirical comparison in Table 1 shows that InfiniGen with full KV offloading achieves only 70.13 average on RULER for Llama-3-8B-1M vs. 86.88 for ShadowKV—a gap that widens on harder subtasks.

The Gap: No Method Simultaneously Achieves All Three Goals

The paper's characterization of the problem space in Section 1 makes this gap explicit:

"Consequently, an ideal system for long-context LLM inference with sparse attention should: (i) reduce GPU memory usage, (ii) minimize inference latency, and (iii) maintain accuracy within limited sparse KV cache budgets."

Existing methods achieve at most two of three. KV eviction achieves (i) and (ii) but fails (iii). Dynamic sparse attention achieves (ii) and (iii) but fails (i). CPU offloading with sparse attention potentially achieves (i) and (iii) but fails (ii) due to PCIe bottleneck. The paper's contribution is a system that achieves all three simultaneously through a specific set of design choices motivated by empirical observations about the structure of the KV cache.

How This Paper Positions Itself: Leveraging Low-Rank Structure and Spatial Locality

The paper's positioning is not "here is yet another way to compress the KV cache." It argues that by making the right empirical observations about where and how the KV cache exhibits low-rank structure and spatial locality, one can design a system that fundamentally breaks the tradeoffs that constrained prior work. The key observations, detailed in Section 3, are:

Observation 1: Pre-RoPE keys are exceptionally low-rank—far more than post-RoPE keys, values, or weights. Figure 1 (left) shows the singular value decay of various components when processing a PG-19 sample through Llama-3.1-8B. The pre-RoPE key cache exhibits the sharpest singular value decay—meaning most of its information is concentrated in relatively few dimensions. This property is not shared by the value cache (which is not low-rank). Prior work like Palu (Chang et al., 2024) applied low-rank compression to weight matrices, which the paper shows in Figure 5 (left) to be far less compressible than pre-RoPE keys at the same rank. Other work like InfiniGen uses SVD projections for KV selection (not compression). ShadowKV's novel contribution is applying online, prompt-dependent SVD directly to the pre-RoPE key cache to achieve 6× compression without accuracy loss—a finding validated in Figure 5 (left) where accuracy on needle retrieval remains at 1.0 even when SVD rank is reduced from 1024 to 160.

Observation 2: Pre-RoPE keys within a sequence share low-rank subspaces, but across sequences they do not. Figure 1 (middle) measures the subspace similarity $D(\Psi_1\Psi_1^\top, \Psi_2\Psi_2^\top)$ (defined as normalized Frobenius inner product of projection matrices) between the right singular vectors of pre-RoPE key SVDs. When comparing a 16K context sequence to its 16K+2K continuation, the subspace similarity is extremely high (near 1.0 for layers 20-30)—the continuation's pre-RoPE keys live in essentially the same low-rank subspace as the original context. When comparing two different 16K sequences, the similarity drops substantially. This means that low-rank projections computed during pre-filling remain valid for newly generated tokens (motivating ShadowKV+, which stores generated tokens as low-rank states using the same projection matrices), but a fixed, offline projection matrix (as used in InfiniGen) would be suboptimal because it cannot capture the sequence-specific subspace structure.

Observation 3: Post-RoPE keys exhibit strong spatial locality within chunks, with rare outliers. Figure 5 (middle) visualizes the minimum cosine similarity between the chunk mean and individual key vectors within chunks of size 8. For the vast majority of chunks, this similarity is high (most data points cluster above 0.5)—meaning the chunk mean is a good approximation of any individual key vector's direction. However, a small number of "outlier chunks" (highlighted) show very low similarity—as low as -0.2. These outlier chunks represent regions where the chunk-level approximation breaks down because the key vectors within the chunk are diverse. The paper finds these outlier chunks constitute only 0.2-0.3% of all chunks. This observation motivates the landmark-based KV selection strategy (use chunk means as landmarks for efficient approximate attention scoring) combined with outlier preservation (store the full KV pairs for outlier chunks as static GPU cache to prevent accuracy degradation from poor approximation).

Observation 4: KV cache exhibits temporal locality across decoding steps. Figure 5 (right) shows the KV cache hit rate as a function of generated token index. The KV pairs selected by adjacent decoding steps have high overlap—the hit rate (fraction of required KV pairs already in cache from the previous step) is typically 60-80%. This means that for each decoding step, only 20-40% of the selected KV pairs are "new" and need to be reconstructed/fetched; the rest are already available from the previous step. The paper exploits this with an index-scan cache mechanism (Algorithm 2) that detects which chunk indices are newly selected and only performs the expensive reconstruction and fetching operations for those missed chunks.

These observations collectively enable ShadowKV's architecture: (1) store low-rank key cache on GPU and offload value cache to CPU (memory reduction from Observation 1), (2) use chunk-level landmarks to select top-k important chunks (fast selection from Observation 3), (3) preserve a tiny number of outlier chunks as static GPU cache (accuracy preservation from Observation 3), (4) overlap key cache reconstruction with value cache fetching using CUDA multi-streams (latency reduction from the independence of key and value operations), and (5) only reconstruct/fetch newly selected chunks at each decoding step (latency reduction from Observation 4's temporal locality).

The paper's theoretical equivalent bandwidth analysis in Section 4.2 formalizes why this design breaks the PCIe bottleneck. A naive offloading method must fetch keys and values at PCIe bandwidth for the selected sparse set. ShadowKV only fetches values at PCIe bandwidth, reconstructing keys from low-rank GPU-resident storage at GPU memory bandwidth. With the parameters in the paper's example (S=128K, C=8, K=256, O=48), the equivalent bandwidth reaches 7.2 TB/s—3.6× higher than the A100's native 2 TB/s memory bandwidth—because the effective data movement is dominated by GPU-resident operations rather than PCIe transfers. This is not a claim that ShadowKV exceeds the physical bandwidth of the hardware, but rather that its computational pattern accomplishes attention-equivalent work with less effective data movement than what full attention requires even if all data were in GPU memory.

The Practical Framing

The paper's ultimate framing is that long-context LLM inference at scale is a systems problem, not an algorithms problem in isolation. The components ShadowKV uses—low-rank decomposition, chunk-based selection, outlier detection, temporal caching—are not individually novel. What is novel is the integrated system design that combines them based on empirical observations about the KV cache's structure to simultaneously address memory, latency, and accuracy constraints. The contribution is not "discover that pre-RoPE keys are low-rank" (which has been observed in other contexts) but rather "show that by combining prompt-dependent online SVD on pre-RoPE keys with landmark-guided sparse attention and PCIe-overlapped reconstruction, you can build a system that supports 6× larger batches and 3× higher throughput than any prior approach while matching full-attention accuracy."

3. Technical Approach

3.1 Reader Orientation

ShadowKV is an inference system that reshapes how the KV cache is stored and accessed during long-context LLM decoding. It solves the problem that serving long sequences at high throughput requires simultaneously reducing GPU memory footprint (to fit larger batch sizes), minimizing decoding latency (to keep tokens flowing), and maintaining accuracy (to avoid degrading model outputs). The solution's shape is a storage-decoupled architecture: compress the key cache into a low-rank form that stays on the GPU, offload the value cache to the CPU, and during decoding, use lightweight "landmarks" to select which sparse KV pairs to reconstruct on-the-fly, overlapping key reconstruction with value fetching to hide latency.

3.2 Big-Picture Architecture (Diagram in Words)

ShadowKV splits inference into two phases with six interacting components:

During Pre-filling (Algorithm 1):

Low-Rank Key Cache Compressor — performs online SVD on the pre-RoPE key cache to extract a rank- $r$ representation stored as two small matrices ( $A$ and $B$ ) on the GPU.
Landmark Builder — segments the post-RoPE key cache into chunks, computes the mean of each chunk, and stores these means as compact "landmarks" for approximate attention scoring.
Outlier Detector — identifies chunks where the chunk mean poorly approximates individual key vectors (measured by cosine similarity), and preserves the full KV pairs for those outlier chunks as static GPU cache.
Value Cache Offloader — sends the value cache for all non-outlier tokens to CPU memory.

During Decoding (Algorithm 2): 5. KV Selector — computes approximate attention scores between the query and landmarks, identifies the top- $k$ scoring chunk indices, and uses a temporal cache to avoid re-selecting chunks that were already fetched for the previous token. 6. Sparse KV Reconstructor — for newly selected chunks (cache misses), simultaneously reconstructs the full key cache from the low-rank GPU-resident matrices and fetches the corresponding value cache from the CPU, using CUDA multi-streams to overlap these operations.

The final step computes exact sparse attention over the reconstructed sparse KV pairs plus the static outlier cache.

3.3 Roadmap for the Deep Dive

I'll explain the components in the order they operate, because each component's design depends on observations made about the preceding stage:

First, the low-rank key cache compression (Section 3.1 observation formalized) — why pre-RoPE keys, why online SVD, why the particular rank and decompression procedure.
Second, the landmark construction and outlier detection — how chunk-level approximations enable fast KV selection, and why a tiny fraction of chunks must be handled differently.
Third, the value cache offloading decision — why values, not keys, go to the CPU, and the memory savings calculation.
Fourth, the decoding-time KV selection algorithm — how landmarks approximate attention, how chunk indices are selected, and the temporal cache mechanism that exploits KV hit rates.
Fifth, the sparse KV reconstruction and overlapping strategy — how the low-rank key representation is decompressed, how value fetching is overlapped with key reconstruction, and the CUDA multi-stream implementation.
Sixth, the theoretical equivalent bandwidth model — a formal tool to understand why the system design achieves throughput gains beyond what memory capacity alone would suggest, tying together the latency and memory contributions.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that by exploiting the specific low-rank structure of pre-RoPE keys and the spatial locality of post-RoPE keys, one can decouple KV cache storage (keys compressed on GPU, values on CPU) from KV cache access (sparse on-the-fly reconstruction) to simultaneously reduce memory footprint and minimize decoding latency while maintaining accuracy.

Low-Rank Key Cache Compression via Online SVD

What gets compressed and why. During the pre-filling phase, for each transformer layer, the model computes key and value activations for all input tokens. ShadowKV targets the pre-RoPE key cache — the key activations before rotary position embeddings are applied. The paper's central empirical finding (Figure 1, left) is that pre-RoPE keys exhibit exceptionally sharp singular value decay compared to post-RoPE keys, values, the key weight matrix $W_K$ , the value weight matrix $W_V$ , and the layer input activations. Concretely, when processing a PG-19 sample through Llama-3.1-8B, the relative singular values of the pre-RoPE key cache drop by roughly three orders of magnitude from the first singular value to the 200th, while post-RoPE keys show much slower decay. This means the pre-RoPE key cache can be accurately represented using far fewer dimensions than its original $d$ -dimensional space.

The paper's Figure 5 (left) quantifies this compression in accuracy terms: storing pre-RoPE keys at SVD rank 160 (a 6.4× compression from $d = 1024$ ) achieves identical needle retrieval accuracy to the uncompressed full-rank key cache. In contrast, compressing the value cache or post-RoPE key cache to rank 160 causes substantial accuracy degradation. This asymmetry — pre-RoPE keys are highly compressible, values are not — is the fundamental insight that determines what gets compressed and what gets offloaded.

Why online SVD and not offline projections. The paper's Figure 1 (middle) reveals a critical property of the low-rank subspaces. Define the similarity metric between two rank- $r$ projection matrices $H_1$ and $H_2$ as:

$D(H_1, H_2) = \frac{\langle H_1, H_2 \rangle}{r}$

where $\langle \cdot, \cdot \rangle$ is the Frobenius inner product, and $H_1 = \Psi_1 \Psi_1^\top$ , $H_2 = \Psi_2 \Psi_2^\top$ are the right singular vector projection matrices from truncated SVDs of two pre-RoPE key matrices $K_1, K_2 \in \mathbb{R}^{n \times d}$ .

What it computes: the normalized Frobenius inner product between two orthonormal projection matrices, yielding a value in $[0, 1]$ . When $D = 1$ , the two subspaces are identical (they project onto the same $r$ -dimensional space). When $D = 0$ , the subspaces are orthogonal (completely disjoint).

Why this form: the Frobenius inner product of projection matrices equals the sum of squared cosines of the principal angles between the two subspaces, divided by $r$ . This gives the average fraction of variance preserved when projecting data from one subspace to the other. It directly answers the question: "can I reuse the projection matrix from sequence A to compress sequence B?"

The empirical result: when comparing a length-16K "context" sequence to its 16K+2K continuation ("extended context"), the subspace similarity exceeds 0.95 for most layers. But when comparing the same context to a different 16K sequence ("inter-context"), the similarity drops substantially, hovering around 0.75–0.85 for most layers (Figure 1, middle). The implication: each sequence induces its own low-rank subspace that is highly consistent within the sequence (new tokens share the same subspace as the context) but varies substantially across sequences. This means an offline, fixed projection matrix — as used in InfiniGen for KV selection — would be suboptimal for compression because it cannot adapt to the specific subspace of each prompt. ShadowKV therefore performs online, prompt-dependent SVD during pre-filling.

The compression procedure. For each transformer layer, given the pre-RoPE key cache $K \in \mathbb{R}^{b \times h_{kv} \times s \times d}$ (where $b$ is batch size, $h_{kv}$ is number of KV heads, $s$ is sequence length, $d$ is head dimension), ShadowKV computes a rank- $r$ truncated SVD:

$K \approx A \times_3 B$

where $A \in \mathbb{R}^{b \times s \times r}$ represents the left singular vectors scaled by singular values (the "coordinates" of each token in the low-rank space), and $B \in \mathbb{R}^{b \times h_{kv} \times r \times d}$ represents the right singular vectors (the "basis" that maps from the low-rank space back to the full $d$ -dimensional key space). The notation $\times_3$ indicates tensor contraction along the third dimension.

What it computes: a rank- $r$ factorization of the key cache. For each KV head and each batch element, the full $s \times d$ key cache is approximated as the product of an $s \times r$ matrix $A$ and an $r \times d$ matrix $B$ , where $r \ll d$ and $r \ll s$ . The factorization minimizes the Frobenius norm of the approximation error $\lVert K - AB \rVert_F$ subject to the rank constraint.

Why this form: storing $A$ and $B$ requires $s \times r + r \times d$ values per head, compared to $s \times d$ for the full cache. The compression ratio is approximately $\frac{sd}{sr + rd} = \frac{sd}{r(s + d)}$ . For typical values ( $s = 128\text{K}$ , $d = 1024$ , $r = 160$ ), this gives roughly $\frac{128000 \times 1024}{160 \times (128000 + 1024)} \approx 6.4\times$ compression — matching the paper's empirical observation. The decomposition is stored as separate $A$ and $B$ matrices (rather than the product) because during decoding, only specific rows of $A$ (corresponding to selected tokens) need to be multiplied by $B$ to reconstruct the key cache for those tokens.

The SVD computational overhead. The paper acknowledges that computing SVD during pre-filling adds computation, but argues the cost is negligible for long contexts. Figure 1 (right) shows the SVD overhead as a percentage of total pre-filling time for sequence lengths from 64K to 384K. The trend is revealing: at 64K, the SVD overhead is approximately 5% of the total pre-filling time. At 128K, this drops to approximately 3%. At 256K, it's approximately 1.8%. At 512K, it falls below 1%. The reason is that SVD cost scales linearly with sequence length ( $O(s \cdot d^2)$ for a $s \times d$ matrix), while the attention computation during pre-filling scales quadratically ( $O(s^2 \cdot d)$ ). The quadratic attention term dominates increasingly at longer sequences, making the SVD overhead asymptotically negligible.

The paper's configuration uses rank $r = 160$ for most experiments (stated in Section 5 setup: "we set the rank to 160"). The key cache head dimension $d$ varies by model — for Llama-3.1-8B and Llama-3-8B-1M, $d = 128$ per head with 8 KV heads, while for GLM-4-9B-1M and Yi-9B-200K, there are 4 KV heads with larger per-head dimensions. The rank of 160 provides approximately 6× compression relative to the original key cache storage.

Landmark Construction and Outlier Detection

Motivation: fast approximate attention scoring. During decoding, the system needs to decide which KV pairs are important for the current query token — but it must do so without accessing the full key cache (which would defeat the memory savings) and without expensive exact attention computation (which would defeat the latency savings). The solution is to maintain a compressed representation of the post-RoPE key cache — called landmarks — that enables approximate attention scoring at a fraction of the cost.

Why post-RoPE for landmarks, not pre-RoPE. The landmarks operate on post-RoPE keys, not pre-RoPE keys. This is a crucial design distinction: the pre-RoPE keys are used for compression (they're low-rank), but the RoPE transformation changes the geometry of the key space by encoding positional information, and sparse attention must operate in the post-RoPE space where position matters for retrieving contextually relevant information. The low-rank compressed pre-RoPE keys are stored on GPU for memory efficiency, but the landmark-based selection operates on chunk-level aggregates of the post-RoPE keys.

The chunking and landmark construction procedure. For each transformer layer, after computing the post-RoPE key cache $K_{\text{RoPE}} \in \mathbb{R}^{b \times h_{kv} \times s \times d}$ , ShadowKV segments it along the sequence dimension into chunks of size $c$ (where $c = 8$ for all experiments). For each chunk, it computes the mean key vector:

$\bar{k}_j = \frac{1}{c} \sum_{i=1}^{c} k_{\text{RoPE}}^{(j-1)c + i}$

where $\bar{k}_j \in \mathbb{R}^d$ is the landmark for chunk $j$ . These landmarks form a matrix $L \in \mathbb{R}^{b \times h_{kv} \times (s/c) \times d}$ — essentially a downsampled version of the post-RoPE key cache at $\frac{1}{c}$ the resolution.

What it computes: for each chunk of $c$ consecutive tokens, a single $d$ -dimensional vector that represents the "average" key direction in that chunk. This reduces the number of vectors that need to be scored during KV selection from $s$ (the full sequence length) to $s/c$ , yielding an 8× reduction in the approximate attention computation for $c = 8$ .

Why chunk-wise means: the paper's Observation 3 (Section 3.2 and Figure 5, middle) shows that within most chunks of size 8, individual post-RoPE key vectors have high cosine similarity to the chunk mean. The chunk mean therefore serves as a high-quality proxy: if the query has high attention to the chunk mean, it likely has high attention to individual tokens within that chunk. This property is what makes chunk-level selection work — it's not an arbitrary coarsening, but one justified by empirical measurements of key vector similarity structure.

The outlier chunk problem. The paper identifies that not all chunks are well-approximated by their mean. Figure 5 (middle) shows a small fraction of chunks where the minimum cosine similarity between the chunk mean and individual key vectors is very low — sometimes even negative (indicating opposite directions). The paper visualizes this by plotting the minimum cosine similarity for each chunk across two specific attention heads (Layer-0 Head-1 and Layer-16 Head-7) on a 128K sequence, showing most chunks cluster at similarity values above 0.5, but a handful of "outlier chunks" drop below 0.0.

These outlier chunks are identified algorithmically in Algorithm 1. For each chunk, compute the cosine similarity between the chunk mean $\bar{k}_j$ and each individual key vector in that chunk. Take the minimum similarity across all vectors in the chunk — this is the worst-case approximation quality. The chunks with the lowest $o$ minimum similarity values (where $o$ is the outlier budget, set to 48 in all experiments) are designated as outliers.

Why 48 outliers: the paper's ablation study in Appendix A.8 (Table 15) systematically varies the number of outlier chunks and measures RULER accuracy. With 0 outliers, accuracy drops from 86.68 (full attention) to 74.05 — a catastrophic degradation. Adding just 8 outliers (0.049% of all chunks) recovers to 86.22. At 16 outliers (0.098%), accuracy reaches 86.42. At 48 outliers (0.293%), accuracy is 86.88 — indistinguishable from full attention and 0.20 points higher than the baseline. The paper argues that the first chunk (the attention sink, identified in prior work by Xiao et al., 2023) is a significant outlier — its full preservation is critical. Beyond 48 outliers, there are diminishing returns. With 48 outliers at chunk size 8, the total GPU memory cost for outlier storage is $48 \times 8 \times 2 = 768$ KV pairs per head per layer — a negligible fraction of a 128K sequence (0.6%).

How outliers are handled. The full KV pairs for outlier chunks are gathered and stored as static GPU cache (K_outlier, V_outlier in Algorithm 1). They are treated as a permanent supplement to the sparse KV cache during every decoding step — the attention computation always includes the outlier tokens alongside the dynamically selected chunks. The corresponding chunk landmarks are removed from the landmark matrix $L$ to avoid double-counting during selection.

What gets offloaded to the CPU. After identifying outliers, the remaining value cache (V \ V_outlier) is offloaded to CPU memory. The corresponding landmarks (L = C \ Gather(C, I) where $I$ are the outlier chunk indices) are kept on GPU. The key cache is not offloaded because it is stored in compressed low-rank form on the GPU.

Memory savings quantification. The paper provides a formula in Appendix A.2:

$\text{Memory Savings} = \frac{2SM}{SM/C + 2(K + O)C + Sr + rM}$

where $M$ is the byte size of each K or V vector, $S$ is the sequence length, $C$ is the chunk size, $K$ is the number of chunks selected per decoding step (the sparse budget), $O$ is the number of outlier chunks, and $r$ is the pre-RoPE key cache SVD rank. For the paper's default parameters ( $M = 1024$ bytes for BF16, $C = 8$ , $S = 128\text{K}$ , $K = 256$ , $O = 48$ , $r = 160$ ), this yields approximately 7.08× memory savings — meaning the GPU KV cache footprint is reduced to about 14% of the original. The term $2SM$ in the numerator is the original KV cache size (keys + values, $S$ tokens, $M$ bytes each). The denominator includes: $SM/C$ for the landmarks, $2(K+O)C$ for the selected sparse KV pairs plus outliers in the decoding cache, $Sr$ for the $A$ matrix of the low-rank decomposition, and $rM$ for the $B$ matrix (per-head but small relative to other terms).

Decoding-Time KV Selection via Landmark-Guided Approximate Attention

The selection problem. At each decoding step, the system receives a query $Q \in \mathbb{R}^{b \times h_q \times s_q \times d}$ (where $s_q$ is the number of query tokens — typically 1 for autoregressive decoding, but may be larger for speculative decoding or multi-query attention). The goal is to identify the top- $k$ most important chunks (each containing $c$ tokens) without computing exact attention over all $S$ tokens.

Step 1: Compute chunk-level approximate attention scores. The query is multiplied against the landmark matrix $L$ to produce chunk-level scores:

$P = \text{MatMul}(Q, L^\top) \in \mathbb{R}^{b \times h_q \times s_q \times n_c}$

where $n_c = s/c$ is the number of chunks (excluding outlier chunks). The softmax is applied to produce approximate attention probabilities:

$S = \text{Softmax}(P / \sqrt{d})$

What it computes: for each query token in each attention head, a probability distribution over $n_c$ chunks representing the approximate importance of each chunk's tokens.

Why softmax with $\sqrt{d}$ scaling: this mirrors the standard attention formulation, where the scaling factor $\sqrt{d}$ counteracts the growth of dot products with dimension. Using the same scaling ensures the approximate scores are calibrated similarly to exact attention scores, making the top- $k$ selection faithful.

Step 2: Aggregate across query tokens and map to KV heads. For multi-query attention (where the number of query heads $h_q$ may exceed the number of KV heads $h_{kv}$ ), the scores are aggregated in two stages:

$S_1 = \text{sum}(S, \text{dim} = -2) \in \mathbb{R}^{b \times h_q \times n_c}$ $S_2 = \text{maxkv\_group}(S_1) \in \mathbb{R}^{b \times h_{kv} \times n_c}$

The first operation sums the chunk-level scores across all query tokens (if $s_q > 1$ ). The second operation (maxkv_group) groups query heads that share the same KV head and takes the maximum score per chunk — the idea being that if any query head within a group considers a chunk important, the entire KV head should fetch it. This is standard in grouped-query attention implementations.

Step 3: Select top- $k$ chunk indices. For each KV head, the $k$ chunk indices with the highest aggregated scores are selected:

$I = \text{ArgTopK}(S_2, k) \in \mathbb{R}^{b \times h_{kv} \times k}$

What it computes: the indices of the $k$ most important chunks out of $n_c$ for each KV head, where $k$ is the sparse budget. For the paper's default configuration, $k = 256$ chunks. Each chunk contains $c = 8$ tokens, so the sparse attention budget is $k \times c = 256 \times 8 = 2048$ tokens per KV head, plus $48 \times 8 = 384$ outlier tokens, totaling 2432 tokens. For a 128K sequence, this represents a sparse budget of 1.90% (or 1.56% when expressed as fraction of full attention computation, since the paper's efficiency numbers use 1/16 = 1.56% as the target ratio for fair comparison with baselines).

Why $k=256$ chunks: the paper sweeps over sparse budgets in Figure 8, showing that accuracy on most RULER tasks stabilizes at or before 1.56% of full attention. At 256 chunks, the computational cost of the exact sparse attention is roughly 1/64 of full attention for a 128K sequence, matching the 1.56% figure. The landmark-guided approximate scoring (which uses $n_c = 16000$ chunk scores for a 128K sequence) adds negligible overhead since it operates on dimension-reduced representations.

The temporal KV cache mechanism (cache hit/miss logic). The paper observes (Figure 5, right) that the chunk indices selected for adjacent decoding steps exhibit high overlap — the "KV cache hit rate" averages 60-80%, meaning only 20-40% of the selected chunk indices are new. ShadowKV exploits this with an index-scan mechanism in Algorithm 2. The system maintains a record of which chunk indices were already fetched and reconstructed for the previous decoding step. When the new top- $k$ indices $I$ are computed, the system performs an index scan to identify which indices are new (cache misses) versus already available (cache hits). Only the missed indices trigger expensive reconstruction and fetching operations.

What this mechanism saves: if the hit rate is 60%, ShadowKV avoids reconstructing and fetching 60% of the selected KV pairs — cutting the already-sparse decoding overhead by more than half. The paper reports this as "reducing computations and data movements by over 60% for each decoding step" (Section 3.2, Figure 5 caption). This is implemented as an optimized CUDA kernel that performs the set-difference operation between old and new index sets efficiently in GPU memory.

Sparse KV Reconstruction and Overlapping Strategy

The reconstruction of keys from low-rank storage. For each newly selected chunk index $i \in I_{\text{miss}}$ , the corresponding rows of the $A$ matrix (the left singular vectors scaled by singular values) are extracted. The chunk contains $c$ consecutive tokens, so rows $i \cdot c$ through $(i+1) \cdot c - 1$ of $A$ are gathered. The full key cache for those tokens is reconstructed via matrix multiplication:

$K_{\text{sparse}} = \text{MatMul}(\text{Gather}(A, I_{\text{miss}}), B)$

where $\text{Gather}(A, I_{\text{miss}}) \in \mathbb{R}^{b \times (|I_{\text{miss}}| \cdot c) \times r}$ are the low-rank coordinates of the selected tokens, and $B \in \mathbb{R}^{b \times h_{kv} \times r \times d}$ are the right singular vectors. The output $K_{\text{sparse}} \in \mathbb{R}^{b \times h_{kv} \times (|I_{\text{miss}}| \cdot c) \times d}$ is the reconstructed full-dimensional key cache for the selected tokens.

What this computes: a decompression of the low-rank key representation back to the full $d$ -dimensional space. The operation is a batched matrix multiplication: for each KV head and batch element, multiply the $(|I_{\text{miss}}| \cdot c) \times r$ gathered $A$ submatrix by the $r \times d$ matrix $B$ to recover the approximate full keys. Since $r = 160 \ll d = 128$ (for Llama models), this multiplication is relatively cheap — roughly $|I_{\text{miss}}| \cdot c \cdot r \cdot d$ FLOPs.

Why reconstruct only the selected tokens: if the sparse budget is 1.56% and the cache hit rate is 60%, only about $0.016 \times 0.4 = 0.64\%$ of the total key cache needs to be reconstructed per decoding step. The remaining 99.36% of the key cache incurs zero computation because it's never accessed. This is the fundamental efficiency mechanism: most of the key cache is stored in a compressed form that requires decompression only for the small fraction actually needed.

RoPE application after reconstruction. The reconstructed keys are in pre-RoPE form (since the SVD was performed on pre-RoPE keys). The rotation position embedding must be applied after reconstruction to obtain the correct post-RoPE keys for attention computation:

$K_{\text{sparse}}^{\text{RoPE}} = \text{RoPE}(K_{\text{sparse}})$

This is necessary because the low-rank representation cannot capture position-dependent rotations, which are inherently full-rank operations.

Fetching values from the CPU. Simultaneously, the system fetches the corresponding value cache entries for the missed chunk indices from CPU memory. For each selected chunk index $i$ , the value cache entries $V[i \cdot c : (i+1) \cdot c]$ are transferred over PCIe. The total data transferred is $|I_{\text{miss}}| \cdot c \cdot M$ bytes, where $M = d \times 2$ for BF16 (2 bytes per value).

The overlapping mechanism via CUDA multi-streams. This is the critical systems engineering insight that prevents the PCIe transfer from becoming a bottleneck. The key cache reconstruction (GPU computation, using GPU-resident matrices $A$ and $B$ ) and the value cache fetching (CPU-to-GPU data transfer over PCIe) are independent operations — they use different data, different hardware paths, and have no dependencies. ShadowKV launches them on separate CUDA streams:

Stream 1: GPU kernel for MatMul(Gather(A, I), B) to reconstruct the key cache.
Stream 2: CUDA memory copy from CPU to GPU for the value cache entries.

The GPU scheduler can execute the matrix multiplication on the tensor cores while the DMA engine simultaneously transfers value data over PCIe. The paper's timings in Table 13 show that the "Recompute K (Overlapped)" operation and the "Fetch V" operation run concurrently. For example, at 24×128K batch size, the key reconstruction takes 1.36 ms, the value fetch takes 1.66 ms, but they are overlapped — the total wait time is approximately the maximum of the two, not their sum.

Assembly of the final sparse KV cache. After reconstruction and fetching, the sparse KV cache for the current decoding step consists of three components concatenated together:

$K = [K_{\text{outlier}}; \text{RoPE}(K_{\text{sparse}}); K_{\text{new\_tokens}}]$ $V = [V_{\text{outlier}}; V_{\text{sparse}}; V_{\text{new\_tokens}}]$

where $K_{\text{new\_tokens}}, V_{\text{new\_tokens}}$ are the KV pairs for newly generated tokens (not part of the prompt context). These are stored in full precision on the GPU since they are few in number (one per decoding step) and need to be accessed frequently.

The ShadowKV+ extension for generated tokens. The paper describes (Section 4.1, Appendix A.1) an extension where newly generated tokens are also stored in low-rank form using the same projection matrices from the pre-filling SVD. If $\Psi \in \mathbb{R}^{d \times r}$ (where $B^\top \approx \Psi$ up to reshaping) is the right singular matrix from the context's pre-RoPE key SVD, then a new pre-RoPE key $k' \in \mathbb{R}^d$ can be stored as its low-dimensional projection $k' \Psi \in \mathbb{R}^r$ and reconstructed when needed as $(k' \Psi) \Psi^\top$ . The paper validates that this preserves accuracy (Tables 5 and 6 show ShadowKV+ matching ShadowKV on RULER and LongBench). This extension is valuable for long output sequences where the newly generated tokens' KV cache would otherwise accumulate on the GPU.

Theoretical Equivalent Bandwidth Model

The formal model. The paper introduces the concept of theoretical equivalent bandwidth to analyze why ShadowKV's design achieves throughput gains beyond what memory capacity alone would suggest. The model, presented in Section 4.2, computes the effective bandwidth at which the system processes data for attention computation, accounting for all data movements:

$B_{\text{eq}} = \frac{2SB_{\text{GPU}}}{S/C + 2(K + O)C + (1 - \alpha)KC \cdot B_{\text{GPU}}/B_{\text{PCIe}}}$

where $S$ is the sequence length, $C$ is the chunk size, $K$ is the number of selected chunks per decoding step, $O$ is the number of outlier chunks, $\alpha$ is the KV cache hit rate (fraction of selected chunks already available from the previous step), $B_{\text{GPU}}$ is the GPU memory bandwidth (2 TB/s for A100), and $B_{\text{PCIe}}$ is the PCIe bandwidth (31.5 GB/s for A100).

What it computes: the bandwidth that an equivalent system with full GPU-resident KV cache would need to achieve the same attention throughput as ShadowKV with its hybrid GPU/CPU storage and sparse access pattern. The numerator $2SB_{\text{GPU}}$ represents the ideal case where all $2S$ KV vectors (keys and values) are accessed at full GPU bandwidth — this is the baseline full-attention bandwidth. The denominator accounts for the actual data movement in ShadowKV: $S/C$ for loading the landmarks (GPU bandwidth), $2(K+O)C$ for the exact sparse attention computation on selected chunks plus outliers (GPU bandwidth), and $(1 - \alpha)KC \cdot B_{\text{GPU}}/B_{\text{PCIe}}$ for the PCIe penalty — the missed chunks that must be fetched over the slower PCIe bus, weighted by the bandwidth ratio to convert PCIe bytes into equivalent GPU-wide bandwidth penalty.

Why this form: it decomposes total data movement into operations that happen at GPU bandwidth (landmark access, sparse attention computation) and operations that happen at PCIe bandwidth (value fetching), with the PCIe penalty scaled by the bandwidth ratio $(2\text{ TB/s}) / (31.5\text{ GB/s}) \approx 63.5\times$ . A naive CPU offloading method (fetching the entire KV cache for selected chunks at PCIe speed) would have the PCIe term dominate, yielding $B_{\text{eq}} \ll B_{\text{GPU}}$ . ShadowKV minimizes this term by (1) only fetching values (not keys) over PCIe — cutting the PCIe data in half; (2) overlapping the fetch with key reconstruction — hiding the latency; and (3) using the temporal cache (the $(1-\alpha)$ factor) to avoid fetching already-available chunks — reducing PCIe traffic by the hit rate.

Plugging in the paper's numbers. For $S = 128\text{K}$ , $C = 8$ , $K = 256$ , $O = 48$ , $\alpha = 0.6$ , $B_{\text{GPU}} = 2\text{ TB/s}$ , $B_{\text{PCIe}} = 31.5\text{ GB/s}$ :

$B_{\text{eq}} = \frac{2 \cdot 128000 \cdot 2\text{ TB/s}}{128000/8 + 2(256 + 48) \cdot 8 + (1 - 0.6) \cdot 256 \cdot 8 \cdot 2\text{ TB/s} / 31.5\text{ GB/s}}$

$= \frac{512000\text{ TB/s}}{16000 + 4864 + 0.4 \cdot 256 \cdot 8 \cdot 63.5}$

$= \frac{512000\text{ TB/s}}{16000 + 4864 + 20808}$

$= \frac{512000}{41672} \text{ TB/s} \approx 12.3\text{ TB/s}$

The paper reports 7.2 TB/s, likely using different assumed parameters or a more conservative model. Regardless, both values substantially exceed the A100's 2 TB/s memory bandwidth — the key point is that ShadowKV achieves an effective throughput that would require 3.6× or more GPU memory bandwidth if implemented as full attention. This isn't physically exceeding hardware limits; rather, it demonstrates that the combination of compression, sparsity, temporal caching, and overlapping converts what would be a memory-bandwidth-bound workload into one that is closer to compute-bound.

The practical implication: the equivalent bandwidth formula explains why ShadowKV can achieve throughput exceeding "infinite batch size under infinite GPU memory assumptions" (Table 3). The "infinite" scenario assumes the entire KV cache is in GPU memory and accessed at 2 TB/s, but full attention still needs to read all $2S$ vectors. ShadowKV reads far fewer bytes total (landmarks + sparse KV + outliers), and only a small fraction of those bytes come over PCIe. The effective data movement is lower even than the infinite-GPU-memory full-attention case, which is why throughput can exceed that theoretical limit.

Summary of Key Design Decisions and Their Justifications

Online, prompt-dependent SVD on pre-RoPE keys (not offline, and not on post-RoPE keys or values): justified by the observation that pre-RoPE keys are exceptionally low-rank, unlike values, and that the low-rank subspaces vary across sequences (Figure 1, middle), making fixed offline projections suboptimal.
Rank 160 for SVD compression: justified by Figure 5 (left) and Figure 9 (right), showing accuracy saturation at this rank across multiple RULER tasks — higher ranks provide negligible benefit while reducing compression ratio.
Chunk size 8 for landmarks: justified by Figure 9 (left), showing that larger chunk sizes increase batch size capability (more compression) but reduce accuracy beyond size 8 — the paper identifies this as the sweet spot where the chunk mean is still a good proxy for individual key vectors (Figure 5, middle).
48 outlier chunks stored as static GPU cache: justified by Table 15 (Appendix A.8), showing accuracy recovery from 74.05 (0 outliers) to 86.88 (48 outliers), matching full attention at 86.68 — the outliers are predominantly attention sinks (first chunk) and regions of high key vector diversity.
Only values offloaded to CPU, keys stored as low-rank on GPU: justified by the asymmetric compressibility of keys vs. values (Figure 5, left), and by the ability to overlap key reconstruction (GPU computation) with value fetching (PCIe transfer) — this would not be possible if keys were also offloaded.
Temporal cache with index-scan hit/miss logic: justified by the 60-80% KV cache hit rate across decoding steps (Figure 5, right), which cuts reconstruction and fetching overhead by more than half.
Sparse budget of 1.56% (256 chunks of size 8): justified by Figure 8, showing accuracy on RULER subtasks stabilizes at or before this budget, with minimal gains from larger budgets — this budget matches the 1/16 computational cost used for fair comparison with Quest and other baselines.

4. Key Insights and Innovations

Innovation 1: Redrawing the Storage-Access Boundary — Keys Compressed, Values Offloaded

The dominant assumption across prior KV cache compression work is that keys and values are compressed symmetrically — both are quantized, both are evicted, or both are offloaded together. Methods like KIVI apply different quantization granularities to keys and values, but still treat both as needing the same type of treatment (bit-width reduction). Palu compresses KV weight matrices — again, symmetrically.

ShadowKV makes a clean conceptual break: keys and values should be stored entirely differently because they serve fundamentally different roles in attention and exhibit fundamentally different mathematical structure. Keys participate in dot-product scoring with the query, which makes their directional information (the subspace they span) critical. Values are simply aggregated via the attention weights — no scoring, no directional matching. This functional asymmetry maps onto an empirical asymmetry: pre-RoPE keys are exceptionally low-rank (Figure 1, left; Figure 5, left), while values are not.

The intellectual move is not just "compress what's compressible." It's recognizing that decoupling key storage from value storage — and decoupling GPU-resident (compressed keys) from CPU-resident (values) — creates a new degree of freedom in system design that prior work never exploited. The keys remain on GPU in compressed form because they're needed to perform the selection itself (computing approximate attention scores, reconstructing exact scores for selected tokens). The values can be offloaded because they're only needed after selection, and their retrieval can be overlapped with key reconstruction. This storage decoupling is what enables the overlapping strategy (CUDA multi-streams, Section 4.2) that breaks the PCIe bottleneck that doomed prior offloading approaches like InfiniGen.

The significance extends beyond raw performance. This insight implies that future KV cache compression research should not treat keys and values as a monolithic unit but as two distinct resources with different access patterns, compressibility properties, and latency tolerances. The concept of asymmetric KV cache management — applying qualitatively different storage strategies to keys versus values — is a framing contribution that generalizes beyond the specific mechanisms in this paper.

Evidence: The accuracy results in Figure 5 (left) make the empirical case — pre-RoPE keys at rank 160 achieve full accuracy while post-RoPE keys and values at the same rank do not. The memory savings formula in Appendix A.2 and the equivalent bandwidth model in Section 4.2 make the systems case — the gains come from the asymmetry, not just from compression.

Innovation 2: The Difficulty Is Not Missed Tokens, It's Missed Chunks — Chunk-Level Locality as a Diagnostic Tool

Prior sparse attention methods treat individual tokens as the unit of selection. Quest selects token pages, but its selection logic operates by approximating per-token attention scores within each page. Loki uses per-token PCA-based selection. InfiniGen prefetches individual KV entries. The field's default framing is: "which tokens are important?"

ShadowKV reframes the problem around chunks as the natural atomic unit, justified not by computational convenience but by an empirical diagnostic: the measurement of within-chunk cosine similarity (Figure 5, middle). The observation that most post-RoPE key vectors within a chunk of size 8 have high cosine similarity to their chunk mean is not an algorithmic trick — it's a structural property of how attention representations organize themselves. The existence of rare "outlier chunks" where this property breaks down is equally important: it tells us where the chunk-level approximation fails and how to compensate (preserving the full KV pairs for those chunks as static cache).

This is a diagnostic contribution masquerading as an algorithmic one. The paper doesn't just propose a chunk-based selection method — it provides an empirical methodology for determining when chunk-level approximation is valid and when it isn't. The minimum-within-chunk-cosine-similarity metric introduced in Section 3.2 is a lightweight diagnostic that can be applied to any model to determine: (1) what chunk size is appropriate, (2) how many outlier chunks exist, and (3) whether chunk-based approximation is viable at all for a given architecture. This is fundamentally different from prior work that treated chunk size as a hyperparameter to be tuned by trial and error on downstream accuracy.

The significance: this diagnostic reframes the sparse attention problem from "design a good selection heuristic" to "understand the structure of the key cache, then exploit that structure." The chunk-level locality finding (that most chunks are internally homogeneous) is an empirical claim about transformer representations that, if it generalizes across architectures and tasks, has implications beyond ShadowKV — it suggests that attention patterns have a characteristic spatial scale that efficient algorithms should exploit. The 0.2–0.3% outlier fraction (Table 15) quantifies just how rare the "hard" chunks are, explaining why a simple mean-based approximation works so well: the key cache is mostly redundant at the chunk level, with only tiny islands of complexity.

Evidence: Figure 5 (middle) visualizes the chunk similarity distribution, showing the cluster of normal chunks near 1.0 similarity and the sparse outliers dipping below 0.0. Table 15 quantifies the accuracy impact of preserving different numbers of outlier chunks, showing that just 8 outliers (0.049%) recover most of the accuracy gap, while 48 (0.293%) matches full attention at 86.88 vs. 86.68.

Innovation 3: Verifier-Style Selection Without a Verifier — Landmarks as Compressed Proxies

This is the subtlest conceptual contribution. In the PRM/ORM literature (as discussed in the reference example paper), verifiers are learned models that score solution quality — they are separate from the generator, trained on ground-truth outcomes, and used to select among candidates. ShadowKV implements a form of in-model, geometry-based selection that achieves what verifiers do (identifying which parts of the context are important for the current query) without any learned component, without training, and without external supervision.

The mechanism is the landmark-guided approximate attention scoring (Algorithm 2). The landmarks — chunk means of post-RoPE keys — serve as compressed proxies for the full key cache. Computing attention against these landmarks produces scores that approximate the exact attention scores. Selecting the top- $k$ chunks based on these approximate scores is functionally equivalent to a verifier selecting which KV pairs to attend to — but the "verifier" is not a separate model; it's the model itself, using its own key representations at reduced resolution.

What makes this intellectually distinctive is that it inverts the usual relationship between compression and selection. In prior work, compression and selection are separate: you compress the KV cache (via quantization, eviction, or low-rank decomposition), and then you need a separate mechanism to decide what to access (attention scores, heuristics, learned verifiers). ShadowKV's landmark construction makes compression serve selection directly — the same chunk means that reduce storage also enable fast approximate attention scoring. There's no separate selection model, no training, no hyperparameter tuning for a selection policy.

The comparison to InfiniGen is instructive. InfiniGen uses offline SVD to create projection matrices for KV selection — a fixed, non-adaptive "verifier" that must generalize across all prompts. ShadowKV's landmarks are prompt-dependent by construction (they're computed from the actual key cache of the current sequence) and query-dependent at runtime (the approximate attention scores depend on the query). This adaptation costs nothing extra because the landmarks are already being built during pre-filling for storage purposes. The selection quality inherits from the geometric structure of the key cache itself, not from a separately optimized selection policy.

Evidence: Figure 8 shows ShadowKV matching or exceeding Quest's accuracy at the same sparse budget across RULER subtasks. Quest uses a more complex selection mechanism (page-level max-attention approximation), but ShadowKV's simpler, geometry-based approach performs comparably or better. Table 1 shows ShadowKV achieving 86.88 average on RULER (Llama-3-8B-1M) compared to Quest's 82.03 at the same 1.56% sparse budget — a 4.85-point gap that suggests the landmarks provide better selection fidelity than Quest's heuristic.

Innovation 4: Test-Time Compute Allocation Through Sparse Budget Design — Not More Compute, Smarter Access

The reference example paper on compute-optimal test-time scaling asks: given a fixed FLOPs budget, how should inference compute be allocated across strategies (beam search, best-of-N, revisions)? ShadowKV asks an analogous but structurally different question: given a fixed memory budget, how should KV cache access be allocated across the sequence?

The concept of a sparse KV budget (1.56% in the paper's default configuration) is essentially a form of test-time compute allocation — but instead of allocating FLOPs across candidate solutions, ShadowKV allocates attention computation across tokens. The key insight: you can spend your attention budget (how many KV pairs you compute exact attention over) very differently across sequences and even within a sequence without degrading accuracy, as long as the allocation follows the structure of the key cache.

This framing connects ShadowKV to the broader test-time compute scaling literature in a way the paper doesn't explicitly make, but which is intellectually significant. The chunk-based top- $k$ selection with outlier preservation is a budget-constrained optimization: for each decoding step, given a fixed token budget (1.56% of the full sequence), select the subset of the KV cache that maximizes attention accuracy. The landmark-based approximate scoring solves this optimization cheaply. The outlier preservation ensures the budget isn't wasted on chunks where the approximation is poor.

The significance is that this reframes sparse attention from an accuracy-vs-efficiency tradeoff (less attention = lower quality) to an allocation problem (same total attention, but distributed intelligently). The finding that ShadowKV sometimes outperforms full attention on certain tasks (Table 1: variable tracking improves from 78.54 to 81.67; frequent words extraction improves from 71.85 to 72.57 for Llama-3-8B-1M) suggests that full attention is over-allocating computation to irrelevant tokens, and that the sparse selection actually improves signal-to-noise by filtering out distraction. This is analogous to the finding in the reference example that beam search can hurt easy problems at high budgets through verifier over-optimization — here, full attention can hurt by attending to tokens that add noise rather than signal.

Evidence: Table 1 shows ShadowKV achieving 86.88 average vs. 86.68 for full attention on RULER (Llama-3-8B-1M) — statistically indistinguishable, but with 1.56% of the attention computation. The individual task improvements (VT: 78.54 → 81.67, FWE: 71.85 → 72.57) are suggestive of the "filtering out distraction" hypothesis. Figure 8 shows accuracy as a function of sparse budget for individual tasks — the curves show that beyond 1.56%, additional budget provides minimal gains, consistent with the allocation-framing claim that the budget is sufficient when well-targeted.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three long-context benchmarks: RULER (Hsieh et al., 2024), consisting of 13 complex tasks including retrieval, multi-hop tracking, aggregation, and QA, with adjustable context lengths up to 1M tokens; LongBench (Bai et al., 2023), featuring 6 main categories and 21 diverse tasks spanning single-/multi-document QA, summarization, code completion, and information retrieval across Chinese and English, evaluated only on samples exceeding 4K tokens; and Needle In A Haystack (NIAH) (Kamradt, 2023), a retrieval benchmark testing whether the model can locate information placed at various positions across context windows ranging from 16K to 1M tokens. Additional experiments use InfiniteBench (Zhang et al., 2024), a 10-task benchmark averaging 214K tokens covering QA, coding, dialogue, summarization, and retrieval. The paper also introduces a custom Multi-turn NIAH benchmark to simulate multi-turn conversations requiring flexible access to distributed contextual information across turns.
Base model(s). Six open-weight long-context LLMs are evaluated: Llama-3-8B-1M (Gradient, 2024, supporting 1M-token contexts with 8 KV heads), Llama-3.1-8B (Meta, 2024, supporting 128K-token contexts with 8 KV heads), GLM-4-9B-1M (GLM Team, 2024, supporting 1M-token contexts with 4 KV heads), Yi-9B-200K (01.AI, 2024, supporting 200K-token contexts with 4 KV heads), Phi-3-Mini-128K (Abdin et al., 2024, supporting 128K contexts), and Qwen2-7B-128K (Yang et al., 2024, supporting 128K contexts). The models span different architectures, KV head counts, and context capacities, providing coverage across the design space. Additionally, Llama-3-70B-1M is tested on RULER at 512K and NIAH up to 1M to demonstrate scalability to larger model sizes (Table 11, Figure 10).
Metrics. The primary accuracy metric is task-specific performance on each benchmark: for RULER, average accuracy across 13 subtasks (including single-key and multi-key needle retrieval, multi-query, multi-value, QA, variable tracking, and frequent word extraction); for LongBench, task-specific and average scores across 9 subtasks; for NIAH, binary retrieval accuracy at each context length and depth position. The throughput metric is tokens per second during decoding, measured on an A100 GPU with various batch sizes and context lengths. A secondary efficiency metric is batch size scaling factor — how many more sequences can be served simultaneously compared to full attention at the same context length. The paper also reports GPU memory savings as a multiple of the original KV cache memory footprint (Appendix A.2).
Baselines. Three dynamic sparse attention methods serve as primary baselines: Quest (Tang et al., 2024), which segments tokens into pages and selects pages by approximating the highest attention within each page; Loki (Singhania et al., 2024), which performs PCA on key caches using a calibration dataset and selects tokens based on attention scores in low-dimensional space; and InfiniGen (Lee et al., 2024), which offloads the entire KV cache to the CPU and uses predefined SVD projections for KV selection. For each baseline, two variants are reported: one with the full KV cache offloaded to the CPU (matching ShadowKV's memory reduction), and another with only the value cache offloaded, marked as "(V)" in tables (matching ShadowKV's sparse KV budget of 1.56% but with higher GPU memory usage). For the multi-turn NIAH experiment (Figure 7), two eviction-based baselines are included: SnapKV (Li et al., 2024) and StreamingLLM (Xiao et al., 2023). The efficiency evaluation (Tables 3, 4) compares against Full Attention (the largest batch size fitting entirely on GPU with exact attention) and an Infinite Batch Size baseline that assumes infinite GPU memory and computes attention at A100's theoretical memory bandwidth of 2 TB/s.
Generation budget / compute accounting. The sparse KV cache budget is set to 1.56% of full attention for fair comparison across methods — this means each decoding step computes exact attention over 2048 selected tokens (256 chunks of size 8) plus 384 outlier tokens (48 chunks of size 8), totaling 2432 tokens out of 128K, or approximately 1.9% of the full sequence length but matching the 1.56% computational budget used for baselines like Quest. The paper explicitly states that "the computation cost is set to 1/16 of full attention for selecting sparse KV pairs" (Section 5.1, Baselines paragraph). For ShadowKV, the default configuration uses SVD rank $r = 160$ , chunk size $c = 8$ , number of outlier chunks $o = 48$ , and number of selected chunks $k = 256$ (Section 5.1 setup). The temporal cache hit rate $\alpha$ is measured empirically as 60–80% (Figure 5, right) and reduces the effective per-step reconstruction and fetching overhead. Pre-filling is performed with exact attention for all methods; only the decoding phase uses sparse attention. The SVD overhead during pre-filling is measured as a percentage of total pre-filling time (Figure 1, right; Table 12) and shown to be under 1% at 512K context lengths.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. All accuracy results are reported on standard benchmark test sets without train/validation splits. The methodology for selecting hyperparameters (rank = 160, chunk size = 8, outliers = 48, sparse budget = 1.56%) involves ablation studies on individual parameters measured against downstream accuracy on the test benchmarks (Figures 8, 9; Table 15), but no held-out validation set is used to tune these parameters — they are fixed across all models and all benchmarks. For the FLOPs-matched comparison section, external comparisons are not present because this is a systems paper rather than an algorithmic training paper.

Main Quantitative Results

The paper's evaluation separates naturally into three axes: accuracy preservation (does ShadowKV match full attention?), throughput scaling (does ShadowKV enable larger batch sizes and higher token rates?), and comparative advantage over baselines (does ShadowKV outperform Quest, Loki, and InfiniGen at equivalent configurations?).

Accuracy Preservation: Matching Full Attention Across Models and Benchmarks

RULER (Table 1, left side). On the RULER benchmark at 128K context, ShadowKV with a sparse budget of 1.56% achieves an average accuracy of 86.88 for Llama-3-8B-1M, compared to 86.68 for full attention — a statistically negligible 0.20-point difference. This pattern holds across models: for GLM-4-9B-1M, ShadowKV achieves 85.62 vs. full attention's 86.82 (a 1.20-point gap); for Llama-3.1-8B, ShadowKV achieves 83.57 vs. 85.53 (a 1.96-point gap); for Yi-9B-200K, ShadowKV achieves 65.53 vs. 65.73 (a 0.20-point gap). In no case does ShadowKV show catastrophic degradation on any model.

Critically, ShadowKV's accuracy is not just close to full attention — it exceeds full attention on certain subtasks. For Llama-3-8B-1M, ShadowKV improves Variable Tracking from 78.54 to 81.67 and Frequent Words Extraction from 71.85 to 72.57. For GLM-4-9B-1M, ShadowKV improves Frequent Words Extraction from 72.22 to 68.06 (a slight drop on this subtask) but maintains or exceeds full attention on Multi-Query (98.70 vs. 99.74) and QA-2 (55.21 vs. 55.21). This suggests that the sparse selection mechanism may sometimes filter out distracting tokens, improving signal-to-noise for tasks requiring precise information localization.

The comparison against baselines is stark. At the same 1.56% sparse budget with value-only offloading (the "(V)" variants), Quest achieves 83.99 for Llama-3-8B-1M — ShadowKV leads by 2.89 points. Loki (V) achieves only 24.46 — a 62.42-point gap. InfiniGen (V) achieves 78.31 — an 8.57-point gap. The gaps are most pronounced on complex multi-hop and multi-query tasks: on N-MK2 (multi-key needle retrieval with two keys), ShadowKV achieves 98.96 while Quest (V) achieves 85.42, Loki (V) achieves 1.04, and InfiniGen (V) achieves 76.04 for Llama-3-8B-1M. This demonstrates that ShadowKV's landmark-based selection with outlier preservation is substantially more robust than competing methods' KV selection strategies, particularly for tasks requiring retrieval of multiple pieces of information from different locations in the context.

When full KV offloading is used for all methods (both keys and values on CPU), the accuracy gap widens further due to the additional latency constraints. Quest drops to 82.03 (4.85 points below ShadowKV's 86.88), InfiniGen drops to 70.13 (16.75 points below), and Loki collapses to 9.33 (77.55 points below) for Llama-3-8B-1M. The Loki result deserves special attention: the method's PCA-based selection, trained offline on calibration data, completely fails when the KV cache is offloaded, suggesting its selection mechanism is fragile to the distribution shift between training and deployment conditions — precisely the type of cross-sequence subspace variation that ShadowKV's online, prompt-dependent approach avoids.

LongBench (Table 1, right side). On LongBench, evaluated on samples exceeding 4K tokens, ShadowKV maintains accuracy with the same 256-chunk sparse budget. For Llama-3-8B-1M, ShadowKV achieves an average of 39.94 across 9 subtasks, compared to full attention's 39.86 — a 0.08-point improvement. For GLM-4-9B-1M, ShadowKV achieves 47.89 vs. 48.24 (0.35-point gap). For Llama-3.1-8B, ShadowKV achieves 48.13 vs. 48.96 (0.83-point gap). For Yi-9B-200K, ShadowKV achieves 36.85 vs. 37.41 (0.56-point gap). In all cases, the accuracy is effectively indistinguishable from full attention.

The baseline comparisons follow the same pattern as RULER. Quest (V) achieves 38.17 for Llama-3-8B-1M — 1.77 points below ShadowKV. Loki (V) achieves 21.74 — 18.20 points below. InfiniGen (V) achieves 35.00 — 4.94 points below. The gaps are most pronounced on tasks like LCC (code completion): ShadowKV achieves 63.93 while Quest achieves 60.05, Loki achieves 38.10, and InfiniGen achieves 45.53.

Needle In A Haystack (Figure 6, Figure 11). ShadowKV's retrieval heatmaps for Llama-3-8B-1M (Figure 6) show perfect or near-perfect retrieval accuracy (green) across all context lengths from 16K to 1M and all depth percentiles from 0 to 100. The visual pattern is virtually indistinguishable from full attention's expected heatmap (the paper does not show full attention's heatmap for direct comparison, but reports identical or near-identical accuracy in the RULER needle retrieval subtasks). For GLM-4-9B-1M, Llama-3.1-8B, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K (Figure 11), ShadowKV maintains the same retrieval capability as full attention across their respective context windows, with no systematic degradation at any depth or length. The Yi-9B-200K case is notable: the paper states there is "even a slight performance improvement" with ShadowKV compared to full attention.

InfiniteBench (Table 16, Appendix B.4). On InfiniteBench's 10 tasks averaging 214K context length, ShadowKV achieves accuracy within 1–3 points of full attention across all models and subtasks. For Llama-3-8B-1M, ShadowKV averages across all subtasks with no significant degradation (specific averages not provided in the main text; Table 16 shows task-level results). The most notable gap appears on En.Sum (English summarization) for GLM-4-9B-1M: ShadowKV achieves 23.22 vs. full attention's 28.61 — a 5.39-point drop. However, this is model-specific and not systematic across other models (Llama-3.1-8B shows only a 2.19-point drop on the same task). The paper does not investigate why this particular model-task combination is more sensitive to sparse attention.

Multi-turn NIAH (Figure 7). This experiment tests a crucial capability that eviction-based methods fundamentally cannot support: flexible access to different contextual information across conversation turns. Full attention maintains high accuracy across all 8 turns. ShadowKV tracks full attention closely across all turns, with no visible degradation. In contrast, SnapKV's accuracy drops dramatically after turn 1 — from approximately 0.95 to approximately 0.20 by turn 3 and near zero thereafter. StreamingLLM follows a similar collapse. This demonstrates the fundamental advantage of methods that preserve all information (even in compressed form) over methods that permanently discard tokens: when a future query requires information that was deemed unimportant by an earlier query's attention patterns, eviction-based methods have no recovery path, while ShadowKV (and full attention) can retrieve it.

Multi-turn Llama-3-8B-1M are evaluated at 1M contexts (Table 11, top section). ShadowKV achieves 87.86 average on RULER, essentially matching full attention at 86.82. For the larger Llama-3-70B-1M at 512K, ShadowKV achieves 74.87 vs. full attention at 75.23 — a 0.36-point gap. This demonstrates that ShadowKV's compression and selection mechanisms scale to larger models without additional accuracy cost.

Integration with MInference (Table 2). When combined with MInference (Jiang et al., 2024), an efficient pre-filling method that also uses dynamic sparse attention, ShadowKV maintains accuracy across context lengths from 8K to 256K. At 128K, ShadowKV + MInference achieves 78.32 vs. full attention + MInference's 78.12 — a 0.20-point improvement. The averages across all context lengths are 82.04 (ShadowKV + MInference) vs. 81.98 (full attention + MInference), demonstrating that ShadowKV is compatible with pre-filling acceleration techniques without accuracy loss.

Throughput Scaling: Enabling 6× Larger Batches and Up to 3.04× Higher Token Rates

Batch size scaling (Table 4). The memory savings from ShadowKV's low-rank key cache and offloaded value cache translate directly into larger batch sizes. For Llama-3-8B-1M at 60K context, full attention reaches OOM at batch size 12 (maximum batch size 8 yields 160.62 tokens/s), while ShadowKV scales to batch size 48 (producing 455.14 tokens/s) — a 6× increase in batch size. At 122K context, full attention reaches OOM at batch size 5 (maximum 4 yields 80.77 tokens/s), while ShadowKV scales to batch size 24 (239.51 tokens/s) — a 6× increase. At 244K context, full attention can only fit batch size 2 (40.37 tokens/s), while ShadowKV reaches batch size 12 (119.01 tokens/s) — again a 6× increase. At 488K context, full attention is completely OOM even for batch size 1, while ShadowKV can still serve batch sizes up to 5 (53.46 tokens/s for batch size 5, tapering to OOM at 6). The batch size scaling factor is remarkably consistent at 6× across context lengths, suggesting that ShadowKV's memory savings (calculated at 7.08× in Appendix A.2) are effectively realized in practice.

Throughput improvements (Table 3). The larger batch sizes translate into substantial throughput gains. For Llama-3.1-8B at 60K context, ShadowKV achieves 472.77 tokens/s with batch size 48, compared to full attention's 160.93 tokens/s with batch size 8 — a 2.94× improvement. At 122K context, ShadowKV achieves 245.90 tokens/s with batch size 24 vs. full attention's 80.78 tokens/s with batch size 4 — a 3.04× improvement, the highest reported gain. For Llama-3-8B-1M, the improvements are 2.83× (60K), 2.97× (122K), and 2.95× (244K). For GLM-4-9B-1M, which has only 4 KV heads (reducing the relative benefit of key compression), the improvements are 2.56× (60K), 2.39× (122K), and 2.23× (244K) — still substantial but lower than the 8-KV-head models, as expected since fewer KV heads means less total memory can be saved through key compression. For Yi-9B-200K (also 4 KV heads), the improvements are 2.66× (60K), 2.56× (122K), and 2.54× (244K).

Comparison to infinite batch size (Table 3, "Full Attn (Inf)" column). The paper makes the striking claim that ShadowKV's throughput exceeds what would be possible with infinite GPU memory under full attention. The "infinite batch size" baseline computes attention at the A100's theoretical memory bandwidth of 2 TB/s, assuming no memory capacity constraints. For Llama-3.1-8B at 60K context, ShadowKV's 472.77 tokens/s substantially exceeds the infinite-batch full-attention throughput of 273.07 tokens/s — a 1.73× advantage. At 122K, ShadowKV achieves 245.90 vs. 134.30 (1.83×). This counterintuitive result occurs because full attention, even with infinite GPU memory and ideal bandwidth, must still read all $2S$ KV vectors for every decoding step. ShadowKV reads far fewer bytes through a combination of compression, sparsity, and temporal caching — and most of those bytes come from GPU memory rather than PCIe. The effective data movement is lower than what full attention would require even under ideal conditions.

Detailed throughput under varying batch sizes and context lengths (Table 4). The complete throughput matrix for Llama-3-8B-1M shows the scaling behavior comprehensively. At 60K context, ShadowKV's throughput scales from 89.69 tokens/s (batch size 2) to 455.14 (batch size 48) — a 5.07× increase in throughput for a 24× increase in batch size, reflecting diminishing returns from GPU utilization saturation. At 488K context, ShadowKV operates in a constrained regime: batch size 2 achieves 29.82 tokens/s, scaling to 53.46 at batch size 5, then hitting OOM. The full attention columns show identical throughput at the same batch sizes for those few configurations that fit (e.g., 89.19 vs. 89.69 at batch size 2 for 60K), confirming that ShadowKV introduces negligible per-step overhead — it simply enables configurations that were previously impossible.

Comparative Advantage Over Baselines at Equivalent Sparse Budgets

Figure 8 (sparse budget sweep). The paper systematically varies the sparse KV cache budget from 0.20% to 100% (full attention) and compares ShadowKV against Quest on four RULER subtasks for both Llama-3-8B-1M and GLM-4-9B-1M. On Multi-keys NIAH (multi-key needle retrieval), ShadowKV maintains accuracy above 0.95 at 1.56% budget while Quest drops to approximately 0.80 for Llama-3-8B-1M. At 0.78% budget, ShadowKV still achieves approximately 0.90 while Quest falls below 0.70. On Variable Tracking, ShadowKV achieves approximately 0.86 at 1.56% while Quest achieves roughly 0.85 — a smaller but consistent gap. On Frequent Words Extraction, both methods show similar scaling curves for Llama-3-8B-1M, with ShadowKV slightly ahead at low budgets. For GLM-4-9B-1M, the patterns are similar but with wider gaps: on Multi-keys NIAH at 1.56%, ShadowKV achieves approximately 1.0 while Quest reaches about 0.82. This demonstrates that ShadowKV's landmark-based selection is not just competitive but consistently superior at the same sparse budget, particularly for tasks requiring precise retrieval of information from specific locations.

LongBench (Table 1, right columns, and Table 8 for Yi-9B-200K). The per-subtask breakdown reveals where ShadowKV's advantage is largest. For Llama-3-8B-1M on GovRep, ShadowKV achieves 31.62 vs. Quest (V)'s 29.49 — a 2.13-point gap. On SAMSum, ShadowKV achieves 35.87 vs. 31.65 — a 4.22-point gap. On LCC (code completion), ShadowKV achieves 63.93 vs. 60.05 — a 3.88-point gap. These subtasks all require precise retrieval of specific content from long contexts, where chunk-level selection fidelity matters most. On tasks where the answer requires aggregating information broadly across the context (e.g., NarratQA, DuRead), the gap is smaller or ShadowKV sometimes trails Quest slightly — for Llama-3-8B-1M on DuRead, Quest achieves 27.11 while ShadowKV achieves 31.77 (ShadowKV leads by 4.66 points), but on NarratQA, ShadowKV achieves 17.17 vs. Quest's 20.13 (Quest leads by 2.96 points). The paper does not investigate why NarratQA favors Quest's selection mechanism, but the difference may relate to the task's need for broader, less localized attention patterns.

Efficiency comparison under CPU offloading constraints (Table 14). When the GPU memory alone cannot accommodate the KV cache even for a single sequence (as occurs with full attention and Quest at 3×1M contexts), both must resort to CPU offloading. Under these conditions, ShadowKV with its 1.56% sparse budget achieves 45.32 tokens/s, compared to Quest's 9.34 tokens/s and full attention's 0.21 tokens/s — a 4.85× advantage over Quest and a 215× advantage over full attention. The Quest result is particularly informative: even though Quest uses the same sparse budget, it must fetch both keys and values from the CPU, while ShadowKV only fetches values and reconstructs keys from the GPU. ShadowKV's 4.85× throughput advantage demonstrates the concrete benefit of the asymmetric storage design.

Ablation Studies and Robustness Checks

Sparse KV cache budget (Figure 8): ShadowKV maintains higher accuracy than Quest across all sparse budgets from 0.20% to 100% on four representative RULER subtasks for both Llama-3-8B-1M and GLM-4-9B-1M. The accuracy curves for ShadowKV flatten at approximately 1.56% budget, with minimal gains from larger budgets, validating this as the default operating point. On Multi-keys NIAH, ShadowKV's advantage is largest at low budgets (0.78–1.56%), suggesting the landmark approximation is particularly effective when the selection is most constrained. For GLM-4-9B-1M, the curves show wider separation across all budgets, indicating that ShadowKV's selection mechanism is more robust to model architecture differences than Quest's page-level approach.

Chunk size (Figure 9, left): Increasing chunk size from 1 to 64 increases the batch size scaling factor from approximately 2× to 9× (more compression enables larger batches), but accuracy on RULER and multi-key NIAH declines when chunk size exceeds 8. At chunk sizes 1–4, accuracy is at or near maximum but batch size scaling is limited to 2–4×. At chunk sizes 16–64, batch scaling reaches 6–9× but accuracy degrades substantially. The paper selects chunk size 8 as the sweet spot where batch scaling reaches approximately 6× while maintaining full accuracy. Interestingly, the chunk hit rate (the temporal cache hit rate) remains around 60% across all chunk sizes — the spatial locality property that enables the cache mechanism is robust to chunk granularity.

Pre-RoPE key cache SVD rank (Figure 9, right): Accuracy on five RULER subtasks (NIAH-S, NIAH-MK, NIAH-MQ, FWE, QA) is measured across ranks from 0 to 1000. All subtasks show accuracy increasing with rank up to approximately 160, after which it stabilizes near full-rank performance. NIAH-MK2 (multi-key needle retrieval with 2 keys, the most challenging retrieval task) shows the most sensitivity: accuracy rises from approximately 0.40 at rank 0 to 0.98 at rank 160, then plateaus. Other tasks show earlier saturation. The paper reports an interesting finding: "in some cases, low-rank approximations achieve better performance" — for certain tasks, intermediate ranks (160–320) slightly exceed full-rank accuracy, possibly due to the low-rank approximation acting as a form of regularization that filters noise in the key representations.

Number of outlier chunks (Table 15): A systematic sweep from 0 to 48 outlier chunks on Llama-3-8B-1M with RULER at 128K reveals a clear pattern: with 0 outliers, average accuracy drops to 74.05 (vs. 86.68 for full attention, a 12.63-point degradation), primarily due to collapse on complex tasks like Multi-Query (70.83 vs. 95.57) and Variable Tracking (73.54 vs. 78.54). Adding just 1 outlier chunk (the attention sink, corresponding to the first chunk) recovers to 85.01 — an 10.96-point jump that demonstrates the critical importance of the attention sink. 8 outliers (0.049% of chunks) achieves 86.22, within 0.46 points of full attention. 48 outliers (0.293%) achieves 86.88, matching full attention's 86.68. The paper notes that the first chunk (attention sink) is a significant outlier, and that without adequate outlier handling, "the performance of the mean-based landmarks in ShadowKV may fall below the min-max approach used by Quest" — an important calibration that the mean-based approximation only works because outliers are separately preserved.

Precision sensitivity (Tables 9 and 10, Appendix A.4): ShadowKV with FP8 precision (torch.float8_e5m2) maintains accuracy on RULER and LongBench. For Llama-3-8B-1M on RULER, ShadowKV achieves 85.95 with FP8 vs. 84.94 for full attention FP8 — the SVD and reconstruction remain accurate even at reduced precision. Quest's FP8 accuracy drops to 81.44 (full offload) and 83.88 (value-only offload). On LongBench, ShadowKV FP8 achieves 39.25 vs. full attention FP8's 39.49 (0.24-point gap). The results confirm that ShadowKV's components (SVD, landmark computation, reconstruction) do not require high precision to function correctly.

Handling of newly generated tokens — ShadowKV+ (Tables 5 and 6, Appendix A.1): The ShadowKV+ extension, which stores newly generated tokens' key cache as low-rank projections using the same SVD matrices from pre-filling, maintains accuracy within 0.5 points of ShadowKV on RULER across all four models. For Llama-3-8B-1M, ShadowKV+ achieves 86.23 vs. ShadowKV's 86.88 (0.65-point gap). For LongBench, ShadowKV+ sometimes outperforms ShadowKV: on Llama-3-8B-1M, ShadowKV+ achieves 40.36 vs. ShadowKV's 39.94 (0.42-point improvement). The consistency across models and benchmarks validates the paper's claim (Section 3.1, Figure 1 middle) that future pre-RoPE keys within a sequence share the same low-rank subspace as the context.

Scalability to larger models and longer sequences (Table 11, bottom section; Figure 10): For Llama-3-8B-1M at 1M contexts on RULER, ShadowKV achieves 72.98 vs. full attention's 72.89 — indistinguishable. Quest with full offloading achieves only 55.91, and Loki collapses to 8.02. For Llama-3-70B-1M at 512K contexts, ShadowKV achieves 74.87 vs. full attention's 75.23 (0.36-point gap). The 70B model results are critical because they demonstrate that ShadowKV's components scale to model sizes where the KV cache memory pressure is even more acute — the 70B model has 80 layers with 8 KV heads each, making the total KV cache size for a 512K context enormous. The NIAH heatmap for Llama-3-70B-1M (Figure 10) shows near-perfect retrieval across all depths and context lengths from 16K to 1M.

Integration with MInference (Table 2): ShadowKV combined with MInference's efficient pre-filling maintains accuracy within 0.5 points of MInference + full attention across context lengths from 8K to 256K. At 256K, ShadowKV + MInference achieves 74.31 vs. 74.57 for full attention + MInference. The combination demonstrates that ShadowKV's decoding-time optimizations are orthogonal to pre-filling accelerations.

Efficiency comparison with baselines under 1M contexts (Table 14): For 3×1M context sequences, where full attention and Quest both require CPU offloading due to GPU memory exhaustion, ShadowKV achieves 45.32 tokens/s — 4.85× faster than Quest (9.34 tokens/s) and 215× faster than full attention (0.21 tokens/s). This ablation isolates the benefit of ShadowKV's asymmetric storage design: both Quest and ShadowKV use the same 1.56% sparse budget, but Quest must fetch keys and values over PCIe while ShadowKV reconstructs keys from GPU memory.

Critical Assessment

Claim 1: ShadowKV reduces GPU memory footprint by over 6× without accuracy degradation.

What was tested: The paper measures accuracy on RULER (128K), LongBench (>4K), NIAH (up to 1M), and InfiniteBench (~214K) across six models. The memory savings are calculated theoretically in Appendix A.2 (7.08× for 128K sequences with the default parameters) and demonstrated practically through the batch size scaling in Table 4 (6× larger batches). The accuracy results consistently match full attention within 1–2 points on average metrics, with occasional slight improvements.

What was demonstrated: The paper convincingly shows that for the tested benchmarks and models, ShadowKV matches full-attention accuracy while enabling approximately 6× larger batch sizes. The evidence is consistent across models, benchmarks, and context lengths. The outlier ablation (Table 15) provides mechanistic validation: without outliers, accuracy collapses; with just 8–48 outliers (<0.3% of chunks), full accuracy is recovered. This confirms that the chunk-level approximation plus outlier preservation the paper proposes is sufficient for the tested tasks.

Genuine weaknesses:

No dense retrieval tasks requiring full-sequence attention. All tested benchmarks (RULER needle retrieval, LongBench QA, NIAH) have answers localized to specific parts of the context. The paper does not test tasks where the answer requires aggregating information from most or all of the context — for example, summarizing a 128K document where every paragraph matters, or computing aggregate statistics across an entire codebase. In such tasks, the 1.56% sparse budget might be insufficient, and the chunk-level selection might miss important dispersed information. The LongBench results provide some evidence here: on tasks requiring broader context (NarratQA), ShadowKV sometimes trails Quest (17.17 vs. 20.13 for Llama-3-8B-1M), though the gap is modest.
Single precision regime (BF16/FP8). While the FP8 ablation (Tables 9, 10) shows robustness, all experiments use floating-point representations. The paper does not test with quantized KV caches (e.g., 4-bit or 2-bit as in KIVI), which is a common deployment optimization. The interaction between low-rank compression and quantization is unexplored — quantized low-rank representations might behave differently, and the SVD reconstruction could amplify quantization errors.
No ablation on the overlap between outlier chunk count and sparse budget. The paper fixes outliers at 48 and selected chunks at 256, but these parameters interact: with fewer outliers, would a larger selected chunk budget compensate? Or conversely, with more outliers, could the selected chunk budget be reduced? The paper's one-factor-at-a-time ablations don't explore the joint space, so it's unclear whether the default configuration is near-optimal or if better combinations exist.
The accuracy "improvements" over full attention are not tested for statistical significance. ShadowKV achieves 86.88 vs. 86.68 on RULER for Llama-3-8B-1M — a 0.20-point difference on a 500-example test set. This is almost certainly within sampling noise, but the paper does not provide confidence intervals or statistical tests. The claim that sparse attention "improves" accuracy on some subtasks (VT: 78.54 → 81.67, a 3.13-point gap) is more suggestive, but the paper does not investigate why this happens or whether it's reliable. If sparse attention genuinely filters distracting tokens, this would be a significant finding — but it's presented as an observation without mechanistic analysis.

Claim 2: ShadowKV supports up to 6× larger batch sizes and boosts throughput by up to 3.04×.

What was tested: Throughput is measured on an A100 with Llama-3-8B-1M, Llama-3.1-8B, GLM-4-9B-1M, and Yi-9B-200K at context lengths from 60K to 488K (Tables 3, 4). Batch sizes are determined by what fits in GPU memory. The latency breakdown (Tables 12, 13) provides per-operation timing.

What was demonstrated: The throughput claims are well-supported. The 6× batch size increase is consistent across context lengths (Table 4). The 3.04× throughput improvement is achieved at 122K for Llama-3.1-8B. The comparison to "infinite batch size" is particularly striking and well-justified through the equivalent bandwidth model.

Genuine weaknesses:

Single GPU architecture. All experiments use an A100 (80GB). The paper does not evaluate on H100 (which has higher memory bandwidth at 3.35 TB/s and higher PCIe bandwidth), A6000 (48GB, consumer-grade), or multi-GPU configurations. The equivalent bandwidth model (Section 4.2) predicts that ShadowKV's advantage would be even larger on GPUs with higher compute-to-memory-bandwidth ratios (like the H100, where the gap between tensor core throughput and memory bandwidth is wider), but this is untested.
No measurement of end-to-end latency for interactive use cases. All throughput numbers are for batched decoding. For interactive applications where a single user waits for a response, the latency per token matters more than total throughput. ShadowKV's per-step operations (landmark scoring, index scan, reconstruction, fetching) add latency compared to full attention, even if the overlapped version hides some of it. The paper reports the per-step latency breakdown in Table 13 (e.g., 0.21–0.29 ms for attention at various batch sizes), but does not compare this to full attention's per-step latency at the same batch sizes. For batch size 1 (not reported), the overlapping might be less effective because there's less parallelism to hide behind.
The pre-filling SVD cost is amortized over decoding length, and the paper only reports long-context scenarios. Table 12 shows SVD overhead decreasing from 6.65% at 64K to 0.97% at 512K. But what about shorter contexts where SVD overhead is higher? If a user submits a 16K context with a short output (e.g., 100 tokens of decoding), the SVD overhead during pre-filling might dominate the total compute. The paper doesn't explore the break-even point where ShadowKV's decoding savings outweigh the pre-filling SVD cost.
No comparison against FlashAttention with CPU offloading at equivalent batch sizes. The paper compares against Quest, Loki, and InfiniGen, but does not compare against a simple baseline: FlashAttention with the full KV cache offloaded to CPU, computing exact sparse attention (rather than approximate) with a simple heuristic like keeping the most recent tokens. This baseline would have lower pre-filling overhead (no SVD, no landmark construction) and might be surprisingly competitive for tasks where recency bias is strong.

Claim 3: ShadowKV outperforms all baselines while maintaining accuracy at equivalent sparse budgets.

What was tested: Tables 1, 7, 8 provide comprehensive comparisons against Quest, Loki, and InfiniGen across RULER, LongBench, and multiple models. Figure 8 sweeps the sparse budget across subtasks. Table 14 provides an efficiency comparison under extreme memory constraints.

What was demonstrated: ShadowKV consistently outperforms all baselines at the same sparse budget. The gaps are substantial: 2–5 points on RULER average, 10+ points on complex subtasks like Multi-keys NIAH, and 5× throughput advantage over Quest under full CPU offloading (Table 14). The evidence is robust across models and benchmarks.

Genuine weaknesses:

The "(V)" baseline variants are not fully fair. The "(V)" variants (value-only offloading) match ShadowKV's sparse budget but use more GPU memory because they keep the full key cache on GPU. This means they achieve lower memory savings than ShadowKV at the same sparse budget — a tradeoff the paper acknowledges but doesn't quantify in memory terms. The full-offloading variants are more memory-comparable but have higher latency. The paper doesn't report a single baseline configuration that simultaneously matches ShadowKV's memory, latency, and sparse budget — because none exists, which is precisely ShadowKV's contribution. But the comparison would be stronger with a Pareto frontier showing accuracy vs. memory vs. throughput for all methods.
InfiniGen's SVD-based selection might perform better with different projection dimensionality. The paper uses InfiniGen "as-is" from the original paper, but doesn't explore whether tuning InfiniGen's projection rank or calibration dataset could close the accuracy gap. InfiniGen's accuracy of 70.13 on RULER (Llama-3-8B-1M) vs. ShadowKV's 86.88 is a 16.75-point gap — large enough to suggest fundamental issues with offline projections, but potentially reducible with better calibration.
No comparison against simple hybrid baselines. A natural baseline would combine chunk-level selection (as in ShadowKV) with full key cache on GPU (as in Quest) — essentially, using ShadowKV's selection mechanism without the low-rank key compression. This would isolate how much of the accuracy advantage comes from the selection strategy vs. the compression strategy. The paper doesn't run this ablation.
The temporal cache mechanism's benefit is reported but not ablated. Figure 5 (right) shows 60–80% hit rates, and the paper claims this reduces computation by 60%, but there's no experiment showing throughput with the cache disabled. This makes it impossible to quantify how much of the 3.04× throughput gain comes from the cache vs. from the sparse budget vs. from the overlapping.

Claim 4: ShadowKV even surpasses the performance achievable with infinite batch size under the assumption of infinite GPU memory.

What was tested: Table 3's "Full Attn (Inf)" column computes theoretical throughput assuming all KV cache data is in GPU memory and accessed at 2 TB/s bandwidth with no capacity limits.

What was demonstrated: The numbers support the claim: ShadowKV at 122K with Llama-3.1-8B achieves 245.90 tokens/s vs. 134.30 tokens/s for the infinite-batch baseline — a 1.83× advantage. The equivalent bandwidth model (Section 4.2) provides the theoretical justification: ShadowKV moves fewer total bytes than full attention would even under ideal conditions.

Genuine weaknesses:

The "infinite batch" baseline assumes no kernel overhead, no scheduling overhead, and perfect bandwidth utilization. Real full attention implementations (even FlashAttention) achieve less than 100% of theoretical memory bandwidth. The 2 TB/s figure is a hardware peak, not an achievable sustained rate. This makes ShadowKV's advantage over "infinite batch" an upper bound — in practice, the advantage would be even larger if the baseline were measured rather than calculated.
The comparison is for decoding only, not end-to-end throughput. The infinite batch baseline doesn't account for pre-filling costs, which would be identical to ShadowKV's (since both use exact pre-filling in the default configuration). But for workloads with many pre-filling requests (e.g., batch evaluation of many independent prompts), the pre-filling phase dominates, and ShadowKV's SVD overhead (even at 1–5%) would reduce its advantage relative to a hypothetical infinite-batch baseline that doesn't need SVD.

Missing Experiments That Would Strengthen the Paper

Scaling study across number of KV heads. The paper notes that GLM-4-9B-1M and Yi-9B-200K (4 KV heads) show lower throughput gains than Llama models (8 KV heads): 2.23–2.66× vs. 2.83–3.04×. A systematic experiment varying KV head count (e.g., testing different model configurations or artificially grouping heads) would clarify how this architectural choice affects ShadowKV's benefits and whether the method is particularly suited to models with many KV heads (like Llama) at the expense of those with fewer (like GLM and Yi).
Accuracy on tasks requiring truly global attention. A stress test like "summarize a 128K document" or "find the passage that contradicts a given statement elsewhere in the document" would probe whether the 1.56% sparse budget is sufficient when the answer depends on tokens distributed throughout the entire context rather than localized to a few chunks. The paper's benchmarks are dominated by retrieval and QA tasks where information is somewhat localized — this may overstate ShadowKV's general applicability.
Interaction with different RoPE frequencies. The paper's key observation (pre-RoPE keys are low-rank) depends on the positional encoding scheme. Models using ALiBi, NoPE, or different RoPE base frequencies might exhibit different low-rank properties. Testing across models with different positional encoding schemes would establish the generality of the core empirical finding.
End-to-end serving system evaluation. The paper measures per-layer, per-step timings and projects to total throughput, but does not implement a full serving system with request queuing, dynamic batching, or variable-length inputs. A real deployment would have overheads (request scheduling, memory allocation, output processing) that could dilute the reported gains.
Comparison against prefix caching approaches. The paper mentions that the SVD could be precomputed and stored as part of a prefix cache (Section 1 footnote). But it doesn't compare against prefix-caching-based systems like Hydragen (Juravsky et al., 2024) or Cascade Inference (Ye et al., 2024), which exploit shared prefixes across requests to reduce per-request KV cache computation and memory. For workloads with shared prefixes (common in chatbot and document QA deployments), prefix caching might be more effective than per-sequence compression. ShadowKV would compose with prefix caching (the SVD could be cached for shared prefixes), but the paper doesn't explore this.
Latency at batch size 1 for interactive use. All throughput numbers assume large batches. But for a single user sending a query and waiting for a response, latency matters more than throughput. What is the time-to-first-token and per-token latency of ShadowKV vs. full attention for a single 128K sequence? The overlapping might be less effective without batch parallelism, and the PCIe transfers would be on the critical path.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted For — The SVD Overhead During Pre-Filling

The assumption or constraint. ShadowKV performs an online, prompt-dependent SVD on the pre-RoPE key cache during the pre-filling phase for every sequence. The paper acknowledges this cost explicitly:

"our experiments do not account for this cost largely for simplicity" (Section 3.1 discussion, though the exact quote refers to difficulty estimation; the SVD overhead is acknowledged in Section 4.1 and Appendix A.6 with the paper stating "the linear cost of low-rank decomposition during pre-filling [is] negligible" at long sequence lengths).

The paper argues that the SVD overhead becomes negligible as sequence length grows because SVD scales linearly ( $O(s \cdot d^2)$ ) while attention scales quadratically ( $O(s^2 \cdot d)$ ). Figure 1 (right) and Table 12 quantify this: at 64K, SVD is 6.65% of pre-filling time; at 128K, 3.25%; at 256K, 1.75%; at 512K, 0.97%.

The consequence. This overhead amortization argument only holds for long sequences. For short-to-medium contexts (under 32K), the SVD overhead may represent a substantial fraction of total pre-filling compute — precisely the regime where many production workloads operate (e.g., 8K–32K document QA, chatbot conversations). At 64K, a 6.65% overhead is already non-trivial if the decoding phase is short (e.g., generating a 50-token answer from a 64K context — the decoding represents a small fraction of total compute, so adding 6.65% to the dominant pre-filling phase directly increases end-to-end latency by approximately that amount). The paper never reports the break-even sequence length where ShadowKV's decoding savings outweigh the pre-filling SVD cost. If a deployment serves mostly short-context requests with short outputs, ShadowKV could be slower than full attention end-to-end, because the decoding savings are small (few decoding steps) while the pre-filling penalty is paid on every request.

A second, subtler consequence: the SVD is computed per-layer but the paper doesn't analyze whether all layers benefit equally from rank-160 compression. If some layers have less low-rank key caches than others, compressing them uniformly to rank 160 may either waste computation (layers that could be compressed more) or lose accuracy (layers that need higher rank). The uniform rank across all layers is a simplifying assumption that may not be optimal.

What evidence exists in the paper. Table 12 provides the per-operation latency breakdown for Llama-3-8B-1M pre-filling at four context lengths (64K, 128K, 256K, 512K). The SVD time is reported as an absolute latency and as a percentage of total Transformer block time. Figure 1 (right) plots SVD percentage vs. sequence length, showing the declining trend. However, the paper does not report SVD overhead for context lengths below 64K, which would be necessary to identify the break-even point for short-context deployments. It also does not report end-to-end latency comparisons that include both pre-filling and decoding for varying output lengths — only per-phase breakdowns and decoding-only throughput.

Mitigation status. The paper mentions in a footnote that "in practical scenarios, the key cache can be offloaded to the CPU to perform SVD asynchronously or precomputed and stored as part of the prefix cache" (Section 1, footnote 1). This suggests two possible mitigations — asynchronous SVD on CPU (which would hide but not eliminate the cost) and prefix caching for shared prefixes (which would amortize the SVD across multiple requests). Neither mitigation is implemented or evaluated in the paper. The prefix caching suggestion is particularly significant: for workloads with shared prefixes (e.g., multiple queries against the same document), computing the SVD once and reusing it would eliminate the per-request overhead. But this requires a serving infrastructure that supports prefix caching, which ShadowKV does not implement. The paper does not explore whether the low-rank subspace from a shared prefix SVD generalizes to different queries against that prefix — an important open question.

Limitation 2: Benchmark Coverage Is Skewed Toward Retrieval and Localized-Information Tasks

The assumption or constraint. All primary evaluations (RULER, LongBench, NIAH, InfiniteBench, Multi-turn NIAH) test tasks where the correct answer depends on information localized to specific parts of the context. Needle-in-a-haystack tasks require retrieving a specific fact; QA tasks require finding relevant passages; multi-hop tasks require chaining together specific pieces of information; variable tracking requires following specific variables through the text. The paper does not evaluate on tasks requiring dense global aggregation — synthesizing information distributed uniformly across the entire context, where the 1.56% sparse budget would necessarily miss most of the relevant content.

The paper states its evaluation covers "QA, multi-hop, reasoning, summarization, code completion" (Section 5.1), implying breadth. But the specific tasks within these categories all have answers that can be found by attending to a relatively small fraction of the context.

The consequence. The core concern is that ShadowKV's sparse attention budget (1.56% of tokens, selected as top-256 chunks of size 8) is fundamentally incompatible with tasks requiring global attention. A document summarization task on a 128K-token legal contract may require the model to understand provisions scattered across the entire document — provisions that individually occupy small fractions of the text but collectively span most of it. If ShadowKV selects only 1.56% of tokens, it has access to at most 1.56% of the document's content. For localized-retrieval tasks, this suffices because the relevant content is localized to a small subset of tokens. For global aggregation tasks, it may be insufficient by design.

There are suggestive signs of this limitation in the paper's own results. On the NarratQA subtask of LongBench (involving narrative understanding across long documents), ShadowKV achieves 17.17 for Llama-3-8B-1M while Quest achieves 20.13 (Table 1) — ShadowKV loses by 2.96 points, one of the few subtasks where a baseline outperforms it. NarratQA typically requires tracking events and characters across a narrative — a task with less extreme localization than needle retrieval. The paper does not investigate this gap, but it may indicate that Quest's page-level selection is better suited to tasks requiring broader context coverage.

What evidence exists in the paper. The LongBench evaluation (Table 1, right side) includes a range of tasks, and ShadowKV's performance is generally strong — but the subtasks where it shows the smallest advantage or slight disadvantages (NarratQA, DuRead) are those with the least localized information requirements. The paper does not include a systematic ablation varying the distribution of relevant information in the context (e.g., spreading a fixed number of "needles" across increasingly many chunks) to test at what sparsity level the sparse budget becomes insufficient.

Mitigation status. Not addressed. The paper does not discuss the tradeoff between sparse budget size and task locality, nor does it provide guidance on how to select the sparse budget based on the expected task type. The suggestion that 1.56% is sufficient for "a broad range of benchmarks" (Section 5.1) may not generalize to global-aggregation tasks. The chunk-level hit rate analysis (Figure 5, right) shows that the temporal cache works well for decoding steps that select similar chunks — but if a subsequent decoding step requires information from a completely different part of the context (as might be needed when switching from retrieving one fact to retrieving another), the hit rate would drop, and the system would need to fetch entirely new chunks.

Limitation 3: The "Infinite Batch Size" Comparison Is a Theoretically Motivated but Practically Incomplete Baseline

The assumption or constraint. The paper claims ShadowKV's throughput can "even surpass the performance achievable with infinite batch size under the assumption of infinite GPU memory" (Abstract, Section 5.2). The infinite batch baseline is calculated by assuming that full attention with all KV cache in GPU memory would achieve throughput limited only by the A100's theoretical memory bandwidth of 2 TB/s — i.e., the full attention operation reads $2S$ KV vectors at 2 TB/s, and this determines the throughput ceiling independent of batch size.

The consequence. This comparison, while intellectually elegant (formalized by the equivalent bandwidth model in Section 4.2), systematically underestimates what a real high-throughput serving system could achieve with more GPU memory. In practice, if GPU memory were truly infinite (or even just larger, as on an H100 with 188GB or an 8-GPU node with 640GB), the system could run larger batch sizes, exploit tensor core parallelism more effectively, and approach a throughput limited by compute rather than memory bandwidth. The 2 TB/s figure represents the memory bandwidth ceiling of a single A100, but an "infinite memory" system would not be limited by a single GPU's memory bandwidth — it could scale out across GPUs, each contributing additional memory bandwidth. The paper's comparison is really ShadowKV vs. a single A100 with infinite HBM capacity but the same 2 TB/s bandwidth — a scenario that doesn't exist and somewhat oversells the advantage.

Furthermore, the comparison does not account for the scheduling, kernel launch, and inter-layer data movement overheads that real full-attention implementations incur. FlashAttention achieves less than 100% of theoretical memory bandwidth due to these overheads. ShadowKV's throughput advantage over a real high-memory system (say, 8× A100s serving in parallel with full attention) would likely be smaller than the reported 1.73–1.83× over the infinite-batch theoretical baseline.

What evidence exists in the paper. The equivalent bandwidth model (Section 4.2) and Table 3's "Full Attn (Inf)" column provide the comparison. The paper does not compare against a multi-GPU deployment with full attention, which would be the practical alternative to ShadowKV for increasing throughput. It also does not compare against FlashAttention-3 or other optimized attention implementations that might achieve higher bandwidth utilization on newer hardware.

Mitigation status. The paper is transparent about the calculation method: "we evaluate a single Transformer block with FlashAttention and then project the number to the entire model. For the infinite batch size, we leverage A100's theoretical memory bandwidth (2 TB/s) for attention computations" (Section 5.2, footnote 5). This is a reasonable theoretical baseline for establishing the principle that sparsity can beat memory bandwidth limits, but it should not be interpreted as ShadowKV outperforming any real system with more memory. The paper does not discuss this distinction.

Limitation 4: All Experiments Use a Single GPU Architecture (A100); the Results Do Not Generalize to Other Hardware Configurations

The assumption or constraint. Every efficiency measurement in the paper (throughput, batch size scaling, latency breakdown) is conducted on a single NVIDIA A100 GPU with 80GB HBM and PCIe 4.0 (31.5 GB/s bandwidth to CPU). The relative performance of ShadowKV's components — GPU-resident key cache reconstruction, PCIe value fetching, landmark-based selection, and temporal caching — depends critically on the balance between GPU memory bandwidth, PCIe bandwidth, and GPU compute throughput, all of which vary across hardware generations.

The consequence. The 3.04× throughput improvement and 6× batch size scaling are hardware-specific numbers that may not hold on different GPUs. Consider key scenarios where the paper's conclusions might not transfer:

H100 (80GB or 188GB): The H100 has 3.35 TB/s memory bandwidth (1.68× higher than A100) and supports PCIe 5.0 (63 GB/s, 2× higher than A100). The higher memory bandwidth reduces ShadowKV's advantage because full attention would be less memory-bandwidth-bound. The higher PCIe bandwidth reduces the penalty for value fetching, but this benefit helps both ShadowKV and CPU-offloading baselines. The net effect is unclear without measurement.
Multi-GPU node (8× A100s or H100s): With tensor parallelism or pipeline parallelism across GPUs, the aggregate memory bandwidth scales, and full attention might achieve higher throughput than ShadowKV on a single GPU. The paper's single-GPU evaluation does not address whether ShadowKV remains beneficial in multi-GPU deployments, where the memory-per-GPU constraint is relaxed and bandwidth aggregation changes the tradeoff.
Consumer GPUs (RTX 4090, A6000): These have lower memory bandwidth relative to compute (the RTX 4090 has 1.0 TB/s memory bandwidth vs. massive compute throughput), which would make full attention even more memory-bound, potentially increasing ShadowKV's advantage. But they also have smaller VRAM (24GB), which makes the memory savings more critical — the 6× batch size scaling might actually be more impactful on these GPUs. The paper does not test any of these scenarios.

What evidence exists in the paper. None beyond the A100 measurements. The equivalent bandwidth model (Section 4.2) is parameterized by $B_{\text{GPU}}$ and $B_{\text{PCIe}}$ , which can be substituted for other hardware values, but the paper never validates the model's predictions against actual measurements on different hardware. The model also assumes perfect linear scaling of all components, which may not hold in practice due to fixed overheads (kernel launch latency, CUDA stream management, PCIe transaction overhead).

Mitigation status. The equivalent bandwidth formula provides a theoretical framework for predicting performance on other hardware, but it is not validated. The paper does not discuss hardware generality as a limitation or suggest hardware-specific tuning (e.g., different chunk sizes or sparse budgets for different GPU memory bandwidth ratios). A practitioner deploying on H100 or consumer hardware would need to re-profile all parameters.

Limitation 5: The Temporal Cache Mechanism's Benefit Is Reported but Not Isolated — It Is Unclear How Much of the Gain Comes from Caching vs. Sparsity vs. Overlapping

The assumption or constraint. ShadowKV reports a KV cache hit rate of 60–80% (Figure 5, right) and claims that the temporal cache mechanism "reduc[es] computations and data movements by over 60% for each decoding step" (Section 3.2, Figure 5 caption). However, the throughput measurements (Tables 3, 4) and latency breakdown (Table 13) include the cache mechanism as an integrated component of the system — there is no ablation that disables the cache and measures the resulting throughput.

The consequence. The reader cannot determine how much of ShadowKV's 3.04× throughput gain is attributable to the static design choices (low-rank key compression, value offloading, sparse budget) vs. the dynamic optimization (temporal caching of selected chunks). This matters for two reasons:

Reproducibility and tuning. If the cache contributes a large fraction of the gain, then the reported numbers depend on the specific hit rate, which in turn depends on the task, the model, and the decoding trajectory. A task where the model's attention shifts rapidly across the context (low hit rate) would see much smaller gains than the reported averages. The paper doesn't provide per-task hit rates or characterize when the cache performs well vs. poorly.
Comparison fairness. Quest and other baselines could also benefit from a temporal cache — the idea of caching recently accessed KV pairs across decoding steps is not novel (e.g., vLLM's PagedAttention already implements prefix caching). By including the cache as part of ShadowKV's design without giving it to baselines, the comparison may overstate ShadowKV's algorithmic advantage. The paper's Quest throughput numbers in Table 14 (9.34 tokens/s for Quest at 3×1M contexts) do not include any temporal caching — it's unclear how much of ShadowKV's 4.85× advantage (45.32 vs. 9.34) comes from the cache vs. from the asymmetric storage vs. from the selection quality.

What evidence exists in the paper. Figure 5 (right) shows the hit rate for three specific layer-head combinations (Layer-5 KV Head-0, Layer-15 KV Head-3, Layer-25 KV Head-7) of Llama-3-8B-1M over 250 decoding steps, with hit rates ranging from about 60–80%. Table 13 provides the latency breakdown with "Recompute K (Overlapped)" and "Fetch V" times, but does not separate the cache hit vs. miss contributions. The equivalent bandwidth model (Section 4.2) includes the $(1 - \alpha)$ term for cache misses, indicating that the model accounts for the cache theoretically, but this model is not validated against measurements with the cache disabled.

Mitigation status. Not addressed. The cache mechanism is described as an implementation detail (Algorithm 2: "Based on the insight that the KV cache has temporal locality, we conduct an index scan to detect the missed chunks and only rebuild the necessary KV pairs on-the-fly") rather than as a separately evaluated component. The paper does not provide an ablation with the cache disabled, nor does it characterize the hit rate across different tasks, context lengths, or model architectures. A practitioner would need to measure the cache hit rate for their specific workload to determine whether the temporal caching provides significant benefit.

Limitation 6: Single Precision Regime and Lack of Quantization Interaction Analysis

The assumption or constraint. All accuracy and efficiency experiments use BF16 precision for both model weights and KV cache, with a supplementary FP8 experiment (Appendix A.4, Tables 9 and 10) to test precision sensitivity of the SVD and reconstruction. However, the paper does not test ShadowKV in combination with KV cache quantization — an increasingly common deployment optimization where keys and values are stored at 4-bit or even 2-bit precision (as in KIVI, KVQuant, and related work).

The consequence. The low-rank key cache compression and value offloading in ShadowKV are not independent of quantization — they interact in non-obvious ways. Specifically:

Low-rank key cache + key quantization: If the key cache is already quantized to 4 bits, the additional memory savings from low-rank compression are reduced (the baseline is smaller). The $B$ matrix in the SVD decomposition ( $r \times d$ per head) is stored at full precision and is not compressed — at rank 160 and head dimension 128, this requires $160 \times 128 \times 2 = 40$ KB per head in BF16. For a model with 32 layers and 8 KV heads, this totals 10.2 MB — negligible relative to the original KV cache, but potentially comparable to the size of a heavily quantized KV cache. The relative benefit of low-rank compression shrinks as the baseline gets smaller through quantization.
Reconstruction error amplification: The SVD reconstruction approximates the full key cache as $\tilde{K} = A \times B$ . If the model's keys were originally quantized, the SVD is applied to quantized values, and the reconstruction error compounds with the quantization error. The paper's FP8 experiment (Tables 9, 10) shows that ShadowKV remains accurate at FP8 — but 8-bit floating point has a much larger dynamic range and precision than 4-bit integer quantization. The interaction with aggressive quantization is unknown and could cause accuracy degradation beyond what either method causes independently.
Value offloading + value quantization: If values are quantized to 4 bits, offloading them to the CPU saves 75% less memory and transfers 75% less data over PCIe — reducing both the memory advantage and the overlapping benefit. The relative advantage of ShadowKV over on-GPU quantized attention would narrow.

What evidence exists in the paper. The FP8 experiments in Appendix A.4 (Tables 9, 10) demonstrate that ShadowKV's SVD and reconstruction are robust to moderate precision reduction. On RULER with Llama-3-8B-1M, ShadowKV FP8 achieves 85.95 vs. 84.94 for full attention FP8 (ShadowKV actually improves slightly). On LongBench, ShadowKV FP8 achieves 39.25 vs. 39.49. This suggests the SVD is not catastrophically sensitive to reduced precision, but FP8 is still quite high precision compared to 4-bit integer quantization.

Mitigation status. The paper states that "quantization methods reduce the KV cache bit width, which is orthogonal to our approach" (Section 2, Quantization paragraph). This correctly identifies that ShadowKV's compression (dimensionality reduction) and quantization (bit-width reduction) operate on different axes and could be combined. However, calling them "orthogonal" implies no interaction, which is an untested assumption. The paper does not evaluate any quantized configuration, leaving it to the reader to determine whether the combination is synergistic or redundant.

7. Implications and Future Directions

How This Work Changes the Landscape

ShadowKV introduces a conceptual reframing of KV cache management that departs from the dominant assumption in prior work: that keys and values should be compressed, evicted, or offloaded symmetrically. The paper's central empirical finding—that pre-RoPE keys are exceptionally low-rank while values are not (Figure 1, left; Figure 5, left)—disentangles two resources that the field has traditionally treated as a monolithic unit. This is not merely an optimization trick; it establishes a design principle for asymmetric KV cache management: keys should be stored differently from values because they serve fundamentally different functional roles in attention and exhibit fundamentally different mathematical structure. Keys participate in dot-product scoring (directional matching matters, dimensionality can be reduced), while values are passively aggregated (directional structure is irrelevant, compression must preserve per-token fidelity). The practical consequence is a storage architecture where compressed keys remain on GPU (enabling fast approximate attention scoring and on-the-fly reconstruction) while values are offloaded to CPU (where they are only fetched for the small fraction of tokens actually selected).

This reframing resolves a tension that has persisted across three families of prior work. KV eviction methods (StreamingLLM, H2O, SnapKV) succeeded at reducing GPU memory but permanently discarded tokens, causing accuracy collapse on multi-turn conversations (Figure 7) and any task requiring flexible access to distributed context. Dynamic sparse attention methods (Quest, Loki) preserved accuracy but failed to reduce memory footprint, leaving batch size—and therefore throughput—unchanged. CPU offloading with sparse attention (InfiniGen) reduced GPU memory but introduced PCIe bottlenecks that dominated decoding latency, and used offline SVD projections that failed to adapt to sequence-specific key subspace structure (as ShadowKV's Figure 1, middle, shows that low-rank subspaces vary across sequences). ShadowKV's asymmetric design essentially says: keys and values face different bottlenecks, so treat them differently. Keys need to be accessed for selection, so keep them on GPU in compressed form. Values only need retrieval after selection, so offload them and overlap the fetch with key reconstruction. This breaks the trilemma that constrained prior work, simultaneously achieving memory reduction, low latency, and maintained accuracy.

The paper's second landscape-shifting contribution is the demonstration that chunk-level sparsity with outlier preservation is not a heuristic compromise—it is grounded in the empirical structure of post-RoPE key representations. The within-chunk cosine similarity measurement (Figure 5, middle) quantifies why chunk-level approximation works (most chunks are internally homogeneous) and where it fails (the 0.2–0.3% of outlier chunks). This transforms chunk size selection from a hyperparameter-tuning exercise into a diagnostic procedure: measure the minimum within-chunk cosine similarity distribution for a given model, identify the outlier tail, and set chunk size and outlier budget accordingly. Prior work like Quest treated chunking as an implementation detail (page size as a memory management parameter); ShadowKV treats it as a property of the model's learned representations that can be empirically characterized. The finding that just 8–48 outlier chunks (0.049%–0.293% of all chunks, Table 15) recover full accuracy—with the first chunk as the dominant attention sink—provides a mechanistic explanation for why sparse attention can work at such extreme sparsity levels (1.56%): the key cache is overwhelmingly redundant at the chunk level, and the few non-redundant regions are concentrated, identifiable, and preservable.

The paper also shifts the conversation around throughput optimization from a memory-capacity-centric view to a data-movement-centric view. The theoretical equivalent bandwidth model (Section 4.2) formalizes why ShadowKV can exceed the throughput of ideal full attention with infinite GPU memory: full attention must read all $2S$ KV vectors at GPU memory bandwidth regardless of memory capacity, while ShadowKV's combination of compression, sparsity, and temporal caching moves fewer total bytes. The 7.2 TB/s equivalent bandwidth—3.6× higher than the A100's 2 TB/s physical bandwidth—is not a claim about exceeding hardware limits but rather a demonstration that the effective data movement per unit of useful computation can be far lower than what full attention requires. This reframes the throughput problem: the goal is not just to fit more sequences in GPU memory, but to minimize the bytes moved per token generated, regardless of where those bytes reside. The practical implication is that memory capacity expansion (more HBM, larger GPUs) may be a less cost-effective path to higher throughput than rearchitecting how the KV cache is stored and accessed—a finding with direct economic implications for LLM serving infrastructure.

Finally, the paper reconciles the apparent contradiction between the success of sparse attention on retrieval tasks and its failure on tasks requiring flexible context access. The multi-turn NIAH experiment (Figure 7) shows that SnapKV and StreamingLLM—which make irreversible eviction decisions based on first-turn attention patterns—catastrophically fail on subsequent turns that require different information. ShadowKV succeeds because it never discards information; it only selectively accesses it. This establishes a clear taxonomy: eviction = permanent information loss, sparsity = temporary access reduction. For any deployment where future queries might need different parts of the context than current queries (multi-turn conversations, interactive document exploration, agent-based workflows), eviction-based methods carry an irreducible risk of information loss that sparsity-based methods avoid.

Research directions that become more attractive: (1) characterizing and exploiting structure in transformer representations to guide system design (rather than treating the KV cache as an opaque memory blob); (2) developing diagnostic tools (like the within-chunk cosine similarity metric) that measure representation structure to automatically tune system parameters; (3) exploring asymmetric storage architectures where different parts of the KV cache are stored at different locations and accessed with different strategies based on their functional role and structural properties.

Research directions that become less attractive: (1) purely heuristic eviction strategies that make irreversible discard decisions without characterizing the structure of what is discarded; (2) symmetric compression or offloading approaches that treat keys and values identically; (3) offline, fixed projections for KV selection (as in InfiniGen) that cannot adapt to sequence-specific structure.

Follow-Up Research This Work Enables

Characterizing low-rank key cache structure across architectures, tasks, and layers. ShadowKV demonstrates that pre-RoPE keys in Llama-3.1-8B are highly low-rank (rank 160 from dimension 128 achieves full accuracy, Figure 5 left), but provides no cross-architecture characterization. A systematic study would measure the singular value decay of pre-RoPE keys across diverse model families (Llama, Mistral, Gemma, Qwen, DeepSeek), positional encoding schemes (RoPE with different base frequencies, ALiBi, NoPE), and model scales (1B to 70B+). The key question: is the low-rank property universal, or does it depend on specific architectural choices like RoPE base frequency, GQA ratio, or training data? Within a single model, is the optimal rank uniform across layers, or do early layers (which attend to local syntax) require different compression rates than late layers (which attend to long-range semantics)? The paper's Figure 9 (right) sweeps rank across RULER subtasks but doesn't break out results by layer. A layer-wise analysis could reveal that most of the rank budget is needed for a minority of layers, enabling variable-rank compression with higher overall memory savings. A strong follow-up would measure the per-layer reconstruction error (Frobenius norm between original and rank- $r$ reconstructed keys) and correlate it with downstream task accuracy, producing layer-specific rank recommendations.

Dense global aggregation tasks as stress tests for the sparse attention budget. The paper's benchmarks (RULER, NIAH, LongBench QA) are predominantly retrieval and localized-information tasks where answers depend on finding specific passages. The sparse budget of 1.56% (2048 tokens out of 128K) is sufficient for these tasks because the relevant information is concentrated. A critical stress test would evaluate ShadowKV on tasks requiring dense global synthesis: summarizing 128K legal opinions where key provisions span the entire document, answering multi-hop questions where each hop depends on a different section, or detecting contradictions between statements separated by tens of thousands of tokens. The hypothesis to test: there exists a critical "information dispersal threshold" beyond which the 1.56% budget is fundamentally insufficient, and accuracy degrades proportionally to how widely the relevant information is distributed. A strong experiment would construct a synthetic benchmark with $N$ needles distributed across the context, vary $N$ from 1 to 1000, and measure how ShadowKV's accuracy scales with the sparse budget. This would produce a practical guideline: "for tasks with at most $X$ independent pieces of relevant information, a $Y$ % sparse budget suffices." The NarratQA result in Table 1—where ShadowKV scores 17.17 vs. Quest's 20.13 on Llama-3-8B-1M—hints that broader-context tasks may be more challenging for chunk-level selection, but the paper doesn't investigate this.

Combining ShadowKV's asymmetric storage with KV cache quantization. The paper calls quantization "orthogonal to our approach" (Section 2), and the FP8 experiments (Tables 9, 10) show that the SVD and reconstruction are robust to moderate precision reduction. However, the interaction between low-rank compression and aggressive quantization (4-bit or 2-bit, as in KIVI or KVQuant) is unexplored and potentially non-trivial. If the key cache is stored at 4 bits, the baseline memory is already 4× smaller than BF16, so the relative savings from rank-160 compression shrink. If the value cache is quantized, the PCIe transfer volume during decoding is similarly reduced. A combined system might use different quantization strategies for different components: the $B$ matrix (right singular vectors, $r \times d$ per head) might need higher precision because it is multiplied many times during reconstruction, while the $A$ matrix (token coordinates, $s \times r$ ) might tolerate lower precision because each row is used only when its chunk is selected. A strong experiment would evaluate the Pareto frontier of accuracy vs. total GPU memory (keys + values + SVD matrices) under joint optimization of SVD rank and quantization bit-width, for both keys and values. The result would be a deployment guide: "for a 128K sequence on Llama-3-8B, the memory-optimal configuration is rank- $r$ with $b_k$ -bit keys and $b_v$ -bit values."

Dynamic sparse budget allocation conditioned on query difficulty or attention entropy. ShadowKV uses a fixed sparse budget (256 chunks, 1.56%) for all decoding steps regardless of the query or the attention pattern. This is analogous to early best-of-N sampling in the test-time compute literature—uniform allocation regardless of difficulty. Future work could make the sparse budget dynamic: for decoding steps where the attention distribution is highly concentrated (low entropy, as measured by the landmark-based approximate attention scores), use a smaller budget; for steps where attention is diffuse (high entropy, as when the model needs to integrate information from many parts of the context), use a larger budget. The mechanism would be simple: compute the entropy of the approximate attention distribution $S_2$ (Algorithm 2), threshold it, and adjust $k$ accordingly within a total budget constraint averaged across the decoding sequence. A strong experiment would measure the variance of attention entropy across decoding steps for different tasks (needle retrieval likely has high variance: concentrated when retrieving, diffuse when generating reasoning) and evaluate whether dynamic allocation maintains accuracy with a lower average sparse budget than the fixed 256-chunk configuration. This connects ShadowKV to the compute-optimal test-time scaling paradigm—allocating sparse attention computation where it matters most rather than uniformly.

ShadowKV for encoder-decoder architectures and cross-attention. The paper exclusively evaluates decoder-only (GPT-style) models with self-attention. Encoder-decoder models (T5, UL2, instruction-tuned variants) have two attention mechanisms: self-attention in the encoder and decoder, and cross-attention where decoder queries attend to encoder outputs. Cross-attention has a fundamentally different structure: the key-value cache is fixed (encoder outputs for the entire input sequence) and the queries change (each decoder step), but the sequence length is typically shorter (input documents rather than entire conversation histories). ShadowKV's low-rank key compression and landmark-based selection might apply differently to cross-attention because (a) the pre-RoPE structure may differ between encoder and decoder representations, (b) cross-attention patterns are typically more sparse (each decoder token attends to a small subset of encoder tokens), and (c) the memory pressure from encoder KV caches may be less acute than from decoder KV caches for long-output generation. A strong experiment would measure the low-rank properties of encoder keys in a model like FLAN-T5 on long-document QA tasks, implement ShadowKV-style compression for the cross-attention KV cache, and evaluate whether the compression ratios and sparse budgets that work for self-attention transfer to cross-attention.

Prefix-aware SVD caching for shared-context workloads. The paper mentions in a footnote (Section 1) that the SVD "can be offloaded to the CPU to perform SVD asynchronously or precomputed and stored as part of the prefix cache"—but never evaluates this. In production deployments where many queries share a long prefix (document QA: multiple questions against the same document; chatbot: multiple turns with the same conversation history; code completion: multiple queries against the same codebase), recomputing the SVD for each request would be wasteful. A prefix-aware system would compute the SVD once for the shared prefix, store the $A$ and $B$ matrices in a prefix cache, and reuse them for all requests sharing that prefix. This is technically non-trivial because the low-rank subspace captured by the SVD is specific to the prefix—appending new tokens (the user's question) changes the key cache and potentially the optimal subspace. A strong experiment would measure how well the prefix-derived SVD matrices compress the key cache for the prefix+continuation, as a function of continuation length, using the subspace similarity metric $D(H_1, H_2)$ introduced in Section 3.1 (Figure 1, middle). If the similarity remains high for continuations of typical length (hundreds to low thousands of tokens), prefix SVD caching becomes a practical optimization that eliminates the per-request SVD overhead entirely.

Practical Applications and Downstream Use Cases

High-throughput batch evaluation on long-context benchmarks and datasets. Organizations evaluating LLMs on benchmarks like RULER, LongBench, or InfiniteBench—or generating synthetic training data from long documents—face a common bottleneck: running thousands of examples through a model with 128K+ context windows is prohibitively slow due to small batch sizes. ShadowKV's 6× batch size scaling and 2.2–3.0× throughput improvements (Table 3) directly accelerate these workloads. For a benchmark like RULER with 500 test examples at 128K context, full attention on an A100 processes approximately 500 × (128K input + 32 output tokens) ÷ 160 tokens/s ≈ 400K seconds ≈ 111 hours for Llama-3-8B-1M. ShadowKV at the reported 455 tokens/s reduces this to approximately 39 hours—a 2.83× reduction in wall-clock time. For organizations running daily or weekly evaluations across multiple model checkpoints, this translates to faster iteration cycles and lower compute costs. The key deployment consideration: ShadowKV requires no fine-tuning, no calibration dataset, and no task-specific adaptation—it drops into existing evaluation pipelines with the same model weights and prompts.

On-device or edge deployment of long-context LLMs with limited VRAM. Consumer GPUs (RTX 4090 with 24GB, A6000 with 48GB) and edge devices face even more acute memory constraints than datacenter GPUs. Full attention on a 24GB GPU might not fit even a single 128K sequence for an 8B model (the KV cache alone for Llama-3-8B at 128K with 32 layers, 8 KV heads, and 128 dimensions requires approximately 32 × 2 × 8 × 128K × 128 × 2 bytes ≈ 16.8 GB in BF16, leaving insufficient room for model weights and activations). ShadowKV's 7.08× memory reduction (Appendix A.2) reduces this to approximately 2.4 GB, making long-context inference feasible on hardware that would otherwise be out of memory. The paper does not benchmark on consumer GPUs, but the memory savings formula is hardware-independent. A deployment scenario: running Llama-3.1-8B with 128K context on an RTX 4090 for document QA or code analysis, workloads that currently require cloud GPU instances due to memory constraints. The practical benefit is not just throughput but feasibility—enabling workload classes that were previously impossible on consumer hardware.

Multi-turn conversational agents and retrieval-augmented generation (RAG) with persistent context. In RAG systems where a long document (e.g., a legal contract, a research paper, a codebase) is loaded as context and the user asks multiple follow-up questions, the KV cache for the document must persist across turns. Full attention requires holding the entire document's KV cache on GPU for the duration of the session, limiting the number of concurrent sessions per GPU. Eviction-based methods like SnapKV discard tokens based on the first query's attention patterns, causing accuracy collapse on subsequent queries that need different parts of the document (as demonstrated in Figure 7). ShadowKV's combination of compressed persistent storage (low-rank key cache on GPU, values on CPU) and query-dependent sparse retrieval solves both constraints: memory per session is reduced by 7.08×, and each query selects the chunks relevant to that query regardless of what previous queries accessed. A concrete deployment: a customer support system where each support ticket includes a 50K-token conversation history and the agent asks clarifying questions, checks policy documents, and drafts responses across 5–10 turns. ShadowKV would allow serving 6× more concurrent support tickets per GPU (Table 4, 60K context: 48 vs. 8 batch size) while maintaining accuracy across all turns.

LLM-powered code analysis and repository-level understanding. Tools that analyze entire codebases (e.g., GitHub Copilot's workspace awareness, code review systems, automated refactoring tools) must process tens or hundreds of files simultaneously, with context windows reaching 128K–1M tokens. These workloads combine long contexts with structured content (code has strong locality: functions, classes, and modules form natural chunks) and benefit from the high-throughput batch processing that ShadowKV enables. The chunk-level locality exploited by ShadowKV's landmarks (Figure 5, middle) may be even stronger for code than for natural language, because code representations are more structured—functions and class definitions form coherent units with internal consistency. A concrete deployment scenario: a code review system processing 100 pull requests simultaneously, each with 64K tokens of diff + surrounding context, running on an 8-GPU node. ShadowKV's 6× batch size scaling would allow processing 600 concurrent reviews instead of 100, reducing queue time from hours to minutes. An interesting open question: does the code domain require different chunk sizes or outlier budgets than natural language, given the different attention patterns (syntax at short range, cross-references at long range)?

When to Prefer This Method

The paper positions ShadowKV against three families of alternatives: full attention (no memory reduction, no latency overhead, perfect accuracy), CPU offloading with sparse attention (memory reduction, high latency, accuracy varies by selection quality), and KV eviction (memory reduction, low latency, irreversible accuracy loss). The decision boundaries among these, as established by the paper's experiments, follow clear patterns:

Prefer ShadowKV when:

Context lengths exceed ~32K tokens and the SVD overhead during pre-filling amortizes to under ~5% of total compute (Figure 1 right; Table 12 shows SVD at 6.65% for 64K and declining). Below 32K, the SVD overhead may dominate if decoding is short; above 128K, it asymptotically approaches zero.
Throughput, not per-request latency, is the primary metric. ShadowKV's overlapping and sparse attention reduce total FLOPs but add per-step operations (landmark scoring, index scan, cache hit/miss logic) that may increase latency for batch size 1 compared to full attention with FlashAttention. The paper's measured latencies (Table 13) are for batch sizes 6–48; latency at batch size 1 is not reported.
The workload involves multi-turn interactions or any scenario where future queries need access to different parts of the context than past queries (Figure 7: SnapKV and StreamingLLM collapse after turn 1; ShadowKV tracks full attention).
GPU memory is the binding constraint on batch size (as in Tables 3 and 4, where full attention is OOM at batch sizes ShadowKV supports). If the workload already fits comfortably in GPU memory, the memory savings provide no benefit, and the per-step overhead of sparse selection may reduce throughput.
Tasks have moderate locality—answers depend on finding or synthesizing information from a subset of the context, not from uniformly distributed content across the entire sequence. The paper's benchmarks are predominantly localized (retrieval, QA, multi-hop); the 1.56% sparse budget may be insufficient for global aggregation tasks (document summarization, contradiction detection) where every part of the context contributes.

Prefer full attention (or FlashAttention with no sparsity) when:

Context lengths are short (~<16K) and the SVD overhead represents a significant fraction of total compute, while the memory savings provide minimal benefit because the KV cache is already small relative to model weights.
Latency at batch size 1 is critical (interactive chat with a single user) and the per-step overhead of landmark scoring, index scanning, and sparse reconstruction outweighs any throughput gains.
The task requires uniformly dense attention to the entire context—summarizing a long document where every sentence matters, or computing aggregate properties across the whole sequence. ShadowKV's 1.56% sparse budget is fundamentally insufficient for such tasks.
Verification of exact equality to full attention is required (e.g., safety-critical applications where any deviation, even within the 0.2–2 point accuracy range observed on RULER, is unacceptable). ShadowKV matches full attention within statistical noise on most benchmarks but does not guarantee bit-identical outputs.

Prefer KV eviction methods (SnapKV, StreamingLLM) when:

The workload consists of single-turn, one-shot queries where the attention pattern for the query is known to align with the eviction heuristic (e.g., recency-biased tasks where StreamingLLM's attention sink + recent tokens policy is sufficient). In these limited settings, eviction methods achieve memory reduction with lower computational overhead than ShadowKV's SVD and reconstruction.
The model is deployed in severe memory-constrained environments where even ShadowKV's compressed GPU footprint is insufficient—eviction can achieve higher compression ratios by permanently discarding more tokens, accepting the accuracy penalty.

Prefer CPU offloading with dynamic sparse attention (Quest + offloading) when:

The hardware has unusually high PCIe bandwidth (e.g., H100 with PCIe 5.0 at 63 GB/s vs. A100's 31.5 GB/s) and low GPU memory bandwidth—this narrows the gap between ShadowKV's asymmetric approach and full-KV-fetching approaches. However, even at 2× PCIe bandwidth, ShadowKV's 4.85× throughput advantage over Quest at 3×1M contexts (Table 14) suggests the asymmetric design dominates in practice.
Simplicity of implementation is prioritized—Quest with CPU offloading is conceptually simpler than ShadowKV (no SVD, no low-rank reconstruction, no outlier detection), and may be preferred in settings where engineering complexity is costly.

The paper does not explicitly frame these tradeoffs as a decision matrix, but the experimental evidence supports these boundaries. A practitioner deploying long-context LLM inference should measure their specific workload's context length distribution, task locality, batch size requirements, and latency constraints against the profiles above to determine whether ShadowKV's asymmetric storage and sparse access design provides net benefit over the alternatives.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

READ MORE2024· serving← in-domain predecessor
- Orca: A Distributed Serving System for Transformer-Based Generative Models2024· serving← in-domain predecessor
Inference Scaling for Long-Context Retrieval Augmented Generation2024· reasoning← topic thread
- Writing in the Margins: Better Inference Pattern for Long Context Retrieval2024· inference-optimization← LLM-judged similar
- Not All LLM Reasoners Are Created Equal2024· reasoning← in-domain predecessor
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads2024· inference-optimization← topic thread
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression2024· architecture← topic thread
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States2024· architecture← in-domain predecessor
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management2024· inference-optimization← topic thread
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer2025· architecture← similar topic
- Read More2023· architecture← similar topic
Hydragen: High-Throughput LLM Inference with Shared Prefixes2024· serving← similar topic
- FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design2024· serving← in-domain predecessor
- DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference2026· serving← similar topic
- Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference2025· serving← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

The Pitfalls of KV Cache Compression2025· inference-optimization→ LLM-judged similar
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction2026· inference-optimization→ LLM-judged similar
- Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models2025· inference-optimization→ in-domain predecessor
- Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost2025· llm-systems→ topic thread
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression2024· inference-optimization→ similar topic
- Inference-Time Hyper-Scaling with KV Cache Compression2025· inference-optimization→ similar topic
MARCONI: PREFIX CACHING FOR THE ERA OF HYBRID LLMS2024· serving→ in-domain predecessor
- Efficiently Serving LLM Reasoning Programs with Certaindex2024· serving→ in-domain predecessor
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training2024· architecture→ topic thread
- Efficient Long-context Language Model Training by Core Attention Disaggregation2025· training-methods→ LLM-judged similar
- Hymba: A Hybrid-head Architecture for Small Language Models2024· architecture→ in-domain predecessor
Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding2025· inference-optimization→ topic thread
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference2025· code→ LLM-judged similar
- Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation2025· inference-optimization→ LLM-judged similar
- KLASS: KL-Guided Fast Inference in Masked Diffusion Models2025· inference-optimization→ LLM-judged similar
- Fast and Accurate Causal Parallel Decoding using Jacobi Forcing2025· inference-optimization→ LLM-judged similar
- Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed2025· training-methods→ LLM-judged similar
- DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencoder2025· vision→ LLM-judged similar
- Efficient Autoregressive Video Diffusion with Dummy Head2026· inference-optimization→ LLM-judged similar
- DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing2026· multimodal→ LLM-judged similar
- Rectified Sparse Attention2025· inference-optimization→ in-domain predecessor
- Fast Inference from Transformers via Speculative Decoding2022· inference-optimization→ similar topic
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads2024· inference-optimization→ similar topic
- Break the Sequential Dependency of LLM Inference Using LOOKAHEAD DECODING2024· inference-optimization→ similar topic
DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale2022· serving→ similar topic
- EFFICIENTLY SCALING TRANSFORMER INFERENCE2022· serving→ in-domain predecessor
- Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs2023· serving→ similar topic
Hydragen: High-Throughput LLM Inference with Shared Prefixes2024· serving→ similar topic
- ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition2024· serving→ LLM-judged similar
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve2024· serving→ similar topic

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Long-Context LLMs Are Memory-Bound, Not Compute-Bound

The Prior Approach Landscape: Three Families, Three Failures

Family 1: KV Eviction Strategies

Family 2: Dynamic Sparse Attention

Family 3: CPU Offloading with Sparse Attention

The Gap: No Method Simultaneously Achieves All Three Goals

How This Paper Positions Itself: Leveraging Low-Rank Structure and Spatial Locality

The Practical Framing

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Low-Rank Key Cache Compression via Online SVD

Landmark Construction and Outlier Detection

Decoding-Time KV Selection via Landmark-Guided Approximate Attention

Sparse KV Reconstruction and Overlapping Strategy

Theoretical Equivalent Bandwidth Model

Summary of Key Design Decisions and Their Justifications

4. Key Insights and Innovations

Innovation 1: Redrawing the Storage-Access Boundary — Keys Compressed, Values Offloaded

Innovation 2: The Difficulty Is Not Missed Tokens, It's Missed Chunks — Chunk-Level Locality as a Diagnostic Tool

Innovation 3: Verifier-Style Selection Without a Verifier — Landmarks as Compressed Proxies

Innovation 4: Test-Time Compute Allocation Through Sparse Budget Design — Not More Compute, Smarter Access

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Accuracy Preservation: Matching Full Attention Across Models and Benchmarks

Throughput Scaling: Enabling 6× Larger Batches and Up to 3.04× Higher Token Rates

Comparative Advantage Over Baselines at Equivalent Sparse Budgets

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: ShadowKV reduces GPU memory footprint by over 6× without accuracy degradation.

Claim 2: ShadowKV supports up to 6× larger batch sizes and boosts throughput by up to 3.04×.

Claim 3: ShadowKV outperforms all baselines while maintaining accuracy at equivalent sparse budgets.

Claim 4: ShadowKV even surpasses the performance achievable with infinite batch size under the assumption of infinite GPU memory.

Missing Experiments That Would Strengthen the Paper

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted For — The SVD Overhead During Pre-Filling

Limitation 2: Benchmark Coverage Is Skewed Toward Retrieval and Localized-Information Tasks

Limitation 3: The "Infinite Batch Size" Comparison Is a Theoretically Motivated but Practically Incomplete Baseline

Limitation 4: All Experiments Use a Single GPU Architecture (A100); the Results Do Not Generalize to Other Hardware Configurations

Limitation 5: The Temporal Cache Mechanism's Benefit Is Reported but Not Isolated — It Is Unclear How Much of the Gain Comes from Caching vs. Sparsity vs. Overlapping

Limitation 6: Single Precision Regime and Lack of Quantization Interaction Analysis

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

READ MORE

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval