LLM Systems2025~108 minarXiv:2511.18643

Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost

KV cache / paged attention FP8 / low-precision

Haojun Xia, Xiaoxia Wu, Jisen Li +12Together AINov 23, 2025cs.LG

🎯 Pitch

Uniform 2-bit KV cache quantization destroys reasoning accuracy, but Kitty reveals that protecting just 12.5% of Key-cache channels at 4-bit is enough to eliminate the gap—achieving FP16-level scores on AIME while approaching 8× memory savings. The system decomposes these mixed-precision pages into dual 2-bit tensors to avoid scattered reads, delivering 2.1–4.1× higher throughput than FP16 under the same memory budget.

1. Executive Summary

Kitty introduces an algorithm–system co-design for mixed-precision KV cache quantization that closes the accuracy gap between 4-bit and 2-bit Key-Value caching for long-context LLM inference. Evaluated on reasoning benchmarks (GSM8K, MATH-Algebra, HumanEval, GPQA-Diamond, AIME) across Qwen3 (8B–32B) and LLaMA3 (8B–70B) model families, the paper proposes Dynamic Channel-wise Precision Boost — a method that ranks Key-cache channels by quantization sensitivity (measured via attention-score MSE) and preserves only the most critical 12.5–25% of channels at INT4 precision while aggressively quantizing the remainder to INT2 — achieving near-zero accuracy loss relative to FP16 while approaching 2-bit memory. The core system challenge is maintaining coalesced page layouts and uniform dequantization despite per-channel mixed precision; Kitty solves this by decomposing each mixed-precision Key page into two dense 2-bit tensors (a full dense matrix and a structured sparse matrix for the high-order bits), yielding a page-centric memory layout with Triton-compatible kernels that enable up to 8× larger batch sizes and 2.1×–4.1× higher throughput under the same memory budget compared to FP16 baselines, establishing that aggressive 2-bit KV cache quantization is practical only when critical channels are selectively protected from the noise that uniform low-bit quantization introduces.

2. Context and Motivation

The Core Problem: 2-Bit KV Cache Quantization Destroys Reasoning Accuracy

The fundamental problem this paper tackles is deceptively simple: why does quantizing the Key-Value cache to 2 bits destroy model accuracy on reasoning tasks, and how can we fix it without giving up the memory savings? This matters because the KV cache has become the dominant memory bottleneck in LLM inference—its size grows linearly with both sequence length and batch size, and for long-context serving scenarios, it can exceed model weights by nearly an order of magnitude.

The paper opens with a concrete motivating example: serving 32 requests of LLaMA3-70B with 128K context requires more than 1.2 TB of KV cache storage, while state-of-the-art datacenter GPUs like the NVIDIA B200 provide only 192 GB at a cost exceeding $30,000 each. This is not a marginal inefficiency—it's a deployment blocker. Without compression, long-context inference at scale is economically infeasible.

KV cache quantization is the natural remedy. Unlike weight quantization (which is applied once post-training), KV cache quantization must happen dynamically during inference as new tokens are generated, making it a more challenging problem. But the payoff is substantial: if you can safely reduce KV cache precision from FP16 (16 bits per element) to 2 bits, you cut memory consumption by roughly 8×, directly translating to larger batch sizes, higher throughput, and lower hardware requirements.

The paper's core empirical finding is summarized in Table 1 and Section 2.2: when applying the state-of-the-art KIVI quantization algorithm (Liu et al., 2024) across multiple benchmarks:

4-bit KIVI (KIVI-K4V4) largely preserves accuracy. For Qwen3-8B, the drop from FP16 to 4-bit KIVI is negligible across GSM8K, MATH, GPQA, and HumanEval. This is consistent with prior work showing that 4-bit KV quantization is a solved problem.
2-bit KIVI (KIVI-K2V2) catastrophically degrades accuracy. On Qwen3-8B, MATH-Algebra drops from 88.26 to 47.29 (a 41-point collapse), GSM8K drops from 94.79 to 89.13, and AIME24 drops from 71.67 to 57.00. The average accuracy drop across tasks is -15.76 for Qwen3-8B and -10.15 for LLaMA3-8B.

This is the central gap the paper addresses: 4-bit works, 2-bit breaks, and we need 2-bit for the memory savings to be truly transformative. The question is why 2-bit fails so badly and whether there's a more nuanced approach that can recover 2-bit's memory efficiency without the accuracy penalty.

Why This Problem Matters: The Memory Wall in Long-Context Inference

The significance of this gap extends beyond academic curiosity into practical deployment economics. As LLMs are increasingly deployed for tasks requiring extended reasoning chains (chain-of-thought), document understanding, and multi-turn dialogues, context lengths are growing rapidly—models like GPT-4 and LLaMA3 now support up to 128K tokens. The KV cache, which stores the key and value activations for every previous token to avoid recomputation during autoregressive generation, scales as:

$\text{KV cache size} \propto 2 \times B \times H \times L \times D$

where $B$ is batch size, $H$ is the number of KV heads, $L$ is sequence length, and $D$ is the head dimension. For a model like LLaMA3-70B with $H = 8$ , $D = 128$ , serving 32 requests at 128K context, this is:

$2 \times 32 \times 8 \times 128\text{K} \times 128 = 2 \times 32 \times 8 \times 131072 \times 128 \approx 8.59 \text{ billion elements}$

At FP16 (2 bytes each), this is ~17 GB per tensor type (keys and values), totaling ~34 GB... wait, the paper claims 1.2 TB. Let me use the paper's calculation: for LLaMA3-70B with more KV heads (the model uses Grouped Query Attention, but the full KV head count for attention is larger), the paper states that 32 requests at 128K require over 1.2 TB. This dwarfs the model weights themselves, which for a 70B parameter model at FP16 are ~140 GB. The KV cache is therefore the dominant memory consumer, and any practical deployment at scale must compress it.

Beyond pure memory, the KV cache exacerbates inference latency through data movement. Each decoding step must read all previous keys and values from GPU High-Bandwidth Memory (HBM) into compute units—the attention mechanism is fundamentally memory-bound, with its FLOPs-to-byte ratio determined by how efficiently the KV cache can be streamed. Reducing the KV cache from 16 bits to 2 bits per element reduces both the memory footprint and the data movement volume, potentially accelerating inference even when memory headroom is not the binding constraint.

Prior Approaches and Their Shortcomings

The paper situates itself relative to several existing strategies for KV cache compression, each with limitations that motivate Kitty's design.

Per-Token vs. Per-Channel Quantization. The KV cache for a single head is an $(L, D)$ matrix— $L$ tokens, each represented by a $D$ -dimensional vector. Quantization can be applied along either axis:

Per-token quantization: a separate scale (and optionally zero-point) for each token's key or value vector. This captures token-level variation but can miss channel-level structure.
Per-channel quantization: a scale shared across all tokens for each channel dimension. Prior work (KIVI, KVQuant, and others) has observed that per-channel quantization preserves accuracy better for the Key cache, while per-token works well for the Value cache. Kitty inherits this finding: Key cache uses per-channel quantization, Value cache uses per-token (with a sliding window of recent tokens kept in full precision).

However, neither per-token nor per-channel quantization alone addresses the fundamental limitation: at 2 bits, the representational capacity is simply too low for uniform treatment of all elements. Some structure (certain tokens, certain channels) matters more than others, and uniform 2-bit quantization squanders precision equally on critical and non-critical elements alike.

Mixed-Precision and Hybrid Schemes. Prior work has recognized that uniform low-bit quantization is insufficient and has proposed various forms of mixed precision:

KVQuant (Hooper et al., 2024) identifies "outliers" in the KV cache and stores them in FP16 using a sparse representation while quantizing the rest at low precision. The paper acknowledges this as conceptually related but notes a critical practical flaw: "KVQuant is not hardware-friendly and usually suffers from low system-level efficiency, since it introduces additional runtime overhead from sparse-dense multiplications, which could be slow on GPUs." The sparse-dense format breaks the coalesced memory access patterns that GPUs rely on for bandwidth efficiency. This is an important constraint that shapes Kitty's system design: any mixed-precision scheme must maintain coalesced page layouts and uniform dequantization to be practical—you can't scatter higher-precision elements throughout the cache and expect efficient retrieval.

KIVI (Liu et al., 2024) and BitDecoding (Du et al., 2025) retain the most recent tokens in full precision to preserve an accurate local context—a form of temporal mixed precision. While this helps (the paper's own "KIVI-K2V2*" variant with 32 sink tokens preserved in FP16 improves average accuracy by +8.28 on Qwen3-8B), it still leaves a significant gap to FP16. The paper's Table 2 shows that KIVI-K2V2* (preserving initial and recent tokens in FP16) still underperforms the FP16 baseline by -7.35 on average for Qwen3-8B, with MATH-Algebra at 74.92 vs. 88.26 FP16. The gap is especially pronounced on reasoning-intensive tasks—the very tasks where long contexts matter most.

MiniKV (Sharma et al., 2024) proposes layer-discriminative bit allocation: different layers get different precisions. KVTuner (Li et al., 2025) tunes layer-wise mixed precision bitwidths. These approaches operate at a coarser granularity (whole layers) and are orthogonal to Kitty's channel-wise approach. The paper positions Kitty as complementary: layer-wise allocation could potentially be combined with channel-wise allocation for further gains.

QuaRot (Ashkboos et al., 2024) applies orthogonal rotations to activations and the KV cache to make distributions outlier-free, which reduces the dynamic range that quantization must capture. While effective, this requires modifying the model's forward pass (applying rotations to weights as well) and is primarily a training-time intervention. Kitty targets post-training quantization that works with any pre-trained model.

Token Pruning vs. Quantization. An alternative direction (not evaluated in the paper) is token pruning—discarding less important tokens from the KV cache entirely (e.g., H2O, Zhang et al., 2024b). The paper's position is that quantization has a fundamental advantage over pruning: "Unlike token pruning, quantization preserves all contextual information without discarding tokens." This is an important philosophical and practical distinction. For tasks requiring retrieval of specific facts from long contexts (document QA, multi-turn dialogue), losing tokens risks losing critical information. Quantization compresses representation while retaining coverage.

The Key vs. Value Asymmetry. An important observation that shapes the paper's design comes from Table 2's ablation study comparing KIVI-K2V4* (2-bit Key, 4-bit Value) against KIVI-K4V2* (4-bit Key, 2-bit Value). The results are stark: KIVI-K4V2* substantially outperforms KIVI-K2V4*, approaching FP16 accuracy across multiple benchmarks. For Qwen3-8B, KIVI-K4V2* achieves 87.92 on MATH vs. 82.50 for KIVI-K2V4* (and 88.26 for FP16). This asymmetry—the Key cache being far more sensitive to quantization than the Value cache—is not merely an empirical curiosity; it becomes a design principle. Kitty focuses its channel-wise precision boost exclusively on the Key cache, since that's where the sensitivity lies.

How Kitty Positions Itself

Kitty's central insight is that the Key cache's sensitivity to quantization is channel-dependent: not all channels in the Key cache are equally important, and the accuracy degradation from 2-bit quantization comes disproportionately from quantizing a small fraction of critical channels. The paper's analysis in Section 3.2 operationalizes this through two observations:

Magnitude patterns (Observation 1, Figure 2a): A visualization of Key-cache activations reveals that a subset of channels consistently exhibits higher magnitudes. These high-magnitude channels are naturally more susceptible to quantization error—truncating a large value to 2-bit representation introduces more absolute error than truncating a small value.
Channel-wise quantization sensitivity (Observation 2, Figure 2b): By quantizing one channel at a time and measuring the resulting MSE in attention scores, the paper shows that different channels cause vastly different levels of distortion. A small fraction of channels dominate the error, while most channels can be quantized to 2 bits with minimal impact.

These observations motivate the paper's key innovation: Dynamic Channel-wise Precision Boost — identify the most quantization-sensitive channels in the Key cache and give them INT4 precision while aggressively quantizing everything else to INT2. The paper shows that boosting only 12.5% (Kitty) or 25% (Kitty-Pro) of Key-cache channels to 4-bit is sufficient to recover most of the accuracy loss from uniform 2-bit quantization, achieving near-parity with FP16 while maintaining memory savings close to pure 2-bit.

This positions Kitty differently from prior mixed-precision approaches in two crucial ways. First, the mixed precision is channel-wise, not token-wise (KVQuant's outlier approach) or temporal (KIVI's recent-token approach). This is motivated by the empirical observation that channel-level structure—not token-level structure—is the dominant source of quantization sensitivity in the Key cache. Second, the channel selection is dynamic and heuristic-guided (using average magnitude as a proxy for importance) rather than pre-determined or static. The method adapts to each model's specific activation patterns rather than applying a fixed mask.

The System Challenge. The paper's algorithmic contribution raises an immediate system-level challenge that prior mixed-precision schemes struggled with: how do you store and retrieve a Key cache where different channels have different precisions (2-bit vs. 4-bit) without introducing scattered memory accesses, per-element branching, or hard-coded masks—all of which kill GPU throughput? The paper explicitly states this challenge in the abstract:

"The main challenge is handling dynamic 4-bit channel boosts while keeping the page layout coalesced and the dequantization uniform, with no scattered reads or hard-coded masks."

Kitty's solution—decomposing each mixed-precision page into two separate 2-bit tensors (one dense, one structured sparse for the high-order bits of boosted channels)—is what distinguishes the work from being "just another mixed-precision scheme." The decomposition converts a heterogeneous precision layout into two homogeneous 2-bit layouts that can be loaded with uniform, coalesced memory accesses. This design choice is the bridge between accuracy recovery (the algorithm) and practical throughput gains (the system), and it directly addresses the hardware efficiency limitations that afflicted KVQuant's sparse-FP16 approach.

Where Kitty Fits in the Landscape. Kitty is positioned as a post-training quantization method—it requires no model fine-tuning, no architectural changes, and no retraining. This makes it applicable to any pre-trained model without additional computational cost, distinguishing it from rotation-based methods (QuaRot) or pruning approaches that may require retraining to recover accuracy. The paper's evaluation covers two model families (Qwen3 and LLaMA3) at scales from 8B to 70B parameters, suggesting broad applicability. However, the channel importance heuristic (magnitude-based selection) is evaluated with only one heuristic (magnitude) and one baseline (random selection), leaving the development of more principled selection strategies as explicit future work.

3. Technical Approach

This is primarily an algorithm–system co-design paper whose core idea is that 2-bit KV cache quantization can be made accurate by identifying and protecting only the most quantization-sensitive channels of the Key cache, and that this channel-wise mixed precision can be implemented efficiently on GPUs by decomposing heterogeneous-precision pages into two homogeneous 2-bit tensors.

3.1 Reader Orientation

The paper builds an end-to-end inference system that quantizes the Key-Value cache of large language models to roughly 2 bits per element while preserving model accuracy on reasoning tasks—something previous 2-bit methods catastrophically failed at. The system comprises an offline analysis phase (where channel sensitivity is characterized) and a runtime inference phase (where the quantization, storage, and dequantization happen dynamically as tokens are generated). The "shape" of the solution is a mixed-precision Key cache where a small fraction (12.5–25%) of channels are stored at INT4 and the rest at INT2, with a specialized page-centric memory layout that makes this heterogeneous precision look uniform to the GPU's memory subsystem.

3.2 Big-Picture Architecture (Diagram in Words)

The Kitty system has five major components, spanning algorithm and system:

Channel Importance Analyzer (offline/initialization) — identifies which Key-cache channels are most sensitive to quantization by measuring either their average activation magnitude or (in the analytical experiments) their per-channel impact on attention-score MSE. This produces a ranked list of channel indices and a "boost mask" indicating which channels get INT4 treatment.
Channel-wise Precision Boost Quantizer (runtime) — given the boost mask, quantizes each Key-cache channel to either INT2 or INT4 using per-channel quantization with learned scales and zero-points. The Value cache is quantized per-token, uniformly at 2 bits, with a sliding window of recent tokens kept in FP16.
Dense-Sparse Page Decomposition (runtime) — converts each mixed-precision Key-cache page (where some channels are 2-bit, some 4-bit) into two separate homogeneous tensors: a dense 2-bit tensor containing all channels' low-order bits, and a structured sparse 2-bit tensor containing only the high-order bits of boosted channels. Both tensors use the same 2-bit data type, simplifying loading.
Page-Centric Memory Layout (runtime) — organizes the quantized KV cache into fixed-size pages (each containing G tokens, default G = 128) following the PagedAttention paradigm. Key-cache pages store the two decomposed tensors plus metadata (scales, zero-points, boost indices). Value-cache pages store the 2-bit tensor plus full-precision buffers for sink and local tokens.
Triton Dequantization Kernels and Execution Pipeline (runtime) — GPU kernels that load quantized pages from HBM, reconstruct FP16 Key and Value tensors on-chip, and feed them into attention computation. A three-stage pipeline interleaves KV cache insertion, attention computation, and batch quantization to amortize overhead.

Information flows as follows. At model loading time, the channel importance analyzer processes a calibration dataset (or uses a lightweight heuristic) to produce per-layer boost masks. During inference, each new token's Key and Value vectors are first inserted into full-precision buffers (Sink for initial tokens, Q-Buffer for tokens awaiting quantization, Local for recent Value tokens). When the Q-Buffer fills (every G tokens), a quantization kernel converts the buffered tokens into the decomposed page format and writes them to GPU memory. During attention computation, the dequantization kernel loads pages from memory, reconstructs FP16 tensors using the boost index to map sparse high-order bits back to their correct channels, and passes the reconstructed tensors to the attention kernel.

3.3 Roadmap for the Deep Dive

First, the design space exploration that establishes why certain design choices (preserving sink tokens, prioritizing Key over Value) are necessary and how they inform the full method.
Second, the two empirical observations that motivate channel-wise precision boost: the magnitude patterns in the Key cache and the per-channel quantization sensitivity analysis using attention-score MSE.
Third, the channel-wise precision boost mechanism itself—how channels are ranked, how the boost mask is computed, and how the mixed-precision quantization is performed.
Fourth, the overall Kitty quantization scheme that integrates sink token preservation, channel-wise precision boost, group-wise Key quantization, and sliding-window Value quantization into one unified algorithm.
Fifth, the page-centric memory layout and the dense-sparse decomposition that make channel-wise mixed precision hardware-efficient.
Sixth, the GPU dequantization kernel (Algorithm 1) and the three-stage attention execution pipeline that coordinate the full-precision buffers with the quantized pages.

3.4 Detailed, Sentence-Based Technical Breakdown

Design Space Exploration: Establishing What Matters

Before introducing channel-wise precision boost, the paper conducts a systematic exploration of straightforward optimizations to understand where the accuracy gap comes from and which precision investments yield the best returns (Section 3.1). This exploration provides the empirical foundation for the algorithm's design.

Preserving Initial Tokens (Attention Sinks). The initial tokens of a sequence act as "attention sinks" (Xiao et al., 2024b)—they receive disproportionately high attention weights and significantly influence subsequent attention computations. The paper quantifies the benefit of keeping the first 32 tokens in full precision (FP16) rather than quantizing them. The variant KIVI-K2V2* applies the KIVI 2-bit quantization algorithm but preserves the initial 32 tokens in FP16. Table 2 shows this improves average accuracy by +8.28 on Qwen3-8B and +5.33 on LLaMA3-8B compared to uniform 2-bit KIVI (KIVI-K2V2). For instance, on Qwen3-8B MATH-Algebra, accuracy recovers from 47.29 to 74.92—still 13 points below FP16's 88.26, but a substantial recovery. This establishes that sink token preservation is necessary but insufficient for closing the 2-bit accuracy gap.

Asymmetric Sensitivity of Key vs. Value Cache. The paper's most consequential design-space finding comes from comparing two asymmetric precision configurations in Table 2:

KIVI-K2V4*: Key cache at 2 bits, Value cache at 4 bits, sink tokens preserved
KIVI-K4V2*: Key cache at 4 bits, Value cache at 2 bits, sink tokens preserved

The results are starkly one-sided. On Qwen3-8B, KIVI-K4V2* achieves 87.92 on MATH-Algebra (vs. 88.26 FP16, a gap of only 0.34), while KIVI-K2V4* achieves 82.50 (a gap of 5.76). Across all Qwen3-8B benchmarks, KIVI-K4V2* approaches FP16 accuracy, while KIVI-K2V4* lags substantially. The message is clear: the Key cache is far more sensitive to quantization than the Value cache. This asymmetry becomes a design principle—Kitty focuses its precision-boosting budget on the Key cache, keeping the Value cache uniformly at 2 bits with only the most recent tokens in FP16.

Why This Asymmetry Exists (Implicit Reasoning). The paper does not provide an explicit theoretical explanation, but the mechanism is implicit in the attention computation. The Key vectors are multiplied by Query vectors to produce attention logits via QK^T, and these logits feed into a softmax. Quantization errors in Key vectors are therefore amplified through the softmax's exponentiation: small perturbations in the attention logits can cause large changes in the attention weights (which must sum to 1, so increasing one weight necessarily decreases others). In contrast, Value vectors are linearly weighted by attention scores (softmax(QK^T) × V) to produce the attention output. Errors in Value vectors propagate linearly, not exponentially, making them fundamentally more robust to quantization noise.

Layer-Wise Application. The paper applies its quantization scheme uniformly across all layers. Prior work (MiniKV, KVTuner) has explored layer-discriminative precision allocation—assigning different bitwidths to different layers based on their sensitivity. Kitty does not incorporate this, and the authors position layer-wise allocation as potentially complementary to channel-wise allocation (you could do both). The channel-wise boost mask is currently per-layer identical (all layers use the same fraction of boosted channels), though the specific channels boosted may differ per layer based on per-layer importance scores.

Observation 1: Channel-Wise Magnitude Patterns in the Key Cache

The first motivating observation is visual and qualitative (Section 3.2, Figure 2a). The authors visualize the absolute values of Key-cache activations for a specific layer (Layer 10 of Qwen3-8B) by plotting activation magnitude across both the token dimension (horizontal axis, across all tokens in the sequence) and the channel dimension (vertical axis). The visualization reveals a consistent pattern: a subset of channels exhibits persistently higher magnitudes across all tokens, while most channels have relatively low magnitudes. The paper states that "similar patterns are observed on other layers."

This matters for quantization because of how uniform quantization works. In uniform affine quantization, a floating-point value x is mapped to an integer x_q via:

$x_q = \text{round}\left(\frac{x - z}{s}\right)$

where s is the quantization scale and z is the zero-point. The scale s determines the step size between representable levels. For a fixed bitwidth (say, 2 bits = 4 levels), the scale s must be chosen to cover the range of values in the group being quantized. If a channel has large-magnitude values, the scale must be large, which means the quantization step size is larger—and therefore the quantization error (the difference between the original float and its quantized approximation) is larger in absolute terms.

In per-channel quantization (which Kitty uses for the Key cache), each channel gets its own scale. So a high-magnitude channel and a low-magnitude channel both get scales appropriate to their own ranges. But the relative error introduced by quantizing to only 4 levels (2 bits) is inherently larger for channels that carry more signal. A channel whose values span [0, 10] will have quantization steps of size 10/3 ≈ 3.3 when quantized to 2 bits; a channel whose values span [0, 1] will have steps of 1/3 ≈ 0.33. The high-magnitude channel loses more information per quantization step.

The observation, therefore, is not that high-magnitude channels can't be quantized per-channel—they can—but rather that they suffer more from the limited representational capacity of 2 bits. Giving these channels 4 bits (16 levels instead of 4, with 4× finer granularity) should disproportionately improve accuracy compared to giving the same precision boost to low-magnitude channels.

Observation 2: Channel-Wise Quantization Sensitivity via Attention-Score MSE

The second observation is quantitative and causal (Section 3.2, Figure 2b). The authors measure the per-channel sensitivity to quantization by conducting a controlled experiment: quantize exactly one channel of the Key cache to 2-bit INT2 (leaving all other channels at FP16), then compute how much the attention scores change. The metric is:

$\text{MSE}_{\text{channel } i} = \text{MSE}\left(\text{softmax}\left(\frac{Q K^T}{\sqrt{d}}\right), \text{softmax}\left(\frac{Q K^T_{\text{perturbed}, i}}{\sqrt{d}}\right)\right)$

where K_perturbed,i is the Key cache with only channel i quantized to 2 bits and all other channels at full precision.

What this measures. For each channel, the experiment isolates the impact of quantizing that single channel on the final attention weights. The attention weights determine how much each previous token's Value vector contributes to the current output, so perturbations here directly affect the model's computation. By measuring MSE between the original attention weight matrix and the one produced after quantizing only channel i, the method quantifies that channel's marginal contribution to attention distortion under quantization.

Key findings from Figure 2b. The paper plots the MSE for each channel across multiple query heads (since Grouped-Query Attention means multiple Q heads share the same K head). The results show that:

Different channels cause vastly different levels of attention-score distortion. A small number of channels produce large MSE values (several orders of magnitude higher than typical channels), while most channels produce negligible MSE.
The pattern is consistent across query heads. The paper notes that "The pattern is consistent between different Q heads who share the same Key cache due to grouped-query attention." In the plot, different query heads are shown in different colors, and they follow the same channel-wise sensitivity profile—the same channels are important across all heads that share the key.
Only a small fraction of channels dominate the error. The "heavy tail" in the MSE distribution means that boosting a small fraction of the most sensitive channels to higher precision should capture most of the benefit of full-precision Key cache.

Why this is not just a magnitude proxy. While there is likely correlation between magnitude (Observation 1) and sensitivity (Observation 2), they are conceptually distinct. A channel could have high magnitude but be orthogonal to the Query vectors being used at inference time, making it less impactful on attention scores. Conversely, a channel with moderate magnitude could be aligned with Query directions, making it disproportionately influential. The MSE measurement directly captures the downstream impact on attention computation, which is what matters for model accuracy, rather than relying on a proxy like magnitude. However, computing per-channel MSE at runtime is prohibitively expensive (it requires the full attention score computation against an FP16 baseline for every channel), which is why the paper uses a lightweight magnitude-based heuristic for actual deployment.

The Channel-Wise Precision Boost Mechanism

The core algorithmic innovation of Kitty is the method for selecting which Key-cache channels receive higher precision (Section 3.2, "Key Innovation").

The Central Tradeoff. For a Key-cache head of dimension D, boosting a fraction α of channels from 2-bit to 4-bit increases the per-element storage from 2 bits to α × 4 + (1-α) × 2 = 2 + 2α bits on average. That is, the memory overhead relative to pure 2-bit is α, and the compression ratio relative to FP16 is 16 / (2 + 2α). For α = 0.125 (Kitty's default: 12.5% of channels boosted), the average bits per element is 2.25, yielding a ~7.1× compression vs FP16 (compared to 8× for pure 2-bit). For α = 0.25 (Kitty-Pro), the average is 2.5 bits, yielding ~6.4× compression. The paper's experiments show that α = 0.125 already recovers most of the accuracy, and α = 0.25 achieves near-parity with FP16. The small memory overhead (0.25–0.5 bits per element) is the price of closing the accuracy gap.

Channel Importance Scoring (Runtime Heuristic). At runtime, the system needs to decide which channels to boost without computing the expensive per-channel MSE analysis. The paper uses a magnitude-based heuristic: the importance score of channel i is its average absolute activation magnitude across all tokens in the current quantization group:

$s_i = \frac{1}{T} \sum_{t=1}^{T} |x_{i,t}|$

where T is the number of tokens in the group (typically G = 128), x_{i,t} is the floating-point activation of channel i at token position t, and |·| denotes absolute value.

What this computes. For each channel in the Key cache, sum the absolute values of that channel's activations across all tokens in the quantization group, then divide by the number of tokens. The result is a scalar s_i representing the average magnitude of that channel. Channels with higher average magnitudes are deemed more important and more deserving of the INT4 precision boost.

Why magnitude works as a heuristic. The paper provides two intuitions. First, "channels with larger average magnitudes are more susceptible to quantization error"—the relative error argument discussed under Observation 1. Second, "such channels tend to exert greater influence on attention scores"—the correlation between magnitude and attention impact observed empirically. The ablation study in Figure 4 compares magnitude-based selection against random channel selection, showing that magnitude-based selection yields "substantially greater benefits," confirming that the heuristic captures real signal beyond random chance.

Why this is "dynamic." The term "Dynamic" in "Dynamic Channel-wise Precision Boost" refers to the fact that the channel importance scores are computed at runtime on the actual activations, not pre-determined from a calibration dataset. Each time a quantization group (page of G tokens) is formed, the scores s_i are recomputed on those tokens' activations. This means the boosted channels can change between quantization groups if the activation patterns shift. The paper does not extensively analyze how much the boosted channel set varies across groups, but the mechanism allows it.

Selecting the Top-K Channels. Given the importance scores s_i for all D channels and a target boost fraction α, the system selects the K = α × D channels with the highest scores. In Kitty's default configuration with α = 0.125 and a typical head dimension of D = 128, this means K = 16 channels are boosted to INT4. In Kitty-Pro with α = 0.25, K = 32 channels are boosted. The remaining D - K channels are quantized to INT2.

The Boost Mask. The set of boosted channels is represented as a boolean mask of length D, where element i is True if channel i is boosted and False otherwise. This mask is stored as metadata alongside each quantized page and is used during dequantization to correctly reconstruct the full-precision tensor (mapping the sparse high-order bits back to the correct channels).

Per-Channel Quantization Parameters. For each channel (whether boosted or not), the system computes per-channel quantization parameters: a scale s and a zero-point z. For boosted channels, these parameters correspond to INT4 representation (16 quantization levels); for non-boosted channels, they correspond to INT2 representation (4 levels). The scale and zero-point are computed to minimize the L2 distance between the original floating-point values and their quantized approximations, following standard min-max or MSE-optimal quantization calibration.

Quantization Step. Given a floating-point Key-cache slice of shape (D, G) (channels × tokens in the group) and the boost mask:

For each channel i, compute scale_i and zero_point_i based on the value range of that channel across all G tokens.
For each channel i, determine the target bitwidth: 4 bits if boosted, 2 bits if not.
For each element x at channel i, token t, quantize to integer via q = round((x - zero_point_i) / scale_i), then clamp to the valid range [0, 2^b - 1] where b is the bitwidth.
Store the quantized integers, scales, zero-points, and boost mask as the page representation.

Kitty: The Overall Quantization Scheme

Kitty integrates channel-wise precision boost with several complementary techniques into a unified quantization algorithm (Section 3.3). The full scheme is illustrated in Figure 1 and operates as follows:

Key Cache Quantization (Per-Channel, Grouped, with Sink and Boost).

Sink tokens: The first S = 32 tokens' Key vectors are stored in full precision (FP16) and never quantized. This preserves the attention sinks identified as disproportionately influential.
Quantization grouping: Beyond the sink tokens, Key vectors are accumulated in a Q-Buffer until G = 128 tokens have been collected. Each group of G tokens forms one quantization group (one page).
Channel-wise mixed precision: Within each group, per-channel importance scores s_i (Equation 2) are computed. The top K = α × D channels are selected for INT4 quantization; the remaining D - K channels are quantized to INT2. Per-channel scales and zero-points are computed for all channels based on their respective value ranges within the group.
Storage decomposition: The quantized group is decomposed into two 2-bit tensors (detailed in Section 3.4 page-centric memory layout).

Value Cache Quantization (Per-Token, with Sliding Window).

Sink tokens: The first S = 32 tokens' Value vectors are stored in FP16.
Local window: The most recent R = 128 tokens' Value vectors are also stored in FP16, creating a sliding window of full-precision recent context. This addresses the high temporal locality of attention: recent tokens are attended to more heavily, and preserving them in full precision reduces error where it matters most.
Quantized region: Tokens between the sink and the local window are quantized per-token (not per-channel) to 2 bits. Per-token quantization for the Value cache follows KIVI's approach: each token's Value vector gets its own scale and zero-point, quantized uniformly to 2 bits across all channels.
No channel-wise boost for Value cache: Based on the Key-vs-Value asymmetry finding (Table 2: KIVI-K4V2* >> KIVI-K2V4*), the Value cache does not receive channel-wise precision boost. The Value cache's relative robustness to quantization means the additional memory cost of boosting would have diminishing returns.

Default Configuration. The paper specifies a default parameter set that balances accuracy and memory: S = 32 (sink tokens), R = 128 (local window for Value), G = 128 (quantization group size). For the boost fraction, Kitty uses α = 0.125 (12.5% of Key-cache channels boosted to INT4) and Kitty-Pro uses α = 0.25. These values were chosen empirically: the ablation in Figure 4 shows that accuracy improves monotonically with the boost fraction, and 12.5% already captures most of the recovery while keeping memory overhead minimal.

Relationship to KIVI. The paper states that Kitty "builds upon the foundation of KIVI" and inherits several of its design choices: per-channel quantization for the Key cache, per-token quantization for the Value cache, and the use of grouped quantization. Kitty's additions are (1) sink token preservation in FP16 (KIVI-K2V2 does not preserve sinks; KIVI-K2V2* adds them but was introduced as a variant in this paper), (2) channel-wise precision boost with magnitude-based selection, and (3) the dense-sparse decomposition for hardware-efficient page storage.

Page-Centric Memory Layout and Dense-Sparse Decomposition

The algorithmic choice to boost only some Key-cache channels to INT4 creates a system-level challenge: how to store pages where different channels have different bitwidths without scattered memory accesses or per-element branching during loading (Section 4.1). The solution is a dynamic dense-sparse decomposition inspired by prior work on mixed-precision weight storage (Wu et al., 2023; Xia et al., 2024).

The Problem. A quantized Key-cache page after channel-wise precision boost is heterogeneous: some channels are stored as 4-bit integers (needing 4 bits per element to represent 16 possible values), others as 2-bit integers (2 bits per element, 4 possible values). If you try to store this naïvely—interleaving 2-bit and 4-bit elements in a single array—you cannot use uniform memory access patterns. The GPU would need to branch on each element's precision, killing throughput. Prior mixed-precision approaches like KVQuant's sparse FP16 representation also suffered from this: irregular memory access patterns from sparse-dense layouts introduce "additional runtime overhead from sparse-dense multiplications, which could be slow on GPUs."

The Decomposition. Kitty decomposes each mixed-precision page into two separate tensors, both using a uniform 2-bit representation:

Tensor_2bits: shape (D, G), storing the lower 2 bits of every channel (both boosted and non-boosted). For non-boosted channels, this is the complete 2-bit quantized value. For boosted channels, this is the lower 2 bits of their 4-bit representation.
Tensor_High_2bits: shape (D_boosted, G), storing the higher 2 bits of only the boosted channels. D_boosted = K = α × D is the number of boosted channels. This is a "structured sparse" tensor—sparse because it only contains entries for boosted channels, but structured because the sparsity pattern is channel-wise (entire channels are either present or absent), not element-wise scatter.

Why This Works. A 4-bit integer can be expressed as two 2-bit components: the lower 2 bits (values 0–3) and the upper 2 bits (values 0–3, representing multiples of 4). The reconstructed value is low_bits + 4 × high_bits. By storing the low bits of all channels together and the high bits of boosted channels separately, both tensors become uniform 2-bit arrays. The GPU can load both as contiguous blocks of 2-bit data without any per-element precision dispatch.

The Boost Index Tensor. To correctly reconstruct the full tensor during dequantization, the system needs to know which channels the entries in Tensor_High_2bits correspond to. This is stored as an index tensor:

Boost_IDX_uint8: shape (D,), containing integer indices for each of the D channels. For boosted channels, the entry is the physical offset (0 to D_boosted - 1) into Tensor_High_2bits. For non-boosted channels, the entry is a sentinel value of D_boosted + 1 (an out-of-range index). This creates a mapping from the logical channel dimension (0 to D-1) to the compact boosted subspace.

Page Structure for Key Cache (Figure 3a). Each Key-cache page (representing G tokens across D channels) occupies GPU memory as:

Tensor_2bits: packed 2-bit representation, shape (D, G), physically stored as uint8 elements where each byte packs 4 consecutive 2-bit values from the token dimension.
Tensor_High_2bits: packed 2-bit representation, shape (D_boosted, G), also packed into uint8 with the same packing scheme.
Boost_IDX_uint8: index mapping, shape (D,).
Scales and zero-points: Per-channel quantization parameters for Key cache, stored as FP16 arrays of shape (D,). Two sets: one for INT2 channels, one for boosted INT4 channels (though since each channel has its own parameters derived from its value range, the "precision" is reflected in the quantization range, not in separate scale arrays).

Additionally, the full-precision buffers (Sink and Q-Buffer) are stored outside the paged system as contiguous FP16 tensors.

Page Structure for Value Cache (Figure 3b). Value-cache pages are simpler because the Value cache uses uniform 2-bit per-token quantization without channel-wise boost:

Tensor_2bits: packed 2-bit representation, shape (G, D). Note the transposed shape: per-token quantization means each token (row of length D) gets its own scale, so the natural layout groups by token.
Scales and zero-points: Per-token parameters, shape (G,).
Full-precision buffers for Sink (first 32 tokens) and Local (most recent 128 tokens).

Page Size Choice. The quantization group size G serves double duty as the page size, adopting the PagedAttention paradigm (Kwon et al., 2023) for memory management. The paper chooses G = 128 as the default, noting it "provides a good balance between accuracy and memory savings." A larger G would amortize quantization overhead better but would increase the granularity of memory allocation (wasting memory if sequences don't fill pages exactly). A smaller G would be more memory-efficient but would increase quantization overhead (more frequent quantization launches) and potentially reduce accuracy (smaller groups for computing per-channel statistics).

Memory Overhead of the Decomposition. Relative to a pure 2-bit scheme (which stores only D × G × 2 bits per Key page), Kitty adds:

The boosted high bits: D_boosted × G × 2 bits = α × D × G × 2 bits.
The boost index: D bytes (uint8 indices) = D × 8 bits.
The metadata (scales, zero-points) is comparable between pure 2-bit and Kitty (both need per-channel parameters; Kitty needs two sets but with the same total dimensionality).

For α = 0.125, D = 128, G = 128, the per-page storage for the Key cache is approximately 128 × 128 × 2.25 bits for the quantized data plus 128 × 8 bits for the index, yielding roughly 2.26 bits per element—a negligible overhead over the 2.25 theoretical average from the mixed precision alone.

GPU Dequantization Kernel (Algorithm 1)

The dequantization kernel reconstructs FP16 Key-cache tensors from the decomposed page representation directly in on-chip GPU memory (register file and shared memory) before passing them to the attention computation (Section 4.2). Algorithm 1 presents the Triton-style pseudocode.

Input and Output. The kernel takes as input a quantized Key-cache page (the packed Tensor_2bits, Tensor_High_2bits, the boost index, scales, and zero-points) and produces as output a dequantized FP16 tensor K_fp16 of shape (D, T), where T is the number of tokens in the page (typically G = 128).

Step-by-Step Reconstruction.

Load metadata (line 4): The per-channel scales and zero-points are loaded from the metadata storage. These are FP16 values, one per channel.
Load boost index (line 5): The Boost_IDX_uint8 tensor of shape (D,) is loaded. This provides the mapping from logical channel to boosted-channel offset.
Compute boost mask (line 6): A boolean mask boost_mask of shape (D,) is derived: boost_mask[i] = True if boost_idx[i] <= D_boosted, and False if boost_idx[i] == D_boosted + 1 (the sentinel value). This mask will be used to conditionally load the high-order bits only for boosted channels.
Load and unpack low bits (lines 8–9): The Tensor_2bits is loaded from HBM as a uint8 tensor of shape (D, T/4) (since each byte packs 4 two-bit values along the token dimension). The kernel then unpacks these bytes into the individual 2-bit values. The unpacking uses a bit-shift and mask: X_low = (X_low >> shifts) & 0x3, where shifts is a pre-defined array [0, 2, 4, 6, 0, 2, 4, 6, ...] cycling through the 4 two-bit positions within each byte. Each element of X_low is now a value in {0, 1, 2, 3} representing the lower 2 bits.
Conditionally load and unpack high bits (lines 10–12): The Tensor_High_2bits is loaded from HBM as a uint8 tensor of shape (D_boosted, T/4). However, this loading is conditional: only channels where boost_mask[i] is True read from this tensor, using boost_idx[i] as the physical row index into the compact boosted subspace. For non-boosted channels, the high bits are implicitly zero. The loaded bytes are unpacked identically to the low bits: X_high = (X_high >> shifts) & 0x3.
Combine and dequantize (lines 13–15): The full integer representation is reconstructed by combining: X = X_low | (X_high << 2). For boosted channels, this produces a 4-bit integer (0–15). For non-boosted channels (where X_high is 0), this produces a 2-bit integer (0–3). The combined integer is then dequantized to floating point: K_fp16 = X ⊙ scale + zero_point, where ⊙ denotes element-wise multiplication (broadcasting the per-channel scale across the token dimension), and the addition similarly broadcasts per-channel zero-points.

Why This Kernel Design is Efficient. The key efficiency properties are:

Coalesced memory access: Both Tensor_2bits and Tensor_High_2bits are loaded as contiguous blocks of uint8 data. Even though Tensor_High_2bits is sparse (only some channels read from it), the conditional load uses the boost index to compute a direct offset, avoiding scattered reads. All threads in a warp access adjacent memory locations for the tiles they are responsible for.
No divergent branching on precision: The unpacking operations are identical for all channels (same bit-shift and mask logic), regardless of whether the channel is boosted or not. The only difference is whether X_high comes from memory (boosted) or is zero (non-boosted), which is handled by the conditional load without control-flow divergence.
On-chip reconstruction: The entire reconstruction happens in GPU registers and shared memory, avoiding intermediate writes to HBM. The dequantized K_fp16 tensor is used directly in the subsequent attention computation (Q × K^T), never written back to memory.
No hard-coded masks: The boost index tensor is data, not code. The same kernel works for any boost pattern without recompilation, supporting the "dynamic" aspect of the algorithm where boosted channels can change between pages.

Lightweight Attention Execution Pipeline

The runtime pipeline coordinates the interaction between the full-precision buffers (Sink, Q-Buffer, Local) and the quantized pages during autoregressive generation (Section 4.3). The pipeline has three stages that execute at each decoding step, with the third stage (quantization) triggered only periodically.

Stage 1: Inserting New KV Vectors in Full Precision.

When a new token is generated, its Key and Value vectors (computed by the model's attention projection layers) must be added to the KV cache. The destination depends on the current sequence position:

If the Sink buffer is not yet full (sequence position < 32): The new Key and Value vectors are directly inserted into the Sink buffer, which stores them in FP16. No quantization occurs for these tokens—they remain in full precision for the lifetime of the sequence.
If the Sink is full (sequence position ≥ 32): For the Key cache, the new Key vector is inserted into the Q-Buffer, a temporary FP16 buffer that accumulates tokens until it reaches size G (128 tokens). For the Value cache, the new Value vector is inserted into the Local buffer, a sliding window of the most recent R = 128 tokens in FP16. If the Local buffer is already full (which happens once the sequence exceeds S + R = 160 tokens), the oldest token in the Local buffer is evicted and appended to the Value Q-Buffer.

This staging ensures that:

The initial tokens (Sink) and the most recent tokens (Local for Value) are always in full precision, providing accurate attention for these critical regions.
The Key cache has no Local window (only Sink and Q-Buffer/quantized pages) because the channel-wise precision boost is applied to all non-sink tokens, and recent keys don't receive special FP16 treatment beyond the sink window.
The Value cache's Local window provides full-precision recent context for the Value side, complementing the channel-wise precision boost on the Key side.

Stage 2: Attention Computation.

The attention computation uses two custom Triton kernels (qk_kernel and sv_kernel) plus a PyTorch softmax operator:

qk_kernel: Loads the current Query vector (single token, shape (B, H, 1, D)) from registers or shared memory. Loads the full Key cache, which includes:
- The FP16 Sink tensors (from the Sink buffer)
- The dequantized Key pages (using Algorithm 1 to reconstruct FP16 from quantized pages)
- The current Q-Buffer contents (FP16 tokens not yet quantized)
All these sources are concatenated to form the complete Key tensor of shape (B, H, L, D) where L is the current sequence length. The kernel then computes the matrix multiplication Q × K^T, producing attention logits of shape (B, H, 1, L).
Softmax: The attention logits are passed through the standard PyTorch softmax function (with scaling by 1/√d), producing attention weights.
sv_kernel: Loads the attention weights and the full Value cache (constructed similarly from Sink FP16, Local FP16, Q-Buffer FP16, and dequantized Value pages). Computes the weighted sum Attention_weights × V, producing the attention output of shape (B, H, 1, D).

The separation into two kernels (qk and sv) rather than a fused FlashAttention-style kernel is a deliberate simplification for the proof-of-concept implementation. The paper acknowledges that "fusing these kernels into one" (following Dao et al., 2022) is left for future work. The current design prioritizes correctness and modularity over maximum performance—each kernel can be developed and debugged independently.

Stage 3: Quantization and Packing (Periodic).

After the attention computation, the Q-Buffer is checked. If the Q-Buffer has accumulated G = 128 tokens (which happens exactly once every 128 decoding steps, since one token is added per step after the Sink fills), a quantization and packing kernel is launched:

Key cache quantization: The G Key vectors in the Q-Buffer are processed:
- Per-channel importance scores s_i (Equation 2) are computed using the average absolute magnitude across the G tokens.
- The top K = α × D channels are selected for INT4 quantization.
- Per-channel scales and zero-points are computed for all channels.
- The quantized values are decomposed into Tensor_2bits and Tensor_High_2bits, packed into uint8 format, and written to a newly allocated page in GPU HBM.
- The boost index and metadata are stored alongside.
Value cache quantization: The G Value vectors in the Value Q-Buffer (which were evicted from the Local buffer) are quantized per-token to INT2, packed, and written to a Value page.
Buffer clearing: The Q-Buffer is cleared, ready to accumulate the next 128 tokens.

Amortization of Quantization Overhead. Because quantization happens only once every G = 128 decoding steps (less than 1% of steps), the overhead is effectively amortized to zero. The paper notes that "this quantization process can be launched at most once every G decoding steps. In this way, the quantization overhead is efficiently amortized and becomes negligible." This is critical for throughput: if quantization were performed at every decoding step, the latency of computing channel importance, fitting quantization parameters, and packing would dominate the attention computation itself.

Integration with PagedAttention. The page-based memory layout is compatible with the PagedAttention memory management scheme (Kwon et al., 2023). Sequences are allocated pages dynamically as they grow, with a page table mapping logical sequence positions to physical page addresses. When a sequence's Q-Buffer fills, a new page is allocated from a free-page pool, quantized, and linked into the sequence's page table. When a sequence completes, its pages are returned to the pool. This avoids memory fragmentation and enables memory sharing across sequences (e.g., in beam search or parallel decoding), though the paper does not evaluate multi-sequence sharing scenarios.

Memory Footprint Breakdown. For a complete sequence, the KV cache storage consists of:

Sink: S × D FP16 elements for keys + S × D FP16 elements for values = 2 × S × D × 2 bytes (at FP16).
Q-Buffer (active, not yet paged): At most G × D FP16 elements for keys + at most G × D FP16 elements for values = 2 × G × D × 2 bytes.
Local buffer (Value only): R × D FP16 elements = R × D × 2 bytes.
Quantized pages: For a sequence of length L > S, there are approximately ceil((L - S - R) / G) Key pages and the same number of Value pages (the Value pages account for the region between Sink+Local and the total length). Each Key page stores D × G elements at ~2.25 bits each (for Kitty) plus metadata. Each Value page stores D × G elements at 2 bits each plus metadata.

For very long sequences (L ≫ S, R, G), the paged memory dominates, and the effective bits-per-element approaches 2.25 for keys and 2.0 for values (averaging ~2.125 bits per element across the full KV cache), yielding the nearly 8× compression relative to FP16 that the paper reports.

Design Rationale Summary

The technical approach reflects a series of deliberate design choices, each with a clear justification grounded in the paper's empirical findings:

Per-channel quantization for Key cache (inherited from KIVI): Channels have different value ranges; per-channel scales prevent high-magnitude channels from dictating a coarse step size that hurts low-magnitude channels.
Per-token quantization for Value cache (inherited from KIVI): Value vectors have token-level variation that per-token quantization captures better than per-channel.
Sink token preservation (from StreamingLLM/Xiao et al., 2024b): Initial tokens receive disproportionately high attention; preserving them in FP16 eliminates quantization noise where its impact on attention weights is largest.
Channel-wise precision boost on Key only (from Table 2 asymmetry): The Key cache is far more sensitivity to quantization than the Value cache; allocating the precision budget to Key channels yields higher accuracy per bit.
Magnitude-based channel selection (from Observations 1 and 2): High-magnitude channels are both more susceptible to quantization error and more influential on attention scores; magnitude is a cheap, effective proxy for the expensive per-channel MSE analysis.
Dense-sparse page decomposition (from GPU hardware constraints): Mixed-precision storage kills coalescing; decomposing into uniform 2-bit tensors restores uniform memory access patterns at the cost of a small metadata overhead.
Q-Buffer batching of quantization (from amortization): Quantizing every G tokens rather than every token makes the quantization overhead negligible relative to the attention computation cost.
Value Local window (from temporal locality): Recent tokens are heavily attended to in autoregressive generation; preserving them in FP16 provides full-precision context for the most impactful Value vectors without the complexity of channel-wise boost on the Value side.

4. Key Insights and Innovations

Innovation 1: Identifying Channel-Wise Sensitivity as the Fundamental Axis for KV Cache Precision Allocation

Prior work on mixed-precision KV cache quantization operated almost exclusively along the token dimension—preserving important tokens (KVQuant's outliers, KIVI's recent tokens) or entire layers (MiniKV, KVTuner) in higher precision. The dominant implicit assumption was that which tokens you protect determines accuracy, because attention is fundamentally about token-to-token interactions. Kitty breaks from this assumption with a simple but powerful diagnostic: the sensitivity to quantization lives primarily in the channel dimension of the Key cache, not the token dimension.

The evidence in Figure 2b makes this case decisively. By measuring the per-channel MSE induced in attention scores when each Key-cache channel is individually quantized to 2 bits, the paper reveals a heavy-tailed distribution where a small fraction of channels accounts for the vast majority of attention distortion—and this pattern holds consistently across query heads within the same GQA group. This is not an incremental refinement of token-wise mixed precision; it is a reframing of where the information bottleneck lives. The field had been asking "which tokens should we protect?" when the more incisive question was "which feature dimensions within each token's key representation carry the signal that matters for attention?"

This matters beyond the specific solution (boosting channels to INT4). It implies that per-channel quantization, already known to be better than per-token for the Key cache (KIVI, KVQuant), is not just a calibration convenience—it's addressing a structural property of how key representations encode information. The channel dimension is where representational capacity is most strained under aggressive quantization, and therefore where precision should be differentially allocated.

The paper's Key-vs-Value asymmetry finding (Table 2: KIVI-K4V2* achieves near-FP16 accuracy while KIVI-K2V4* lags substantially) corroborates this at a coarser level: the Key cache's sensitivity arises because Key vectors participate in the softmax via exponentiation, where small per-channel perturbations get amplified nonlinearly, while Value vectors contribute through a linear weighted sum. This asymmetry was implicitly understood (prior work treated Key and Value differently), but Kitty provides the first explicit diagnostic showing how the sensitivity localizes to specific channels within the Key cache, enabling targeted rather than blanket precision allocation.

The conceptual shift is from temporal or token-wise mixed precision to feature-wise mixed precision. This reframing opens a design space that prior work hadn't explored: rather than protecting tokens that are "important" (a notion that depends on the query and is hard to determine statically), protect the feature dimensions that are inherently more fragile to quantization, independent of the specific query. The channel-wise importance is a property of the model's learned representations, not of the particular inference-time computation.

Innovation 2: Decomposing Heterogeneous Precision into Homogeneous Storage as a System Design Principle

The paper's dense-sparse page decomposition (Section 4.1) is not merely an implementation detail—it's a system design principle that resolves a tension that had plagued prior mixed-precision KV cache approaches. The tension is this: mixed precision (storing some elements at higher bitwidth than others) is necessary for accuracy, but heterogeneous memory layouts destroy GPU throughput because they force divergent memory access patterns, per-element dispatch logic, or sparse-dense multiplication overhead.

Prior work dealt with this tension poorly. KVQuant (Hooper et al., 2024) stored outliers in a sparse FP16 format alongside low-bit dense tensors, which the paper explicitly critiques: "KVQuant is not hardware-friendly and usually suffers from low system-level efficiency, since it introduces additional runtime overhead from sparse-dense multiplications, which could be slow on GPUs." The insight is that keeping the precision heterogeneity in the storage format is the wrong approach—instead, decompose the heterogeneous representation into multiple homogeneous components that can each be accessed uniformly.

Kitty's decomposition converts one mixed-precision page into two 2-bit tensors: a dense tensor holding the lower two bits of every channel, and a structured sparse tensor holding the higher two bits of only the boosted channels. Both tensors use the same data type (2-bit unsigned integers), the same packing scheme (four 2-bit values per byte), and the same access pattern (contiguous loads). The sparsity in the high-bits tensor is structured (entire channels are present or absent, indexed by a small metadata array), not scattered element-wise. This converts a heterogeneous precision problem into a homogeneous storage problem with a lightweight indirection.

The significance of this principle extends beyond KV cache quantization. It's a design pattern for any system that needs to combine accuracy-preserving mixed precision with GPU-friendly memory access: push the heterogeneity into the metadata and decomposition logic, keep the bulk data tensors uniform. The boost index (Boost_IDX_uint8, shape (D,)) is tiny compared to the page data (shape (D, G) with G = 128), so the overhead is negligible. The dequantization kernel (Algorithm 1) never branches on per-element precision—it loads both tensors uniformly and uses the boost index only to determine which channels read from the high-bits tensor, with non-boosted channels getting implicit zeros.

This principle is what makes Kitty's algorithmic contribution (channel-wise precision boost) deployable. Without it, the accuracy gains from channel-wise mixed precision would remain a simulation-only result, unrealizable in a production inference system. The paper's claim of 2.1×–4.1× throughput improvement depends not just on the quantization scheme's memory reduction but on the decomposition enabling efficient GPU execution of attention over that quantized cache.

Innovation 3: Dynamic, Heuristic-Guided Channel Importance as a Practical Proxy for Expensive Sensitivity Analysis

The paper introduces a method for identifying quantization-sensitive channels at runtime using a lightweight magnitude-based heuristic (Equation 2), rather than the expensive per-channel MSE analysis used to motivate the approach (Figure 2b). This is not itself a novel heuristic—magnitude-based importance scoring is widespread in model compression. What's distinctive is the validation that such a simple heuristic suffices for this specific problem, and the demonstration that dynamic computation of importance scores on the actual runtime activations outperforms static or random selection.

Figure 4 provides the key evidence: magnitude-based channel selection consistently and substantially outperforms random selection across boost rates on GSM8K and MATH-Algebra. This establishes that the heuristic captures genuine signal about channel sensitivity to quantization, not merely that "any subset of channels at higher precision helps." The monotonic accuracy improvement with higher boost rates (from 0% to 25% and beyond) further confirms that the magnitude ranking is ordered correctly—channels selected earlier (higher magnitude) provide more accuracy benefit per boost than channels selected later.

The "dynamic" aspect of the approach—recomputing importance scores for each quantization group of G = 128 tokens—is conceptually important even if the paper doesn't extensively analyze how much the boosted channel set varies across groups. It means the method can adapt to activation patterns that shift across different regions of long sequences. A static calibration on a fixed dataset would bake in assumptions about the activation distribution that may not hold for out-of-distribution prompts, very long contexts, or different generation temperatures. Dynamic scoring makes the method more robust to distribution shift, even if the magnitude heuristic itself is simple.

This contribution is best understood as establishing a feasibility result: channel-wise precision boost can be made practical at runtime with negligible overhead (the importance computation is amortized across G = 128 tokens in the quantization step) using a heuristic that captures enough of the true channel sensitivity to deliver near-FP16 accuracy. It lowers the barrier to deploying channel-wise mixed precision from "requires expensive offline per-channel MSE analysis" to "compute average magnitudes and pick the top K"—a dramatically simpler operational profile.

5. Experimental Analysis

Evaluation Methodology

The paper evaluates Kitty along two complementary axes: inference accuracy (does the quantization preserve model performance on downstream tasks?) and system efficiency (does the memory reduction translate to higher throughput in a realistic inference engine?). The accuracy evaluation uses a simulation framework integrated with HuggingFace Transformers, while the system evaluation uses a custom-built inference engine with the page-centric memory layout and Triton kernels described in Section 4.

Dataset. Accuracy is evaluated on seven reasoning and generation benchmarks: GSM8K (Cobbe et al., 2021; 8-shot prompts), MATH-Algebra (Hendrycks et al., 2021; 4-shot prompts), GPQA-Diamond (Rein et al., 2023; 5-shot prompts), HumanEval (Chen et al., 2021; pass@1 metric), AIME24 (math ai, 2024), and AIME25 (math ai, 2025). For AIME tasks, the maximum generation length is extended to 32,768 tokens to test long-context robustness; for all other tasks, it is capped at 4,096 tokens. The choice of benchmarks spans mathematical reasoning, code generation, and graduate-level science QA, providing a diverse test of whether quantization degrades different reasoning modalities.
Base model(s). The paper evaluates two model families: Qwen3 (Yang et al., 2025) at scales of 8B, 14B, and 32B parameters, and LLaMA3 (Dubey et al., 2024) at 8B (LLaMA3.1-8B-Instruct) and 70B (LLaMA3.3-70B-Instruct) parameters. These families span different architectures and training recipes, testing whether the channel-wise sensitivity patterns generalize beyond a single model design. The paper states the models were chosen because they are "representative of the capabilities of many contemporary LLMs" (implicitly, reasoning-capable models where KV cache accuracy matters for long-context generation).
Metrics. For GSM8K, MATH-Algebra, GPQA-Diamond, AIME24, and AIME25, the primary metric is accuracy: the fraction of test examples where the extracted model output matches the ground-truth label. For HumanEval, the metric is pass@1 (functional correctness of generated code on the first attempt). For system efficiency, the metrics are GPU memory usage (peak GB during inference) and throughput (tokens generated per second). Accuracy experiments are repeated 3–10 times, and the paper reports both the average accuracy and the "maximum observed deviation" (error bounds not explicitly shown in all tables but stated in Section 5.1).
Baselines. The paper compares against several configurations:
- K16V16: FP16 KV cache with HuggingFace Transformers default implementation. This is the accuracy upper bound (no quantization).
- KIVI-K2V2: The KIVI algorithm (Liu et al., 2024) quantizing both Key and Value caches uniformly to 2 bits. This is the primary target to beat—the state-of-the-art 2-bit method that the paper shows catastrophically degrades accuracy.
- KIVI-K2V2*: A variant of KIVI-K2V2 where the first 32 tokens (attention sinks) are preserved in FP16 for both Key and Value caches. Introduced by the authors as an improved baseline that partially recovers accuracy.
- KIVI-K2V4* and KIVI-K4V2*: Asymmetric precision variants (from Section 3.1) used to diagnose Key-vs-Value sensitivity. Not primary baselines but important for the design space exploration in Table 2.
- HF Dynamic FP16 / HF Static FP16 / HF KIVI INT4: System baselines for throughput evaluation. HF Dynamic FP16 is the default HuggingFace Transformers KV cache (dynamically growing). HF Static FP16 pre-allocates the KV cache for better runtime efficiency. HF KIVI INT4 is HuggingFace's re-implementation of KIVI with 4-bit quantization.
- The paper does not compare against KVQuant (Hooper et al., 2024), MiniKV (Sharma et al., 2024), KVTuner (Li et al., 2025), or QuaRot (Ashkboos et al., 2024) in the main accuracy tables. KVQuant is discussed in Section 2.1 but not benchmarked; the paper cites hardware inefficiency as a practical limitation that makes direct comparison less relevant.
Generation budget / compute accounting. For accuracy evaluation, the "budget" is implicit in the bitwidth configuration—all methods operate under the same sequence lengths and token generation limits. The comparison is not FLOPs-matched (as in training-inference tradeoff papers) but rather accuracy-at-same-memory: given that Kitty uses ~2.25 bits per Key element and 2 bits per Value element (vs. FP16's 16 bits), the claim is that Kitty achieves near-FP16 accuracy while using ~7–8× less KV cache memory. For system throughput evaluation, the comparison is throughput-at-same-memory-budget: the paper increases batch size under each configuration until the GPU runs out of memory, measuring peak throughput for the maximum batch size each method can support.
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the reference example paper which used two-fold CV). The Kitty quantization scheme uses fixed hyperparameters (S = 32, G = 128, R = 128, α = 0.125 or 0.25) chosen from empirical sweeps, but there is no held-out validation set or CV procedure to prevent overfitting these hyperparameters to the test sets. Each accuracy experiment is repeated 3–10 times with stochastic sampling (temperature 0.6, top-p 0.95, top-k 20), and the paper reports averages with maximum observed deviations. However, these deviations are not shown in Tables 3 and 4 (only in the text description of the setup), making it difficult to assess statistical significance of the reported differences between methods.

Main Quantitative Results

The paper's empirical evaluation is organized around three claims: (1) uniform 2-bit quantization substantially degrades accuracy, (2) channel-wise precision boost recovers most of that accuracy, and (3) the reduced memory footprint translates to real throughput gains in an end-to-end inference system. The results are presented in Tables 1–4, Figures 4–5, with extended long-context results in Table 4.

The 2-Bit Failure Mode: KIVI-K2V2 vs. FP16 (Tables 1–3)

The paper's motivating result is the catastrophic accuracy degradation caused by uniform 2-bit KV cache quantization. Table 1 (Section 2.2) establishes the basic pattern on Qwen3-8B and LLaMA3-8B: while 4-bit KIVI (KIVI-K4V4) maintains accuracy within ~1 point of FP16 across all benchmarks, 2-bit KIVI (KIVI-K2V2) causes severe drops. The most dramatic case is Qwen3-8B on MATH-Algebra: 88.26 (FP16) → 47.29 (KIVI-K2V2), a collapse of 40.97 points. GSM8K drops from 94.79 to 89.13 (5.66 points). GPQA-Diamond drops from 40.71 to 32.24 (8.47 points). The average drop across four benchmarks in Table 1 is -15.76 for Qwen3-8B and -10.15 for LLaMA3-8B.

Table 3 (Section 5.2) expands this to all five model configurations (Qwen3-8B, -14B, -32B; LLaMA3.1-8B, LLaMA3.3-70B) with all benchmarks, confirming the pattern is not model-specific. For Qwen3-14B, the average drop from FP16 to KIVI-K2V2 is -8.61 (79.79 → 71.18). For Qwen3-32B, it is -8.63 (77.84 → 69.21). For LLaMA3.1-8B-Instruct, it is -10.15 (53.70 → 43.55). For LLaMA3.3-70B-Instruct, the drop is smaller at -2.86 (73.89 → 71.03), suggesting larger models may be somewhat more robust to KV cache quantization noise—though this could also reflect the 70B model's higher baseline accuracy leaving less room to degrade on some benchmarks. The degradation is consistently worst on mathematical reasoning tasks (MATH-Algebra, GSM8K) and AIME, which require precise multi-step computation where attention errors compound.

Sink Token Preservation Provides Partial Recovery (Table 2, Table 3: KIVI-K2V2*)

Preserving the first 32 tokens in FP16 (KIVI-K2V2*) substantially narrows the gap but does not close it. In Table 3, comparing KIVI-K2V2 to KIVI-K2V2*:

Qwen3-8B: Average accuracy improves from 61.39 to 69.80 (+8.41). MATH-Algebra recovers from 47.29 to 74.92 (+27.63 points), but still trails FP16's 88.26 by 13.34 points. GPQA recovers more modestly from 32.24 to 36.02 (+3.78, still 4.69 below FP16).
Qwen3-14B: Average improves from 71.18 to 76.13 (+4.95). GSM8K shows the largest gain: 75.82 → 89.56 (+13.74).
LLaMA3.1-8B: Average improves from 43.55 to 48.89 (+5.34). MATH-Algebra recovers from 31.45 to 44.12 (+12.67, but still 3.03 below FP16's 47.15).
LLaMA3.3-70B: Average improves from 71.03 to 73.56 (+2.53). The 70B model's smaller initial gap means sink token preservation nearly closes it—the remaining difference vs. FP16 is only -0.33 on average.

The consistent message is that sink token preservation is necessary but insufficient. It recovers a substantial fraction of the accuracy loss (roughly 50–70% of the gap, depending on model and benchmark) but leaves a residual degradation, especially on the most quantization-sensitive tasks like MATH-Algebra. This residual is what channel-wise precision boost targets.

Channel-Wise Precision Boost Closes the Gap (Table 3: Kitty and Kitty-Pro)

The headline result of the paper is that boosting only a small fraction of Key-cache channels to INT4 recovers most or all of the remaining accuracy gap. Table 3 shows:

Kitty (12.5% channels boosted to INT4):

Qwen3-8B: Average accuracy reaches 74.97, compared to 69.80 for KIVI-K2V2* (+5.17) and 77.15 for FP16 (-2.18 gap). MATH-Algebra reaches 85.12 (vs. 74.92 for KIVI-K2V2* and 88.26 for FP16). GSM8K reaches 93.61 (vs. 89.71 for KIVI-K2V2* and 94.79 for FP16). The gap to FP16 on average is -2.18, down from -7.35 for KIVI-K2V2*.
Qwen3-14B: Kitty actually exceeds FP16 on average: 80.08 vs. 79.79 (+0.29). This is not a systematic improvement—it's within noise—but it demonstrates that the quantization is no longer the limiting factor for accuracy.
Qwen3-32B: Average accuracy reaches 77.27, a gap of only -0.57 vs. FP16's 77.84.
LLaMA3.1-8B: Average reaches 51.84, gap of -1.86 vs. FP16's 53.70.
LLaMA3.3-70B: Average reaches 73.53, gap of -0.36 vs. FP16's 73.89.

Kitty-Pro (25% channels boosted to INT4):

Qwen3-8B: Average reaches 76.18, gap of only -0.97 vs. FP16. MATH-Algebra reaches 88.12 (vs. 88.26 FP16—essentially identical). GSM8K reaches 94.34 (vs. 94.79 FP16). GPQA reaches 40.92 (vs. 40.71 FP16—actually slightly higher, within noise).
Qwen3-14B: Not reported for Kitty-Pro in Table 3 (only Kitty is shown for the 14B and 32B models).
LLaMA3.1-8B: Average reaches 52.59, gap of -1.11 vs. FP16.
LLaMA3.3-70B: Average reaches 73.86, gap of only -0.03 vs. FP16's 73.89—essentially at parity.

The progression from KIVI-K2V2 → KIVI-K2V2* → Kitty → Kitty-Pro shows a clear monotonic recovery of accuracy as precision investments are added: sink preservation recovers ~50–70% of the gap, channel-wise boost at 12.5% recovers an additional ~20–30%, and boosting to 25% recovers nearly all of the remainder. The diminishing returns from 12.5% to 25% boost (Qwen3-8B: -2.18 → -0.97, a gain of only 1.21 points for doubling the boosted channels) suggest that most of the benefit is captured by protecting the most critical 12.5% of channels.

Long-Context Robustness on AIME (Table 4)

The extended evaluation on AIME24 and AIME25 with a maximum generation length of 32,768 tokens (Table 4) tests whether the accuracy benefits hold under longer contexts—a critical test because quantization errors can compound across more attention steps. The results show that Kitty maintains its advantage:

Qwen3-8B: KIVI-K2V2 achieves 57.00 on AIME24 and 52.33 on AIME25 (average 54.67), compared to FP16's 71.67 and 66.00 (average 68.84), a gap of -14.17. KIVI-K2V2* recovers to 67.67 and 57.67 (average 62.67), gap of -6.17. Kitty further recovers to 70.67 and 59.67 (average 65.17), gap of -3.67 vs. FP16.
Qwen3-14B: The gap progression is -12.83 (KIVI-K2V2) → -7.00 (KIVI-K2V2*) → -3.50 (Kitty).
Qwen3-32B: The gap progression is -11.92 → -8.10 → -2.66.

The key observation is that the benefit of channel-wise precision boost does not diminish at longer context lengths—if anything, the relative advantage of Kitty over KIVI-K2V2* is slightly larger at 32K contexts than at 4K contexts (comparing the gap reductions). This is important because it suggests the channel sensitivity patterns are stable across sequence positions and the magnitude heuristic remains effective even as the sequence grows.

However, notable is that at 32K contexts, even Kitty does not fully close the gap to FP16 for Qwen3-8B (gap of -3.67, vs. -2.18 at 4K contexts on the main benchmarks). The 8B model appears more fragile under extended generation, possibly because quantization errors in early keys propagate through more attention steps, or because the AIME tasks are inherently harder (requiring more precise reasoning) than the MATH-Algebra subset used in Table 3.

Ablation: Boost Rate vs. Accuracy (Figure 4)

The ablation study in Figure 4 sweeps the fraction of channels boosted from 0% (pure KIVI-K2V2* equivalent) to 100% (all channels at INT4) on Qwen3-8B for GSM8K and MATH-Algebra. The findings:

Monotonic improvement with boost rate. Accuracy on GSM8K increases from ~89.7% at 0% boost to ~94.3% at 25% boost, with diminishing returns thereafter—100% boost only reaches ~95%, barely above 25%. On MATH-Algebra, the curve climbs from ~75% at 0% boost to ~88% at 25% boost, again flattening beyond 25%.
Magnitude-based selection substantially outperforms random selection. At 12.5% boost on GSM8K, magnitude-based selection achieves roughly 93.6% vs. ~91.5% for random selection—a gap of ~2 percentage points. On MATH-Algebra at 12.5%, the gap is larger: ~85% vs. ~80.5%, a ~4.5 point advantage. This confirms that the magnitude heuristic captures genuine channel sensitivity beyond what random chance provides.
The curves converge at high boost rates: when 50%+ of channels are boosted, random and magnitude-based selection converge because there's less discrimination (both select mostly the same channels). The fact that they converge suggests the important channels are indeed the high-magnitude ones—random selection eventually includes them by chance.

A limitation: Figure 4 only shows GSM8K and MATH-Algebra, not the full benchmark suite. The paper states "similar trends are observed on other tasks" but does not provide the data.

System Throughput Results (Figure 5)

The system evaluation on Qwen3-8B generating 8,192 tokens on a single NVIDIA A100 (80GB) demonstrates the translation of memory savings into throughput:

GPU Memory Usage (Figure 5a):

HF Dynamic FP16 runs out of memory at batch size 32 (peak ~72 GB). The dynamically growing KV cache is the least efficient, as it incurs fragmentation and allocation overhead.
HF Static FP16 supports up to batch size 32 (peak ~70 GB), slightly better than dynamic due to pre-allocation avoiding fragmentation, but still limited by the FP16 KV cache size.
HF KIVI INT4 (4-bit quantization) supports up to batch size 128 (peak ~67 GB). The 4× compression over FP16 directly enables 4× larger batches.
Kitty-Pro supports up to batch size 256 (peak ~70 GB). This is 8× the FP16 baseline's batch size, matching the theoretical ~8× memory compression (~2.125 bits per element vs. 16 bits for FP16).

The memory curves show that Kitty-Pro's memory usage grows much more slowly with batch size than the FP16 baselines—the slope is roughly 1/8th as steep, consistent with the compression ratio.

Inference Throughput (Figure 5b):

At the maximum batch size each method supports, the throughput (tokens/s) is:
- HF Dynamic FP16: ~310 tokens/s at batch size 32
- HF Static FP16: ~350 tokens/s at batch size 32
- HF KIVI INT4: ~750 tokens/s at batch size 128
- Kitty-Pro: ~1,300 tokens/s at batch size 256

The throughput improvement of Kitty-Pro over the strongest FP16 baseline (HF Static FP16) at the same memory budget is approximately 1,300 / 350 ≈ 3.7×. The paper claims "2.1× → 4.1× higher inference throughput compared to the FP16 baseline"—the range likely reflects different batch size configurations where the FP16 baseline uses fewer than its maximum batch size (giving a lower baseline throughput and thus a higher ratio). The 2.1× lower bound corresponds to a scenario where the FP16 baseline is not memory-saturated.

Comparing Kitty-Pro to HF KIVI INT4 at their respective maximum batch sizes: 1,300 vs. 750 tokens/s, a 1.73× improvement. This is the throughput advantage of 2-bit over 4-bit—a substantial but not 2× gain, because the attention computation (which is memory-bandwidth-bound) benefits from the reduced data movement volume but other components of inference (the model weights, the MLP layers, the softmax) are unaffected by KV cache compression.

An important caveat on the system results: The paper implements the dequantization and attention kernels in Triton as a proof-of-concept. The authors explicitly note that "the inference efficiency of our inference engine can be further improved if we use more low-level programming language, e.g., CUDA, and enable more fine-grained optimizations." This means the throughput numbers are lower bounds on what an optimized CUDA implementation could achieve. However, it also means the relative comparison may be affected by Triton overhead—if Triton is less efficient than CUDA for the FP16 baseline kernels, the speedup ratios could be overstated. The paper does not provide a CUDA-optimized FP16 baseline to control for this.

Ablation Studies and Robustness Checks

Channel selection heuristic (Figure 4): Magnitude-based selection consistently outperforms random selection across all boost rates on both GSM8K and MATH-Algebra. At 12.5% boost (Kitty), the margin is ~2 points on GSM8K and ~4.5 points on MATH-Algebra. At 25% boost (Kitty-Pro), the margin narrows to ~1 and ~2 points respectively. This confirms the heuristic captures genuine sensitivity information. The monotonic accuracy improvement with boost rate for the magnitude-based curve (no dips or plateaus below 25%) suggests the ranking is correctly ordered—channels selected earlier genuinely contribute more to accuracy. The convergence of magnitude-based and random curves at high boost rates (>50%) validates that the "important" channels are a subset of the high-magnitude channels, not an entirely disjoint set.

Key vs. Value precision asymmetry (Table 2): The comparison of KIVI-K2V4* (2-bit Key, 4-bit Value) vs. KIVI-K4V2* (4-bit Key, 2-bit Value) is a non-obvious and important ablation. On Qwen3-8B MATH-Algebra, K4V2* achieves 87.92 vs. 82.50 for K2V4* (a 5.42-point advantage). On GSM8K, the gap is narrower: 93.96 vs. 90.14. Across all benchmarks, K4V2* approaches FP16 while K2V4* consistently lags. This ablation directly motivates the decision to apply channel-wise precision boost only to the Key cache. The paper does not ablate why the Value cache is less sensitive—the softmax vs. linear argument is offered as a hypothesis but not experimentally tested (e.g., by analyzing attention weight distortion under Key vs. Value quantization).

Boost rate sweep (Figure 4): Already discussed above. The key finding is that diminishing returns set in sharply after 25% boost—accuracy on GSM8K at 100% boost (~95.0%) is only ~0.7 points higher than at 25% (~94.3%). This justifies the design choice of 12.5–25% as the practical operating range.

Sink token count: The paper fixes the number of preserved sink tokens at 32 for all experiments. There is no ablation varying the sink token count (e.g., 0, 16, 64, 128) to determine the optimal value or to show that 32 is sufficient. Given that sink preservation provides a substantial fraction of the accuracy recovery (comparing KIVI-K2V2 to KIVI-K2V2*), the sensitivity to this parameter matters for practical deployment—more sink tokens means more FP16 memory, reducing the effective compression ratio.

Quantization group size (G): The group size is fixed at 128 tokens throughout. There is no ablation on larger or smaller groups. A larger group would amortize quantization overhead better but would reduce the "dynamism" of the channel importance calculation and could increase quantization error (since per-channel scales must cover a wider range). A smaller group would be more adaptive but would increase metadata overhead and quantization launch frequency.

Local window size for Value cache (R): Fixed at 128 tokens. No ablation is provided. The choice of 128 matching the quantization group size G is convenient (one page worth of tokens) but whether it's optimal is not tested.

Cross-model and cross-scale generalization: The results in Table 3 span Qwen3 (8B, 14B, 32B) and LLaMA3 (8B, 70B). Kitty and Kitty-Pro consistently outperform KIVI-K2V2* across all model families and scales, with the accuracy gap to FP16 progressively narrowing. The 70B LLaMA3 model shows smaller absolute gaps throughout (even KIVI-K2V2 only loses -2.86 on average), suggesting that larger models are more robust to KV cache quantization noise. The paper hypothesizes that larger models may have more redundant representations in the Key cache, making individual channel perturbations less impactful. However, this is speculative—no analysis of channel sensitivity patterns across model scales is provided.

Extended context length (Table 4): The AIME evaluation at 32K tokens serves as a robustness check against the concern that quantization errors compound over longer sequences. Kitty maintains a consistent ~2–4 point advantage over KIVI-K2V2* at 32K contexts across all three Qwen3 model sizes, confirming that channel-wise precision boost does not degrade with sequence length. However, the residual gap to FP16 is larger at 32K (Qwen3-8B: -3.67 at 32K vs. -2.18 at 4K on the main benchmarks), suggesting that longer contexts may require higher boost rates to fully close the gap. This interaction is not explored.

Missing negative result: The paper does not report a failure case where channel-wise precision boost fails to help—for instance, on a benchmark where the magnitude heuristic selects the "wrong" channels, or on a model where channel sensitivity patterns are uniform (all channels equally sensitive), making the boost inefficient. The uniform improvement across all benchmarks and models suggests the phenomenon (some channels being more important than others) is robust, but the absence of negative results limits confidence in the generality boundary.

Critical Assessment

Claim 1: "Kitty can significantly outperform prior works in accuracy" (contribution bullet 2)

This claim is well-supported for the specific comparison to KIVI-K2V2 and its sink-preserving variant. Tables 3 and 4 show consistent, substantial accuracy improvements of Kitty and Kitty-Pro over KIVI-K2V2* across five model configurations and seven benchmarks. The improvement over KIVI-K2V2* ranges from +5.17 (Qwen3-8B average) to +4.95 (Qwen3-14B average), which is practically meaningful given that the baseline gap to FP16 is only ~3–7 points.

However, "prior works" is a broad claim that the experiments only partially validate. The paper compares against exactly one prior method: KIVI (Liu et al., 2024). KVQuant (Hooper et al., 2024), MiniKV (Sharma et al., 2024), KVTuner (Li et al., 2025), and QuaRot (Ashkboos et al., 2024) are discussed in Section 2 but not included in the experimental comparison. For KVQuant specifically, the paper argues it is "not hardware-friendly" and suffers from low system efficiency, which is a valid concern but does not excuse omitting it from the accuracy evaluation—a system-inefficient method could still set the accuracy ceiling that Kitty must match. For MiniKV and KVTuner (layer-wise mixed precision), these are orthogonal to Kitty's channel-wise approach and could potentially be combined, but the paper doesn't establish whether layer-wise allocation alone matches Kitty's accuracy. The claim "significantly outperform prior works" should therefore be qualified: Kitty significantly outperforms KIVI, the most directly comparable prior work. Comparisons to other prior methods remain to be done.

Claim 2: "Kitty achieves near-zero loss in accuracy drop while approaching 2-bit memory" (abstract)

This claim is supported with qualifications. For Kitty-Pro (25% boost), the accuracy gap to FP16 is indeed near-zero: -0.97 on Qwen3-8B average, -0.03 on LLaMA3.3-70B. For Kitty (12.5% boost), the gap is slightly larger (-2.18 on Qwen3-8B) but still dramatically smaller than KIVI-K2V2 (-15.76). The memory savings are ~7.1× over FP16 for Kitty and ~6.4× for Kitty-Pro, compared to the theoretical maximum of 8× for pure 2-bit.

The qualifications:

"Near-zero" holds more strongly for larger models (70B) and less strongly for smaller models on harder tasks (Qwen3-8B on AIME at 32K context: -3.67 gap for Kitty). If "near-zero" means "within 1 point on average," then Kitty-Pro achieves this for three of the five model configurations in Table 3, and Kitty achieves it for none (the smallest gap is -0.57 for Qwen3-32B, but that's borderline).
The memory claim of "approaching 2-bit" depends on what "approaching" means. At 12.5% boost, the average bitwidth for the Key cache is 2.25 bits and for the KV cache overall is ~2.125 bits. This is closer to 2 bits than to 4 bits, so "approaching" is reasonable. But the paper's own evaluation shows that 4-bit quantization (HF KIVI INT4) already achieves near-FP16 accuracy without any channel-wise boost (Tables 1–2), so the "2-bit" framing is crucial—the advance is in making 2-bit work, not in exceeding 4-bit accuracy.
The claim is validated on reasoning benchmarks at up to 32K context. For much longer contexts (128K+), the compounding of quantization errors might require higher boost rates, reducing the effective compression. The paper does not test this regime.

Claim 3: "Kitty enables up to 8× larger batches and 2.1×–4.1× higher throughput under the same memory budget" (abstract, contribution bullet 3)

This claim is supported by the system evaluation in Figure 5 for Qwen3-8B on a single A100 with 8,192-token generation. The 8× larger batch size (256 vs. 32) directly follows from the ~8× memory compression. The 2.1×–4.1× throughput range is somewhat loosely bounded: at maximum batch sizes, Kitty-Pro achieves ~3.7× higher throughput than the best FP16 baseline (HF Static FP16). The 2.1× lower bound would correspond to a scenario where the FP16 baseline is operating at a batch size giving higher baseline throughput (perhaps batch size 16 or 24, where utilization is better). The 4.1× upper bound is not directly visible in Figure 5 but could correspond to a comparison with HF Dynamic FP16 at a suboptimal batch size.

Critical limitations of the system evaluation:

Single model, single GPU, single sequence length. The throughput results are specific to Qwen3-8B on an A100 at 8,192 tokens. The paper does not report system results for the 14B, 32B, or 70B models, or for different GPU architectures (H100, B200), or for different sequence lengths. The 8× batch size scaling may not hold on GPUs with less memory (where the baseline saturates at smaller batch sizes, compressing the dynamic range) or more memory (where the baseline could support larger batches before Kitty's compression becomes the binding constraint).
Triton vs. CUDA confound. The Kitty kernels are written in Triton, which typically underperforms hand-optimized CUDA. The FP16 baselines use HuggingFace's default implementation, which uses FlashAttention (CUDA-optimized). If Triton introduces overhead, the throughput comparison mixes the effect of quantization with the effect of kernel implementation quality. A fairer comparison would implement the FP16 attention kernel in the same Triton framework.
No latency analysis. The paper reports throughput (tokens/s) but not per-token latency or time-to-first-token. The quantization overhead (Stage 3 in the execution pipeline) is amortized across 128 tokens but still introduces periodic latency spikes that could matter for interactive applications. The paper does not characterize these spikes.
No comparison to 4-bit KV cache systems. The system baseline HF KIVI INT4 represents 4-bit quantization. Kitty-Pro achieves ~1.73× higher throughput than this baseline at their respective maximum batch sizes. This is the throughput gain specifically attributable to moving from 4-bit to 2-bit with channel-wise boost—a meaningful but less dramatic improvement than the 2.1×–4.1× claimed against FP16.

Claim 4: "Channel-wise precision boost maintains near-zero loss in accuracy drop" specifically attributed to ranking by sensitivity (Figure 2b) and selecting only critical channels

This is the paper's central mechanistic claim: that the accuracy recovery comes specifically from protecting sensitive channels, not just from spending more bits on any channels. The magnitude-vs-random ablation in Figure 4 supports this but incompletely. Magnitude-based selection clearly outperforms random, validating that the selection criterion matters. However, the paper does not compare against the "oracle" selection (choosing channels based on the per-channel MSE analysis from Figure 2b) to establish how close magnitude-based selection comes to the theoretical optimum. If magnitude-based selection captures, say, 80% of the benefit of oracle selection, that would be a strong result; if it captures only 30%, then there is substantial room for improvement with better heuristics.

More fundamentally, the paper does not test the causal claim that "these specific channels cause the accuracy degradation." The per-channel MSE analysis (Figure 2b) shows correlation between channel identity and attention distortion, but this is an observational measurement on a single model—it does not demonstrate that protecting different channels would change accuracy outcomes. The magnitude-vs-random comparison is a step toward establishing causality (different selection criteria produce different outcomes), but a stronger test would compare magnitude-based selection against deliberate mis-selection (e.g., boosting the least sensitive channels) to show that the effect is not just "more bits anywhere helps" but rather "bits on the right channels are what matters."

Missing Experiments That Would Strengthen the Paper

Comparison to KVQuant, MiniKV, KVTuner, QuaRot on accuracy. Even if these methods have system-level limitations, establishing the accuracy frontier (how close can any method get to FP16 at ~2-bit KV cache?) would contextualize Kitty's results.
Ablation on sink token count (S) and local window size (R). These parameters contribute substantially to accuracy recovery (Table 2: KIVI-K2V2 vs. KIVI-K2V2*) and directly affect the effective compression ratio. The paper fixes them at S=32, R=128 without demonstrating optimality, sensitivity, or the accuracy-memory tradeoff curve.
Per-layer analysis of channel sensitivity. The paper shows channel-wise patterns for Layer 10 of Qwen3-8B (Figure 2) but does not analyze whether the same channels are important across all layers, whether sensitivity varies with layer depth, or whether boost rates could be allocated differently per layer (combining channel-wise and layer-wise mixed precision). This is a natural extension that the paper explicitly leaves to future work.
System results for larger models and longer contexts. The throughput evaluation on a single 8B model at 8K context demonstrates the principle but does not validate the scalability claims for the 70B model or for 32K–128K contexts where the KV cache dominates memory even more. The accuracy evaluation includes 70B and 32K contexts, but the system evaluation does not match.
Analysis of how the boosted channel set varies across quantization groups. The paper claims the method is "dynamic" because importance scores are recomputed per group, but it does not measure how much the boosted channel set actually changes between groups. If it's 99% static, then "dynamic" is mostly marketing; if it fluctuates significantly, then the adaptivity might be providing robustness that a static mask would not. Quantifying this would strengthen the "dynamic" claim.
Wall-clock latency breakdown between quantization time, dequantization time, and attention computation time. The throughput numbers aggregate everything, but understanding where the time goes would help practitioners decide whether the Triton kernel overhead is acceptable or whether CUDA optimization is urgent.
Evaluation on non-reasoning benchmarks. The paper focuses exclusively on reasoning tasks (math, code, science QA). Performance on other task types—long-context document QA, summarization, multi-turn dialogue—would establish whether the channel sensitivity patterns are reasoning-specific or general properties of the Key cache. If the method only works for reasoning, its applicability is narrower than the paper implies.

Summary of Experimental Rigor

The accuracy evaluation is comprehensive in model coverage (five configurations across two families, 8B–70B), adequate in benchmark coverage (seven diverse reasoning tasks), and convincing in its within-paper comparisons (Kitty vs. KIVI variants show clear, consistent improvements). The system evaluation is more limited—single model, single GPU, single sequence length, Triton implementation—but still demonstrates the key point that memory reduction translates to throughput gains.

The primary experimental weaknesses are: (1) no comparison to non-KIVI prior work (KVQuant, MiniKV, KVTuner, QuaRot) to establish where Kitty sits on the accuracy frontier, (2) limited ablation on key hyperparameters (sink count, group size, local window) that affect the accuracy-memory tradeoff, (3) no oracle channel selection comparison to bound how much better the selection heuristic could be, (4) system evaluation restricted to a single configuration that doesn't validate the scalability claims, and (5) no analysis of per-layer sensitivity patterns despite the paper's motivating visualization suggesting layer-level structure matters. These gaps do not undermine the paper's core contributions—2-bit KV cache quantization can be made accurate through channel-wise precision boost, and this can be implemented efficiently—but they leave open questions about how the method generalizes, how close it is to optimal, and what the practical deployment profile looks like across the full range of models and contexts where it would be used.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The channel-wise precision boost mechanism requires computing per-channel importance scores for each quantization group of G = 128 tokens at runtime (Equation 2, Section 3.2). The paper states this computation uses a magnitude-based heuristic—averaging the absolute activations of each channel across the group's tokens—which is lightweight relative to the per-channel MSE analysis used for diagnosis (Figure 2b). However, the system evaluation in Section 5.5 does not isolate or report the runtime cost of this importance computation, quantization parameter fitting (per-channel scales and zero-points for D channels), or the dense-sparse decomposition and packing into the page format. The paper asserts that "the quantization overhead is efficiently amortized and becomes negligible" because quantization launches "at most once every G decoding steps" (Section 4.3), but no wall-clock breakdown is provided to validate this claim.

The consequence. A practitioner deploying Kitty needs to understand whether the per-group quantization cost introduces latency spikes at page boundaries (every 128 tokens). Even if amortized over 128 decoding steps, a quantization kernel that takes, say, 5 milliseconds would add ~39 microseconds per token on average—potentially negligible for throughput but problematic for latency-sensitive interactive applications where tail latency matters. The paper's throughput curves (Figure 5) aggregate over the entire generation, smoothing any periodic latency spikes. If the quantization step introduces a noticeable pause every 128 tokens, this could manifest as jitter in the token generation rate, which degrades user experience in streaming applications. The paper does not characterize this.

What evidence exists in the paper. Figure 5b reports aggregate throughput at different batch sizes, with no per-step latency breakdown. The execution pipeline (Section 4.3) describes three stages—insertion, attention computation, and (optional) quantization—but provides no timing measurements for each stage. The paper does not report the FLOP count or memory access volume of the quantization kernel relative to the attention computation it accompanies. The claim that overhead is "negligible" is an assertion, not a measurement.

Mitigation status. The paper does not address this. It acknowledges that the Triton implementation is a proof-of-concept and that CUDA-level optimization is future work (Section 5.5), which could reduce quantization kernel overhead. However, the per-group importance scoring fundamentally requires reading all D × G Key activations to compute channel-wise averages, performing a top-K selection, fitting quantization parameters, and packing the results—this is a non-trivial amount of work that scales with the Key cache dimension. A proper mitigation would require either: (a) measuring and reporting the quantization overhead explicitly, or (b) developing a cheaper importance heuristic that avoids reading the full activation tensor (e.g., using a running estimate of channel magnitudes, or amortizing the importance computation across multiple groups by reusing the same boost mask for a window of pages). Neither is explored.

The Method Is Evaluated on a Single Model Family's Channel Sensitivity Patterns Without Evidence of Cross-Architecture Generality

The assumption or constraint. The paper's central mechanistic claim—that certain Key-cache channels are disproportionately sensitive to quantization, and that magnitude-based selection identifies them—is validated empirically on two model families: Qwen3 and LLaMA3. Both families use Grouped-Query Attention (GQA, Ainslie et al., 2023) with similar architectural patterns (transformer-based autoregressive LLMs). The channel sensitivity analysis (Figure 2) is shown for a single layer of a single model (Layer 10 of Qwen3-8B), with the paper stating "similar patterns are observed on other layers/models" without quantification. The paper does not evaluate models with Multi-Head Attention (MHA, where each query head has its own key head), Multi-Query Attention (MQA, where one key-value head is shared across all query heads), or non-transformer architectures. It also does not evaluate encoder-decoder models, mixture-of-experts architectures, or models trained with substantially different objectives (e.g., embedding models, reward models).

The consequence. The channel-wise sensitivity pattern—that a small fraction of Key-cache channels dominates attention-score MSE—may be specific to how GQA models learn to distribute information across channels. In MHA, where each query head has its own dedicated key head with no sharing, the sensitivity patterns could be qualitatively different: there might be less concentration of importance because each key head is specialized to its query head rather than serving multiple queries. In MQA, where a single key head serves all queries, the channel sensitivity might be even more concentrated (since the key must encode information relevant to all queries) or less concentrated (if the model learns to spread information more uniformly). A practitioner deploying Kitty on a non-GQA architecture has no evidence that the 12.5–25% boost rate that works for Qwen3/LLaMA3 will transfer.

What evidence exists in the paper. The paper evaluates Qwen3 (8B, 14B, 32B) and LLaMA3 (8B, 70B), all GQA models. The per-layer channel sensitivity visualization is shown for exactly one layer (Figure 2a), with the qualitative statement that "similar patterns are observed on other layers/models." There is no quantification of how the sensitivity distribution varies across layers (early vs. late), across model scales (8B vs. 70B), or across model families (Qwen3 vs. LLaMA3). The boost rate sweep (Figure 4) is performed only on Qwen3-8B, and the paper states "similar trends are observed on other tasks" without showing the data. The accuracy results in Table 3 use the same 12.5%/25% boost rates across all five model configurations—the paper does not test whether different models would benefit from different boost rates (e.g., whether the 70B model could use a lower boost rate and still achieve parity).

Mitigation status. The paper does not address this directly. The authors position this as a general property of Key caches ("not all channels in the Key cache behave equally," Section 3.2), but the evidence base is narrow. Assessing generality would require: (a) quantifying how channel sensitivity distributions vary across architectures on a shared benchmark, (b) testing whether the optimal boost rate differs across model families (or even across layers within a model), and (c) evaluating on at least one non-GQA model to establish the boundaries of the phenomenon. The paper's explicit future work on "more principled or adaptive strategies" for channel selection (Section 3.2) does not address the architecture generality question.

The System Evaluation Is Restricted to a Single Model on a Single GPU at a Single Sequence Length

The assumption or constraint. The throughput evaluation in Section 5.5 demonstrates Kitty-Pro's system performance exclusively on Qwen3-8B running on a single NVIDIA A100 (80GB) GPU, generating 8,192 tokens. This is a limited configuration relative to the accuracy evaluation, which covers models up to 70B parameters and context lengths up to 32,768 tokens. The paper states that "the inference efficiency of our inference engine can be further improved if we use more low-level programming language, e.g., CUDA" (Section 5.5), but does not characterize how the system behavior scales to the larger models and longer contexts where KV cache compression matters most. The paper provides no system results for the 14B, 32B, or 70B model configurations, nor for the A100's successor (H100, which the paper uses for accuracy evaluation of larger models), nor for context lengths other than 8,192.

The consequence. The throughput claims of "2.1×–4.1× higher inference throughput" and "8× larger batch sizes" are validated for exactly one operational point. A practitioner considering Kitty for a production deployment—which likely involves larger models (70B+), longer contexts (32K–128K), or newer hardware (H100/H200)—has no experimental evidence that the claimed throughput scaling holds. At longer contexts, the relative contribution of the KV cache to total memory and attention time increases, which could make Kitty's benefits larger (since more of the workload is KV-cache-bound). Conversely, at larger model sizes, the weight memory and MLP computation may dominate, reducing the relative benefit of KV cache compression. On H100 GPUs with higher memory bandwidth (3.35 TB/s vs. A100's 2.0 TB/s) and larger memory capacity (80GB HBM3 vs. A100's 80GB HBM2e), the memory bandwidth bottleneck that Kitty alleviates may shift, changing the throughput gains. The paper provides no guidance on how to extrapolate the A100/8B/8K results to other configurations.

What evidence exists in the paper. Figure 5 reports memory usage and throughput exclusively for Qwen3-8B at 8,192 tokens on an A100. The paper does report accuracy for larger models (Table 3) and longer contexts (Table 4), establishing that the quantization scheme preserves accuracy in these regimes, but does not pair this with system measurements. The claim that larger contexts amplify the benefit of KV cache compression is implicit in the motivation (Section 1: "the size of the KV cache grows proportionally with both context length and batch size") but untested in the system evaluation.

Mitigation status. The paper partially acknowledges the limitation by noting that the Triton kernels are a proof-of-concept and that CUDA optimization is future work. However, it does not acknowledge the missing scaling experiments as a gap—the throughput claims are presented without qualification about their scope. The future work mention of "more fine-grained optimizations" (Section 5.5) refers to kernel-level improvements, not to extending the system evaluation to larger models, longer contexts, or different hardware. A proper mitigation would require at minimum: (a) throughput results for at least one larger model (e.g., Qwen3-32B or LLaMA3-70B) to validate scaling, (b) results at multiple context lengths (e.g., 4K and 32K) to characterize the throughput-vs-context-length curve, and (c) results on the H100 (which the paper already uses for accuracy evaluation) to demonstrate hardware generality.

Comparison to Prior Art Is Restricted to KIVI Only; the Accuracy Frontier for 2-Bit KV Quantization Remains Unestablished

The assumption or constraint. The paper's experimental comparison in the accuracy evaluation (Tables 2–4) benchmarks Kitty exclusively against variants of KIVI (Liu et al., 2024)—KIVI-K2V2, KIVI-K2V2*, KIVI-K2V4*, and KIVI-K4V2*. The paper discusses several other methods in the background section (Section 2.1): KVQuant (Hooper et al., 2024), which uses sparse FP16 outlier preservation; MiniKV (Sharma et al., 2024), which uses layer-discriminative bit allocation; KVTuner (Li et al., 2025), which tunes layer-wise mixed precision bitwidths; and QuaRot (Ashkboos et al., 2024), which applies orthogonal rotations. None of these are included in the experimental comparison. The paper argues that KVQuant "is not hardware-friendly" (Section 2.1) and that MiniKV and KVTuner's layer-wise approaches are "orthogonal and potentially complementary" (Section 2.1). This implies that the accuracy comparison is intentionally limited to methods that share Kitty's design space (post-training, per-channel Key quantization, per-token Value quantization), but the paper does not state this scope limitation explicitly.

The consequence. A practitioner choosing a 2-bit KV cache quantization method needs to know how Kitty's accuracy compares to the best available alternative—not just to the method it extends. KVQuant's sparse outlier preservation is conceptually similar to Kitty's channel-wise boost (both protect the most sensitive elements at higher precision), and layer-wise allocation (MiniKV, KVTuner) could potentially achieve comparable accuracy by giving entire sensitive layers 4-bit precision while quantizing others more aggressively. Without experimental comparison, it is unknown whether Kitty's 12.5% channel-wise boost outperforms, matches, or underperforms these alternatives on accuracy. The paper's accuracy claims ("significantly outperform prior works," contribution bullet 2; "state-of-the-art 2-bit KV cache quantization," Section 1) implicitly position Kitty at the accuracy frontier, but this positioning is unvalidated against the methods it excludes. The memory savings comparison is also incomplete: KVQuant's sparse format has a different memory-vs-accuracy tradeoff curve that isn't characterized.

What evidence exists in the paper. The paper provides an extensive comparison to KIVI variants across seven benchmarks and five model configurations (Tables 2–4). The background section (Section 2.1) summarizes the excluded methods and explains why each is not directly comparable (hardware inefficiency, orthogonality), but does not provide accuracy numbers for any of them on the same benchmarks and models. The discussion of KVQuant cites hardware inefficiency from sparse-dense multiplications but does not test whether this inefficiency is severe enough to justify excluding it from the accuracy evaluation—accuracy and throughput are separate axes, and a method could be accuracy-competitive while being throughput-limited.

Mitigation status. The paper does not attempt to mitigate this limitation—it does not include the missing methods in the experimental evaluation, does not cite accuracy numbers from their papers on comparable benchmarks, and does not discuss what the accuracy frontier looks like. The framing of MiniKV and KVTuner as "orthogonal" suggests they could be combined with Kitty (layer-wise + channel-wise mixed precision), but this is speculative without experiments. A proper mitigation would require at minimum: (a) implementing at least one alternative mixed-precision method (e.g., KVQuant's outlier preservation) in the same evaluation framework for a like-for-like accuracy comparison on the same models and benchmarks, or (b) citing and discussing published accuracy numbers from those methods on overlapping benchmarks, noting where differences in evaluation protocol might affect the comparison. The paper's explicit future work on "layer-wise mixed precision bitwidths" (related to MiniKV/KVTuner) partially addresses the combination question but not the competitive comparison.

The Interaction Between Sink Preservation, Local Window, and Channel-Wise Boost Is Not Characterized; Hyperparameters Are Fixed Without Tradeoff Analysis

The assumption or constraint. The Kitty quantization scheme integrates three distinct accuracy-preserving mechanisms: sink token preservation (S = 32 tokens in FP16), a local Value window (R = 128 recent tokens in FP16), and channel-wise precision boost (12.5% or 25% of Key channels at INT4). These mechanisms each contribute to accuracy recovery, but their relative contributions and interactions are not isolated in the experimental design. Comparing KIVI-K2V2 (no sink, no boost) to KIVI-K2V2* (sink only, no boost) to Kitty (sink + 12.5% boost) shows the aggregate progression, but this does not answer: what accuracy does 25% boost achieve without sink preservation? What accuracy does sink preservation achieve with a larger sink window (S = 64 or S = 128)? Could a larger sink window (costing more FP16 memory) substitute for some of the channel-wise boost (also costing memory), and if so, which combination minimizes memory for a given accuracy target?

The consequence. The hyperparameters S, R, G, and the boost fraction α are set to fixed values (32, 128, 128, 0.125 or 0.25) without any reported sweep or tradeoff analysis. The paper does not establish whether these values are optimal, robust, or on a Pareto frontier of the accuracy-memory tradeoff. A practitioner deploying Kitty may find that their accuracy target could be met with a different configuration—perhaps S = 16 with α = 0.25, or S = 64 with α = 0.125—that yields better memory savings for their specific use case. Without sensitivity analysis, the practitioner has no guidance for tuning these parameters. Moreover, the effective compression ratio depends on the sequence length: for short sequences, the fixed sink and local window overhead dominates (e.g., for a 200-token sequence, 32 sink + 128 local = 160 tokens are in FP16, leaving only 40 tokens to benefit from 2-bit quantization). The paper's ~8× compression claim applies asymptotically as sequence length grows large, but the cross-over point where quantization provides net benefit is not characterized.

What evidence exists in the paper. The paper provides one ablation sweep: the boost rate (Figure 4), showing monotonic accuracy improvement from 0% to 100% boost on Qwen3-8B for GSM8K and MATH-Algebra. No sweeps are provided for S (sink token count), R (local window size), or G (quantization group size). The default values are stated in Section 3.3 (Figure 1 caption: "The default configuration is: S = 32, R = 128, G = 128, which provides a good balance between accuracy and memory savings") without empirical justification for the claim of "good balance." The paper does not report how accuracy changes if S is reduced to 0 (pure channel-wise boost without sink preservation) or increased to 64 or 128.

Mitigation status. The paper does not address this. The fixed hyperparameters are a practical choice for evaluation but leave the method's tunability unexplored. The future work discussion (Section 8 is not included in the provided text, but the main body does not mention hyperparameter sweeps as future work) does not identify this as a gap. A proper mitigation would require: (a) a hyperparameter sensitivity analysis showing accuracy and memory as a function of S, R, and α to establish whether the defaults are near-optimal, and (b) guidance for practitioners on how to trade off these parameters based on their accuracy targets and sequence length distributions.

There Is No Evaluation on Tasks Requiring Retrieval of Specific Facts from Long Contexts; All Benchmarks Are Reasoning-Focused

The assumption or constraint. The paper's accuracy evaluation (Tables 1–4) uses seven benchmarks: GSM8K, MATH-Algebra, GPQA-Diamond, HumanEval, AIME24, and AIME25. All of these are reasoning benchmarks—they test the model's ability to perform multi-step logical inference, mathematical computation, or code generation. None of them test the model's ability to retrieve and utilize specific factual information from long input contexts. Tasks like long-document question answering (e.g., NarrativeQA, Qasper), multi-turn dialogue with long conversation history, in-context learning with many examples, or retrieval-augmented generation with long retrieved passages are absent. The paper claims that "unlike token pruning, quantization preserves all contextual information without discarding tokens" (Section 1), implying that Kitty is particularly suited for retrieval tasks where discarding any token risks losing critical facts. This claim is not tested.

The consequence. Channel-wise precision boost operates on the Key cache, which determines how attention weights are distributed across tokens. For reasoning tasks, attention is often concentrated on a relatively small set of "intermediate result" tokens that carry the reasoning chain forward—the attention pattern is sparse and structured, which may be robust to per-channel quantization noise. For fact-retrieval tasks, attention may be more uniformly distributed across many tokens containing specific facts (dates, names, numbers), and the model may need to attend precisely to the right token at the right position. If quantization noise in the Key cache slightly perturbs attention weights, the model might attend to a semantically similar but factually incorrect token (e.g., retrieving "2023" instead of "2024"), causing factual errors that are penalized differently than reasoning errors. The paper provides no evidence about whether the channel-wise sensitivity patterns observed in reasoning transfer to retrieval, or whether Kitty's 12.5–25% boost rate suffices for factually precise attention.

What evidence exists in the paper. None of the evaluated benchmarks require long-context factual retrieval. The AIME tasks (Table 4) extend to 32K tokens but are still reasoning tasks where the model generates its own chain of thought—there is no long input context with embedded facts to retrieve. The paper motivates the work with long-context scenarios (128K tokens, Section 1: "detailed document understanding, extended dialogues") but evaluates only on reasoning with model-generated long outputs, not on document understanding with long input contexts. The GPQA-Diamond benchmark contains graduate-level science questions that require factual knowledge, but the knowledge is assumed to be in the model's weights (closed-book QA), not retrieved from a provided context.

Mitigation status. The paper does not address this. The benchmark selection is internally consistent (all reasoning-focused) and reasonable for evaluating the core claim about quantization sensitivity in attention mechanisms. However, the absence of retrieval benchmarks leaves a significant gap between the motivating use cases (document understanding, multi-turn dialogue) and the evaluated scenarios. Addressing this would require evaluating Kitty on at least one long-context retrieval benchmark (e.g., a needle-in-a-haystack test, or a long-document QA dataset like NarrativeQA) to establish whether the accuracy benefits transfer to tasks where precise token-level attention matters for factual correctness. The paper's claim that quantization "preserves all contextual information" remains aspirational until tested in a context-retrieval setting.

7. Implications and Future Directions

How This Work Changes the Landscape

Kitty shifts the conversation around KV cache quantization from a mostly token-centric view of precision allocation to a channel-centric one. Prior to this work, the dominant strategies for protecting accuracy under aggressive quantization operated on the token axis: preserve attention sinks (StreamingLLM, KIVI's recent-token windows), preserve outlier tokens (KVQuant), or allocate different bitwidths to entire layers (MiniKV, KVTuner). The implicit assumption was that which tokens you protect is what determines accuracy, because attention is fundamentally about token-to-token interactions. Kitty's per-channel sensitivity analysis (Figure 2b, Section 3.2) challenges this assumption by demonstrating that quantization distortion localizes to specific feature dimensions within each token's key representation — and that these sensitive channels are consistent across query heads within a GQA group.

This is not merely an incremental refinement. It is a reframing of where the information bottleneck lives under aggressive KV cache compression. The field had been asking "which tokens should we protect?" when the more incisive question is "which representational subspaces within the key embeddings carry the signal that matters for attention?" The heavy-tailed distribution in Figure 2b — where a small fraction of channels accounts for the vast majority of attention-score MSE — implies that the channel dimension, not the token dimension, is where representational capacity is most strained under 2-bit quantization. This reframing opens a design space that prior work had not explored: rather than protecting tokens that are "important" (a query-dependent and hard-to-determine-statically property), protect the feature dimensions that are inherently fragile to quantization, independent of the specific query context.

The paper's Key-vs-Value asymmetry finding (Table 2: KIVI-K4V2* achieves near-FP16 accuracy while KIVI-K2V4* lags substantially) provides a mechanistic explanation for this reframing. Key vectors participate in the softmax via exponentiation — small per-channel perturbations get amplified nonlinearly as attention weights redistribute (since weights must sum to one, boosting one weight necessarily suppresses others). Value vectors contribute through a linear weighted sum, where errors propagate additively without nonlinear amplification. This asymmetry was implicitly understood (prior work treated Key and Value differently), but Kitty provides the first diagnostic showing how the sensitivity localizes to specific channels within the Key cache, enabling targeted rather than blanket precision allocation. The finding also implies that improving Key cache quantization should be prioritized over Value cache quantization in future mixed-precision designs — a concrete guidance that prior work had not established with equivalent clarity.

The paper also resolves a practical tension that had limited prior mixed-precision KV cache schemes. KVQuant (Hooper et al., 2024) demonstrated that protecting outlier elements in higher precision can preserve accuracy, but its sparse-FP16 format introduced scattered memory accesses that killed GPU throughput. Kitty's dense-sparse page decomposition (Section 4.1) resolves this tension by converting heterogeneous-precision pages into two homogeneous 2-bit tensors — a design pattern applicable beyond KV cache quantization to any system that needs to combine accuracy-preserving mixed precision with GPU-friendly coalesced memory access. The principle is: push the heterogeneity into lightweight metadata (the boost index tensor, shape (D,), which is tiny compared to (D, G) page data) and keep the bulk data tensors uniform in type and access pattern. This is not a theoretical advance in compression, but it is a practical enabling contribution — without it, channel-wise precision boost would remain a simulation-only result, unrealizable in production inference systems.

Perhaps most importantly, Kitty establishes a feasibility result for 2-bit KV cache quantization that prior work had not achieved. The paper's motivating observation — that 4-bit KIVI preserves accuracy while 2-bit KIVI catastrophically degrades it (Table 1: MATH-Algebra drops from 88.26 to 47.29 on Qwen3-8B) — set a clear failure boundary. By showing that boosting only 12.5% of Key-cache channels to INT4 suffices to recover most of the accuracy (Kitty: -2.18 average gap vs. FP16) and that 25% achieves near-parity (Kitty-Pro: -0.97 average gap), the paper demonstrates that the 2-bit accuracy cliff is not inherent to the precision level — it is an artifact of uniform allocation. The practical implication is that 2-bit KV cache quantization is viable, but only when precision is differentially allocated to the most sensitive representational subspaces. This shifts the research agenda from "can we make 2-bit work?" (answered: yes, with channel-wise boost) to "how do we identify the most sensitive channels with minimal overhead?" and "how does sensitivity vary across architectures, tasks, and context lengths?"

The work also redirects research attention away from ever-more-sophisticated search over quantization parameters and toward understanding the structural properties of learned key representations. The fact that a simple magnitude-based heuristic (Equation 2) captures enough of the true channel sensitivity to substantially outperform random selection (Figure 4) suggests that the sensitivity pattern is not a subtle statistical artifact requiring complex analysis — it is correlated with a readily observable property (activation magnitude). This makes the problem more tractable than it might have appeared: if sensitivity were entirely decoupled from simple statistics, practical deployment would require expensive per-channel MSE analysis at runtime. The magnitude heuristic's effectiveness lowers the barrier to entry and makes real-time adaptive channel selection feasible.

The paper does not resolve how channel sensitivity patterns generalize across architectures — the evaluation is restricted to GQA models (Qwen3, LLaMA3), and the sensitivity analysis is shown for a single layer of a single model. Researchers working on non-GQA architectures (MHA, MQA) or non-transformer models cannot assume the 12.5–25% boost rates will transfer. The field still lacks a theory of KV cache channel sensitivity that would predict, from architectural properties, which channels will be most sensitive and how sensitivity concentrates. Kitty provides the diagnostic toolkit (per-channel MSE measurement) and the practical demonstration (magnitude heuristic works) that make such a theory empirically testable, but the theory itself remains to be built.

Follow-Up Research This Work Enables

1. Oracle channel selection vs. magnitude heuristic: quantifying the ceiling for importance estimation. Kitty establishes that magnitude-based channel selection substantially outperforms random selection (Figure 4: ~4.5 point advantage on MATH-Algebra at 12.5% boost), but the paper's own sensitivity analysis (Figure 2b) provides a ground-truth ranking via per-channel attention-score MSE. A direct follow-up experiment would compare Kitty's accuracy when channels are selected by the magnitude heuristic vs. the oracle per-channel MSE ranking. The key question is: how much accuracy is left on the table by using a cheap heuristic instead of the expensive ground-truth sensitivity measurement? If the gap is small (e.g., <1 point on average), then the magnitude heuristic is effectively solved and research should focus on making it cheaper. If the gap is large (several points), then there is substantial room for improved importance heuristics — perhaps learned channel importance predictors, or statistics beyond magnitude (variance, kurtosis, or correlation with query embeddings). This experiment requires only running the per-channel MSE analysis (which the paper already performs for Figure 2b) on a held-out calibration set, recording the oracle channel ranking, and evaluating Kitty with oracle-selected channels on the test benchmarks. The result would establish an upper bound on what any heuristic-guided channel selection can achieve, providing a target for future heuristic development.

2. Cross-architecture sensitivity transfer: do MHA and MQA models exhibit the same channel concentration? The paper's channel sensitivity analysis (Figure 2) and all accuracy results are on GQA models (Qwen3, LLaMA3). GQA shares each key head across multiple query heads, which may force key representations to encode information relevant to diverse query patterns — potentially concentrating sensitivity into a few channels that serve all queries. A critical stress test would evaluate the same per-channel MSE methodology on (a) pure Multi-Head Attention models (where each query head has its own dedicated key head, potentially allowing sensitivity to be distributed across more channels since each key head is specialized), (b) Multi-Query Attention models (where one key head serves all queries, potentially concentrating sensitivity even more than GQA), and (c) non-transformer attention variants if available. The experiment would measure whether the channel sensitivity distribution (the "heavy-tailedness" visible in Figure 2b) varies systematically with the query-to-key head ratio. A strong result would be: the concentration of sensitivity into a small fraction of channels is a universal property of learned key representations, independent of attention architecture. This would validate Kitty's approach as broadly applicable. A negative result — e.g., MHA models show uniform sensitivity across all channels — would establish a boundary condition: channel-wise precision boost only helps for GQA/MQA architectures, and different strategies are needed for MHA. This negative result would be equally valuable for guiding deployment decisions.

3. Per-layer channel sensitivity profiling and layer-discriminative boost allocation. Kitty applies the same boost rate (12.5% or 25%) uniformly across all layers. The paper's visualization (Figure 2a) shows channel-wise magnitude patterns for Layer 10 of Qwen3-8B, with the claim that "similar patterns are observed on other layers," but does not quantify how the sensitivity distribution varies with layer depth. Prior work on layer-wise mixed precision (MiniKV, KVTuner) has shown that different layers tolerate different levels of quantization. A natural extension would combine these insights: measure per-channel sensitivity separately for each layer (using the same per-channel MSE methodology from Figure 2b, applied to early, middle, and late layers), then allocate boost budgets discriminatively — layers with highly concentrated sensitivity get higher boost rates, layers with diffuse sensitivity get lower boost rates (or none). The experiment would compare: (a) uniform boost (Kitty's current approach) vs. (b) layer-discriminative boost with the same total boost budget (total boosted channels summed across layers equals the uniform case). The key metric is accuracy at fixed total memory. If layer-discriminative allocation improves accuracy at the same memory budget, it would demonstrate that channel-wise and layer-wise mixed precision are indeed complementary, as the paper speculates. This experiment is enabled by the fact that Kitty's per-channel importance scoring is already computed per quantization group — aggregating to per-layer statistics requires only a lightweight profiling step.

4. Interaction between sink preservation, local window, and channel-wise boost: Pareto-optimal hyperparameter configurations. The paper fixes sink tokens (S = 32), local Value window (R = 128), and boost rate (α = 0.125 or 0.25) without any tradeoff analysis among them (Section 6, Limitations). Each mechanism consumes a different type of memory budget: sink tokens cost FP16 memory for the first S positions (fixed overhead regardless of sequence length), the local window costs FP16 memory for the most recent R Value tokens (fixed overhead), and channel-wise boost costs an additional α × 2 bits per Key element across all non-sink tokens (overhead proportional to sequence length). A systematic tradeoff analysis would sweep S (e.g., 0, 8, 16, 32, 64, 128), R (e.g., 0, 32, 64, 128, 256), and α (e.g., 0, 0.0625, 0.125, 0.25, 0.5) on a held-out calibration set, measuring accuracy and effective bits-per-element for each combination at multiple sequence lengths. The deliverable is a Pareto frontier: for a given target accuracy (e.g., within 1 point of FP16), what is the minimum achievable bits-per-element, and which combination of mechanisms achieves it? This is directly actionable for practitioners who need to tune Kitty for their specific accuracy-memory tradeoff. The experiment is straightforward given the paper's simulation framework but requires careful design to avoid overfitting — hyperparameters should be selected on a validation set and evaluated on held-out benchmarks.

5. Long-context factual retrieval: does channel-wise boost preserve precise token-level attention? All of Kitty's accuracy benchmarks are reasoning tasks (GSM8K, MATH, GPQA, HumanEval, AIME) where the model generates its own chain of thought and attention patterns are likely sparse and structured. The paper motivates KV cache compression with long-context document understanding and multi-turn dialogue (Section 1), but never evaluates on a retrieval task where the model must attend to specific factual details embedded in a long input context. A critical stress test would evaluate Kitty on a needle-in-a-haystack benchmark: embed a specific fact (e.g., "The secret code is 7a3f29") at a random position in a long document (4K, 8K, 16K, 32K, 64K tokens), and measure whether the quantized model can retrieve it. The hypothesis to test: channel-wise quantization noise in the Key cache may introduce small perturbations in attention weights that cause the model to attend to semantically similar but factually incorrect tokens. For reasoning tasks, where attention is concentrated on a few intermediate result tokens, this may not matter. For retrieval tasks, where the model needs precise attention to a specific token position, even small attention weight perturbations could cause the model to "miss" the needle. The experiment would measure retrieval accuracy as a function of context length and boost rate. A negative result (Kitty degrades retrieval accuracy at long contexts even when reasoning accuracy is preserved) would establish that channel-wise boost protects semantic attention patterns but not positional precision — an important boundary condition for deployment in document QA systems. A positive result (retrieval accuracy is preserved) would validate the paper's claim that quantization "preserves all contextual information."

6. Learned channel importance predictors: reducing the online computation cost of channel selection. The current magnitude-based heuristic requires reading all D × G Key activations per quantization group to compute per-channel averages, then performing a top-K selection. While amortized across G = 128 tokens, this is still a non-trivial amount of computation and memory access (Section 6, Limitations). A natural extension would train a lightweight channel importance predictor that takes only a small subset of tokens (or even just the query embedding) as input and predicts which channels to boost, avoiding the full activation scan. The training signal would come from the per-channel MSE analysis (Figure 2b): for a calibration dataset, compute oracle channel rankings, then train a small MLP or linear model to predict the top-K channels from reduced features (e.g., the first few tokens' key vectors, the query embedding, or layer-level statistics). The evaluation would compare: (a) the trained predictor's channel selection accuracy (fraction of oracle top-K channels correctly identified) vs. (b) the magnitude heuristic's selection accuracy, and (c) downstream task accuracy when using predictor-selected channels vs. magnitude-selected channels. If a predictor achieves comparable accuracy with, say, 10× fewer activations read, it would reduce the quantization overhead and make dynamic channel selection feasible even at smaller group sizes. This experiment builds directly on the paper's diagnostic methodology (per-channel MSE) and its practical deployment concern (quantization overhead).

Practical Applications and Downstream Use Cases

1. Cost-efficient long-context API serving. For LLM inference providers serving long-context requests (document summarization, codebase analysis, multi-turn conversational agents), KV cache memory is the dominant operating cost because it determines how many concurrent requests can fit on a single GPU. Kitty's ~7–8× memory reduction directly translates to ~8× more concurrent requests per GPU (Figure 5a: batch size 256 vs. 32 for FP16 on Qwen3-8B at 8K context). For a provider running Qwen3-8B on A100 instances, this means roughly 8× lower GPU cost per request for memory-bound long-context workloads. The accuracy evaluation (Tables 3–4) shows that Kitty-Pro preserves accuracy within ~1 point of FP16 on reasoning benchmarks up to 32K context, making the cost reduction achievable without meaningful quality degradation for query types where KV cache compression is most needed. The primary deployment consideration is the per-group quantization cost (discussed in Section 6, Limitations): operators should profile tail latency at page boundaries to ensure interactive response times are maintained. The paper's default G = 128 means quantization launches every 128 decoding steps — on a 32K-token generation, this is 256 quantization launches, each potentially introducing a latency spike. Profiling and potentially tuning G (larger groups → fewer launches but larger spikes) would be essential before production deployment.

2. On-device deployment of medium-sized reasoning models. Smaller LLMs (8B–14B parameters) are increasingly deployed on edge devices for privacy-sensitive applications (on-device email summarization, local code assistants). On-device GPUs and NPUs typically have limited memory (8–16 GB), making KV cache compression essential for supporting long reasoning chains. Kitty enables a Qwen3-8B model to generate 32K-token chain-of-thought responses while keeping the KV cache within ~2.5 GB (vs. ~20 GB for FP16) — a configuration that would otherwise be infeasible on a 16 GB laptop GPU. The accuracy results on AIME24/25 at 32K context (Table 4: Kitty narrows the gap to FP16 from -14.17 to -3.67 on Qwen3-8B) suggest that on-device reasoning quality would remain competitive. The key practical consideration is the Triton kernel dependency: current on-device deployment frameworks (llama.cpp, MLX, ExecuTorch) may not support Triton, requiring reimplementation of the dequantization kernels (Algorithm 1) in the target framework's kernel language. The dense-sparse decomposition is algorithmically simple (two dense loads + a small index lookup), making reimplementation tractable.

3. Batch inference for synthetic data generation and model evaluation. Organizations increasingly use LLMs to generate synthetic training data (for fine-tuning smaller models) or to evaluate large test suites (hundreds of thousands of examples). These workloads are throughput-bound and embarrassingly parallel — the goal is to process as many examples as possible within a fixed compute budget. Kitty's 3.7× throughput improvement over FP16 at the same memory budget (Figure 5b: ~1,300 vs. ~350 tokens/s on Qwen3-8B at 8K context) directly reduces the GPU-hours required for such batch jobs. For a job generating 100K reasoning traces of average length 4K tokens, the FP16 baseline on 8×A100s would require roughly 100,000 × 4,000 / (350 × 8) ≈ 143,000 seconds ≈ 40 hours; with Kitty-Pro, this drops to 100,000 × 4,000 / (1,300 × 8) ≈ 38,500 seconds ≈ 10.7 hours. The 3.7× speedup is particularly valuable here because batch inference jobs are cost-sensitive and can tolerate the periodic latency spikes from quantization launches (since there is no interactive user waiting). The main deployment consideration is ensuring the accuracy of the generated data — since synthetic data quality directly impacts downstream fine-tuning, the ~1-point accuracy gap of Kitty-Pro relative to FP16 (Table 3) should be validated on the specific data generation task. If the generation task involves factually precise outputs (rather than reasoning), the retrieval stress test (Follow-Up Research direction 5) becomes especially relevant.

4. High-throughput multi-tenant inference platforms. Inference platforms serving many concurrent users with diverse query types (some short, some long-context) face a memory fragmentation problem: long-context requests consume disproportionate KV cache memory, forcing the system to either reserve large memory blocks (wasting capacity for short requests) or dynamically manage variable-sized allocations (increasing management overhead). Kitty's page-centric layout with PagedAttention integration (Section 4.1) enables fine-grained, uniform-page-size memory management even with mixed-precision quantization — each page is exactly G = 128 tokens regardless of precision configuration, and the dense-sparse decomposition stores pages as fixed-size blocks. This means the platform can use a single page allocator for all sequence lengths, simplifying memory management and reducing fragmentation. The 8× larger batch capacity (Figure 5a) means the platform can absorb more concurrent long-context users before hitting memory limits, reducing request queuing and improving tail latency under load. The practical consideration is integration with existing PagedAttention implementations (vLLM, TensorRT-LLM): adapting Kitty's page format to work with these frameworks' page tables and scheduling logic requires API-level changes (the page table must understand the two-tensor Key page format and the boost index). The paper provides a Triton reference implementation (Algorithm 1) that would need to be ported to each framework's kernel language.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks2025· llm-systems← in-domain predecessor
- Multilingual E5 Text Embeddings: A Technical Report2024· context-optimization← LLM-judged similar
- DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search2025· llm-systems← in-domain predecessor
- Qwen3Guard Technical Report2025· safety← topic thread
WUSH: Near-Optimal Adaptive Transforms for LLM Quantization2025· alignment← topic thread
- AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration2023· inference-optimization← LLM-judged similar
- Video Generation Models Are Good Latent Reward Models2025· alignment← in-domain predecessor
- QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs2025· rl-training← topic thread
The Pitfalls of KV Cache Compression2025· inference-optimization← topic thread
- Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models2025· architecture← LLM-judged similar
- LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention2025· inference-optimization← LLM-judged similar
- Inference-Time Hyper-Scaling with KV Cache Compression2025· inference-optimization← LLM-judged similar
- Layer-Condensed KV Cache for Efficient Inference of Large Language Models2024· architecture← LLM-judged similar
- GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression2024· architecture← LLM-judged similar
- Transformers are Multi-State RNNs2024· inference-optimization← LLM-judged similar
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference2024· serving← LLM-judged similar
- Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization2025· inference-optimization← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models2025· llm-systems→ in-domain predecessor
- Bolmo: Byteifying the Next Generation of Language Models2025· llm-systems→ in-domain predecessor
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities2025· llm-systems→ similar topic
- MolmoWeb: Open Visual Web Agent and Open Data for the Open Web2026· llm-systems→ similar topic
FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error2025· training-methods→ topic thread
- Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs2025· training-methods→ in-domain predecessor
- Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling2025· inference-optimization→ topic thread
- Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning2025· architecture→ topic thread
HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing2026· architecture→ topic thread
- Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization2026· architecture→ in-domain predecessor
- OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization2026· inference-optimization→ topic thread
- Routing Transformer2020· architecture→ similar topic
- Faster Causal Attention Over Large Sequences Through Sparse Flash Attention2023· architecture→ similar topic
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention2025· architecture→ similar topic