OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

ArXiv: 2605.17757

🎯 Pitch

Standard rotation-based INT2 KV-cache quantization collapses to nearly zero accuracy on long-context reasoning tasks, while OSCAR maintains near-BF16 quality by learning rotations that align with what attention actually reads. By deriving key rotations from query covariance and value rotations from attention-score-weighted value covariance, OSCAR achieves up to 7× throughput gains and 8× memory reduction on models up to 358B parameters. This fundamentally reframes ultra-low-bit quantization as a problem of matching the quantization basis to downstream attention geometry rather than minimizing raw cache reconstruction error.

---

1. Executive Summary

This paper introduces OSCAR (Offline Spectral Covariance-Aware Rotation), an INT2 KV-cache quantization method that derives fixed rotations and clipping thresholds from attention-aware covariance structures estimated offline—key rotations from query covariance (the directions queries actually read along) and value rotations from attention-score-weighted value covariance (the directions attention actually aggregates)—rather than from raw cache reconstruction error, which prior rotation-based methods (QuaRot-style Hadamard transforms) optimize. Evaluated on reasoning models up to 358B parameters (Qwen3-4B-Thinking, Qwen3-8B, Qwen3-32B, and GLM-4.7) across five benchmarks with 32k-token generation and RULER long-context retrieval up to 128k tokens, OSCAR reduces the BF16 accuracy gap to 3.78 and 1.42 points on the 4B and 8B models respectively while naive rotation INT2 collapses to near zero, and achieves up to 7× throughput improvement at large batch sizes under the same memory budget, establishing that covariance-target rotations preserve attention fidelity at 2-bit precision only when the rotation basis is aligned with what downstream attention actually consumes.

2. Context and Motivation

The Core Problem: INT2 KV-Cache Quantization Remains Unusable for Real Deployment

The paper tackles a specific, practical bottleneck in serving large language models at long context lengths. During autoregressive decoding, every generated token must attend to all previously generated keys and values stored in the KV cache. As context lengths grow to tens or hundreds of thousands of tokens, the KV cache becomes the dominant memory consumer on the GPU—often exceeding the model weights themselves in memory footprint. This creates a hard tradeoff: either limit batch size (reducing throughput) or limit context length (reducing capability).

INT2 quantization promises an 8× memory reduction relative to BF16 storage, which would directly translate to larger feasible batch sizes and longer supportable contexts under the same memory budget. However, the paper establishes that INT2 KV-cache quantization has remained practically undeployable despite its attractive theoretical compression ratio. The reason is not merely that INT2 introduces noise—all quantization does—but that at only four representable levels, the noise is so severe that it destroys the model's ability to attend correctly unless the quantization is carefully aligned with what attention actually needs.

The paper quantifies this collapse explicitly in its experiments (Table 2). On Qwen3-8B, naive INT2 quantization without any preprocessing yields 0% accuracy across all five benchmarks under 32k-token generation. Hadamard-rotated INT2 (the QuaRot approach, which is the standard rotation baseline) recovers only 10.14 average accuracy, compared to 70.84 for BF16—a 60.70-point gap. This is not a marginal degradation; it is catastrophic failure. The paper's Figure 3 confirms that this failure manifests as a rapid divergence of the attention distribution from the FP16 reference as context length increases, with KL divergence climbing to 7–12 at 128k context for Hadamard-only rotation while OSCAR stays near 1–3.

Why This Problem Matters Now

Three converging trends make this problem urgent:

1. Long-context inference is becoming the default deployment mode. Reasoning models like Qwen3-Thinking, DeepSeek-R1, and o1-style systems generate reasoning traces that routinely span 16k–64k tokens before producing a final answer. In this regime, the KV cache for a single request can consume gigabytes of memory. For the 358B-parameter GLM-4.7 model, a single 128k-token context would require approximately 7.3 GB of KV cache at BF16—nearly 10% of an H100's 80 GB for just one request's cache state. At serving scale with concurrent users, this memory pressure is the binding constraint on throughput and cost.

2. The batch-size-vs-context-length tradeoff is a real economic bottleneck. Providers serving LLMs charge per token and must maximize GPU utilization to be economical. If KV cache memory forces small batch sizes at long contexts, per-user throughput drops and per-token serving costs rise. The paper demonstrates this directly in its throughput experiments (Figure 4, right): at 100k context and batch size 32, OSCAR achieves 6.17× higher job-level throughput than BF16 on Qwen3-4B-Thinking, and 7.83× on GLM-4.7-FP8. The alternative—keeping BF16 precision—means either accepting these throughput penalties or refusing to serve long-context requests at scale.

3. The gap between 4-bit and 2-bit KV quantization remains wide. 4-bit methods like Saw-INT4 (which OSCAR uses as a baseline) achieve near-lossless accuracy (Table 2: Saw-INT4 drops only 0.87 points on Qwen3-8B and actually exceeds BF16 on Qwen3-32B at +0.24 points). But 4 bits provides only 4× compression over BF16, not 8×. The paper argues that INT2 is the sweet spot—hardware-friendly fixed-width representation with maximum compression—but only if the accuracy gap can be closed. Closing this gap is the central contribution.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work, each with specific limitations that OSCAR addresses:

Rotation-based quantization (QuaRot, TurboQuant): data-oblivious transforms are insufficient at INT2. The key insight from Ashkboos et al. (2024) was that applying fixed orthogonal transforms (specifically, Hadamard matrices) before quantization redistributes activation outliers across channels, producing more uniform distributions that quantize better. This works well at 4 bits—QuaRot-style Hadamard rotation significantly reduces the 4-bit error. However, at INT2, the paper demonstrates that this approach is fundamentally limited. The Hadamard transform is data-oblivious: it applies the same mixing matrix regardless of which directions are important for the downstream computation. At only four quantization levels, the residual error must be pushed systematically into directions that attention reads weakly, not just spread uniformly. Section 2 provides the mathematical motivation: the downstream attention-logit distortion is ∥QK^⊤ - QK̂^⊤∥_F^2 = tr((K - K̂)Q^⊤Q(K - K̂)^⊤), which depends on the query covariance Q^⊤Q, not on the raw key covariance K^⊤K. A rotation that minimizes raw Euclidean reconstruction error of K can still produce large logit errors if the quantization noise lands in directions with high query energy. Figure 2 makes this concrete: Hadamard-only rotation reduces raw K/V MSE somewhat compared to naive INT2, but the attention-score KL divergence and attention-output MSE remain an order of magnitude larger than OSCAR's.

Channel-wise and mixed-precision methods (KIVI, KVQuant, Kitty): accuracy at the cost of deployability. These methods identify that certain channels or tokens are more sensitive to quantization and assign them higher precision, residual buffers, or custom page layouts. For example, KIVI uses asymmetric quantization with per-channel scales and zero points; Kitty promotes sensitive key-cache channels to higher precision with a residual cache. The paper acknowledges these methods can achieve good accuracy—Table 1 shows KIVI-KV2 reaching 52.33 and Kitty reaching 59.67 on Qwen3-8B AIME25 at 32k generation—but identifies a critical practical limitation: "These methods require residual buffers, channel-wise scales, promoted channels, or custom page layouts, and we do not have paged/fused kernels for them at 32K generation length." In other words, these methods break compatibility with the standard paged-attention memory layout (Kwon et al., 2023) and fused decode kernel pipelines (FlashAttention, FlashDecoding) that modern serving systems like vLLM and SGLang depend on. OSCAR's design constraint is explicit: it must produce a uniform dense INT2 representation that drops into existing paged KV-cache infrastructure without requiring sparse metadata, irregular page sizes, or extra memory traffic for reconstruction.

Covariance-aware calibration for weights, not for KV cache attention targets. A separate line of work uses calibration data and Hessian/covariance information to guide weight quantization (GPTQ, AWQ, OmniQuant) or low-rank weight compression (ASVD, SVD-LLM, CorDA). These methods estimate which weight directions matter for the layer output and allocate precision accordingly. However, the paper argues that applying this principle to KV-cache quantization requires a fundamentally different target: for weights, the Hessian captures sensitivity of the loss to weight perturbations; for KV caches, the relevant sensitivity is how quantization error propagates through the attention mechanism to the layer output. Specifically, key errors are filtered through Q^⊤Q (what queries look for) and value errors are filtered through S^⊤S (how much attention is paid to each token). The paper's Appendix A.3 formalizes this: "the rotation should reduce the downstream attention error, not merely the Euclidean reconstruction error of K and V." Prior covariance-aware methods did not derive rotations from these attention-specific covariance targets.

How This Paper Positions Itself

OSCAR sits at the intersection of two design philosophies and claims to combine their strengths while avoiding their weaknesses:

• From rotation-based methods, OSCAR inherits the commitment to fixed orthogonal transforms that preserve dense tensor layout, paged-cache compatibility, and fused kernel integration. This is why the paper emphasizes that OSCAR's rotated INT2 representation is a drop-in replacement for standard KV cache pages in SGLang—there is no per-channel routing, no residual buffers, and no sparse metadata.
• From covariance-aware calibration, OSCAR borrows the idea that the compression transform should be derived from data statistics, but redirects the target from raw parameter sensitivity to attention-consumed covariance. The key claim is that rotating K into the eigenbasis of Q^⊤Q and V into the eigenbasis of V^⊤S^⊤SV minimizes a tractable surrogate for the actual attention error—a claim formalized in Theorem 1 (Appendix A.6) under a frozen-residual assumption.

The paper's positioning is also methodological: it does not introduce a new quantization algorithm (it uses standard affine asymmetric INT2 with per-token min-max scaling) but rather a new calibration target for the rotation that precedes quantization. The Hadamard transform and bit-reversal permutation are still used, but they serve specific geometric purposes after the attention-aware eigenbasis rotation: the Hadamard equalizes the importance-metric diagonal across channels (Lemma 1, Appendix A.4), and the bit-reversal permutation balances importance across quantization groups.

The paper explicitly contrasts this with QuaRot: "OSCAR follows the same broad rotation principle, but asks which rotation target is appropriate for INT2 KV cache quantization." The answer is that the rotation must be derived from Q^⊤Q and V^⊤S^⊤SV, not from K^⊤K and V^⊤V (raw cache reconstruction) and not from a data-oblivious fixed matrix. The ablation in Table 4 confirms this: replacing OSCAR's attention-aware PCA target with raw K^⊤K/V^⊤V PCA drops mean accuracy from 70.01 to 31.12 on Qwen3-8B—a 38.89-point gap that isolates the value of choosing the right covariance target.

Finally, the paper positions its contribution as end-to-end deployable, with a complete SGLang serving implementation that handles prefill quantization, decode dequantization, inverse rotation, and mixed-precision sink/recent window management within fused Triton kernels. This distinguishes it from method-only contributions that demonstrate accuracy improvements without addressing the systems integration challenges.

3. Technical Approach

3.1 Reader Orientation

OSCAR is an offline calibration system that produces fixed, per-layer rotation matrices and clipping thresholds, which are then used online inside a modified serving engine to compress the bulk of the KV cache to 2-bit integers while preserving near-lossless attention fidelity. The core idea is that the rotation applied before quantization should align the coordinate axes with the directions that attention actually cares about — specifically, the directions along which queries read keys and along which attention scores weight values — so that the inevitable INT2 quantization noise is steered into directions that cause minimal downstream attention error.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two phases: offline calibration and online serving.

Offline phase (Section 3, Algorithm 1):

1. Calibration forward pass: The model processes a small calibration dataset (8k tokens from GPQA-Diamond by default). At each transformer layer and each attention head, the raw query, key, and value activations (Q, K, V) and the attention scores S are dumped to storage.
2. Covariance estimation: From these activations, two attention-aware covariance matrices are computed per layer: (a) the query covariance C_Q = (1/N) Σ q_n^⊤ q_n, which captures which directions in key space queries actually project onto; (b) the score-weighted value covariance C_S = (1/N) V^⊤ S^⊤ S V, which captures which directions in value space are amplified by attention aggregation.
3. Eigendecomposition and rotation construction: Each covariance matrix is eigendecomposed (C = U Λ U^⊤). The eigenbasis U_Q becomes the raw key rotation; U_S becomes the raw value rotation. Each is then post-multiplied by a Hadamard matrix H_Had (to equalize per-channel importance) and a bit-reversal permutation P_br (to balance importance across quantization groups), producing final fixed rotation matrices R_K = U_Q H_Had P_br and R_V = U_S H_Had P_br.
4. Clip threshold calibration: The rotated calibration activations are used to fit per-layer clipping percentile thresholds (ρ_K ≈ 0.96, ρ_V ≈ 0.92), which control outlying token values before quantization.
5. Output: The per-layer matrices R_K, R_V and clip ratios are stored as fixed serving parameters. No further learning or adaptation occurs at runtime.

Online phase (Section 4, Algorithm 1):

1. Prefill: Each new key/value vector is rotated into the OSCAR basis, clipped, and quantized to asymmetric affine INT2 with per-token per-group min-max scaling (group size 128 by default). The first S₀ = 64 tokens (attention sinks) and the most recent W = 256 tokens are kept in BF16; the middle history is stored as packed INT2 in standard paged-attention blocks.
2. Decode: At each decoding step, the INT2 history pages are dequantized to FP16, inverse-rotated back to the original coordinate system, and concatenated with the BF16 sink and recent windows. The fused Triton attention kernel computes attention over this mixed-precision cache.
3. Window management: As decoding advances, the oldest recent-window token is demoted from BF16 to INT2 via a fused clip-quantize kernel, maintaining the invariant that only sink and recent tokens consume high-precision memory.

This architecture ensures that (1) the computationally expensive covariance estimation and eigendecomposition happen once offline, (2) the online serving path adds only lightweight fixed-matrix multiplies and quantization operations that are fused into existing decode kernels, and (3) the mixed-precision layout preserves compatibility with paged-attention memory management and prefix-cache reuse.

3.3 Roadmap for the Deep Dive

The detailed technical breakdown follows this order, which mirrors how the system is built from mathematical motivation to production deployment:

1. The mathematical argument for attention-aware rotation targets (Section 3.4.1). This is where the paper derives the key and value distortion objectives — Equations (1) and (2) from Section 2 — and explains why Q^⊤ Q and V^⊤ S^⊤ S V are the right covariance matrices to diagonalize, rather than K^⊤ K and V^⊤ V. Understanding this justification is essential before seeing how it is operationalized.

2. The three-factor rotation construction (Section 3.4.2). This unpacks the composed rotation R = U · H_Had · P_br factor by factor, explaining what geometric problem each factor solves, why they compose in this specific order, and what would go wrong if any factor were omitted. The worked example from Appendix A.4 is incorporated here to make the abstract algebra concrete.

3. Optimality under the frozen-error surrogate (Section 3.4.3). Theorem 1 provides a formal guarantee that R_K = U_Q and R_V = U_S minimize tractable surrogates for the attention error, assuming diagonal frozen residuals. We walk through the proof structure, what the assumptions mean physically, and what the theorem does and does not guarantee about end-to-end decoding.

4. Quantization and clipping mechanics (Section 3.4.4). Once the rotation is fixed, the actual INT2 encoding uses standard asymmetric affine quantization with per-token per-group dynamic scaling and percentile-based clipping. This section details the exact computations for scale, zero point, clipping thresholds, and dequantization, plus the mixed-precision sink/recent-window layout.

5. Online serving integration (Section 3.4.5). This covers the fused Triton kernel design: how prefill writes rotate-clip-quantize in one kernel, how decode reads dequantize-inverse-rotate-attend in fused steps, how the sink/recent window management is implemented, and how value rotation is absorbed into the output projection weight for latency reduction. The kernel profiling data from Table 8 is contextualized here.

3.4 Detailed, Sentence-Based Technical Breakdown

This is fundamentally an inference-time quantization paper whose core technical idea is that the optimal orthogonal transform to apply before INT2 quantization is not a fixed data-oblivious matrix (like a Hadamard transform) but rather the eigenbasis of an attention-aware covariance matrix — specifically, query covariance Q^⊤ Q for keys and score-weighted value covariance V^⊤ S^⊤ S V for values — because these bases diagonalize the sensitivity of the downstream attention computation to per-channel quantization error, allowing the inevitable INT2 noise to be concentrated in channels where it causes minimal distortion to attention logits and layer outputs.

---

3.4.1 Why Attention-Aware Covariance Matters: The Downstream Distortion Objectives

The paper's central mathematical motivation appears in Section 2 and is elaborated in Appendix A.3. The key observation is that the naive Euclidean reconstruction error of a quantized KV cache — the quantity ∥K - Q(K)∥_F^2 — is not what attention actually cares about. Keys and values are consumed by attention through nonlinear operations: keys through the dot-product logits Q K^⊤, and values through the attention-weighted aggregation S V.

Key-side distortion. For keys, the paper derives the downstream logit distortion caused by quantization:

$\|Q K^\top - Q \hat{K}^\top\|_F^2 = \text{tr}\left((K - \hat{K}) Q^\top Q (K - \hat{K})^\top\right)$

where Q ∈ ℝ^{T×d} is the query matrix (T tokens, d head dimension), K ∈ ℝ^{T×d} is the true key matrix, \hat{K} = Q(K R) R^⊤ is the quantized-dequantized key matrix after rotation by R and inverse rotation by R^⊤, and Q^⊤ Q ∈ ℝ^{d×d} is the empirical query covariance matrix (uncentered second moment).

What it computes: the squared Frobenius norm of the difference between the true attention logits Q K^⊤ and the logits computed from quantized keys Q \hat{K}^⊤. This is a scalar measuring total logit distortion across all query-key pairs. The trace form on the right shows that the key reconstruction error (K - \hat{K}) is weighted by the query covariance Q^⊤ Q: a unit of key error in a direction where queries have large energy (large eigenvalue of Q^⊤ Q) causes more logit distortion than the same error in a direction where queries have small energy.

Why this form: the identity follows from cyclic permutation of the trace and the fact that ∥X∥_F^2 = tr(X X^⊤). The critical insight is that Q^⊤ Q appears sandwiched between the key error terms, which means the coordinate system in which we measure key error should be aligned with the eigenvectors of Q^⊤ Q if we want to control logit distortion, not with the eigenvectors of K^⊤ K (which would control raw Euclidean reconstruction error). The paper's Figure 2 (middle-right panel) empirically validates this decomposition: OSCAR's attention-score KL divergence is 1–2 orders of magnitude smaller than Hadamard-only or clip-only baselines, despite comparable raw K/V MSE.

Value-side distortion. For values, the derivation is analogous but the weighting matrix is different:

$\|S V - S \hat{V}\|_F^2 = \text{tr}\left((V - \hat{V})^\top S^\top S (V - \hat{V})\right)$

where S = softmax_row(Q K^⊤ / √d) ∈ ℝ^{T×T} is the attention score matrix, V ∈ ℝ^{T×d} is the true value matrix, and \hat{V} = Q(V R) R^⊤ is the quantized-dequantized value matrix.

What it computes: the squared Frobenius norm of the difference between the true attention output (the weighted sum of values) and the output computed from quantized values. The trace form shows that value error is weighted by S^⊤ S, which is the Gram matrix of attention score vectors. Intuitively, S^⊤ S measures how much pairs of tokens co-attend to the same queries: a token j with a large column norm in S (i.e., token j is attended to strongly by many queries) receives a large diagonal entry in S^⊤ S, meaning quantization error on token j's value vector is heavily penalized. Similarly, tokens i and j that are attended to by similar query patterns (large off-diagonal entry in S^⊤ S) have correlated error penalties.

Why this form: the paper argues that V^⊤ S^⊤ S V (rather than S^⊤ S alone or V^⊤ V alone) is the right target because it represents the attention-weighted value covariance. This is (S V)^⊤ (S V), the second-moment matrix of the attention-weighted value activations. Diagonalizing this matrix means the quantized values preserve the directions in value space that survive attention aggregation with the largest magnitude. Raw value covariance V^⊤ V would preserve directions where values are large regardless of whether those tokens are actually attended to — a token with enormous value activations that receives near-zero attention weight would be preserved unnecessarily under a V^⊤ V target but appropriately deprioritized under the V^⊤ S^⊤ S V target.

The critical gap in prior work. The paper's key critique of rotation-based methods like QuaRot (Ashkboos et al., 2024) is that they apply a fixed Hadamard transform H_Had to K and V before quantization, which has the effect of spreading per-channel outlier energy uniformly across all channels. This improves raw cache reconstruction error because the quantization scale becomes more uniform across channels. However, it does not distinguish between directions that are important for attention and directions that are not. At INT2 (only 4 levels), the residual quantization error is large enough that this distinction matters: the noise must be steered into low-importance directions. OSCAR's contribution is to precede the Hadamard transform with an attention-aware eigenbasis rotation that first aligns the coordinate axes with the importance hierarchy of the downstream attention computation.

Empirical confirmation that Q^⊤ Q and K^⊤ K have different eigenbases. Appendix A.4 reports a crucial measurement: on Qwen3-8B with 2000 calibration tokens per layer, the top-8 eigenvector self-alignment between U_Q (eigenvectors of Q^⊤ Q) and U_K (eigenvectors of K^⊤ K) ranges from 0.05 to 0.15 across the 36 layers. The expected alignment for two random orthogonal bases in 128 dimensions is 1/√d ≈ 0.09. This means the two eigenbases are essentially uncorrelated — diagonalizing Q^⊤ Q does not diagonalize K^⊤ K, and vice versa. This empirically justifies the paper's claim that the choice of which covariance matrix to diagonalize is not a minor detail but a fundamental design decision with large accuracy consequences (confirmed by the 38.89-point gap in Table 4 when switching from Q^⊤ Q to K^⊤ K as the PCA target).

---

3.4.2 The Three-Factor Rotation Construction: Why U · H_Had · P_br and Why in That Order

Section 3 of the main paper and Appendix A.4 construct the final per-layer rotation matrices as the product of three orthogonal factors:

$R_K = U_Q \; H_{\text{Had}} \; P_{\text{br}}$

$R_V = U_S \; H_{\text{Had}} \; P_{\text{br}}$

where U_Q ∈ ℝ^{d×d} is the eigenbasis of the key target covariance C_Q = (1/N) Σ q_n^⊤ q_n, U_S ∈ ℝ^{d×d} is the eigenbasis of the value target covariance C_S = (1/N) V^⊤ S^⊤ S V, H_Had ∈ ℝ^{d×d} is the normalized Walsh-Hadamard matrix (orthogonal, all entries ±1/√d), and P_br ∈ ℝ^{d×d} is a bit-reversal permutation matrix (a specific reordering of the columns).

The paper argues that each factor addresses a distinct failure mode of per-group low-bit quantization, and the factors must compose in this specific order for the mathematical properties to hold. Below, we walk through each factor, its geometric purpose, what would go wrong if it were omitted, and — critically — why the factors are not interchangeable.

Factor 1: U_Q and U_S — Align Channels with Importance Directions

The eigendecomposition of the target covariance is:

$C_Q = U_Q \Lambda_Q U_Q^\top, \quad \Lambda_Q = \text{diag}(\lambda_1, \ldots, \lambda_d), \quad \lambda_1 \geq \cdots \geq \lambda_d \geq 0$

Given the rotated keys \tilde{K} = K R_K, the key-side frozen-error surrogate objective (justified in Section 3.4.3) is:

$\tilde{\mathcal{L}}_K(R_K) = \text{tr}\left(R_K^\top C_Q R_K \; E_K\right)$

where E_K is a diagonal matrix of per-channel quantization residual variances (assumed frozen, i.e., independent of the rotation — this is the key simplifying assumption of Theorem 1). Setting R_K = U_Q diagonalizes the importance metric:

$R_K^\top C_Q R_K = \Lambda_Q$

What this means geometrically: after rotating by U_Q, the j-th channel of \tilde{K} = K U_Q is the projection of each key vector onto the j-th query-principal direction u_j. The eigenvalue λ_j = ∥Q u_j∥_2^2 measures how much total query energy lies along that direction. Quantization noise on channel j propagates to the attention logits with weight λ_j: noise on high-λ_j channels (where queries have large projections) is heavily amplified; noise on low-λ_j channels (where queries have near-zero projections) is almost invisible to attention.

Why this is the right first step: Theorem 1 (Section 3.4.3) proves that, under the assumption that the per-channel quantization errors are independent of the rotation (the "frozen residual" assumption) and that the residual variances are sorted ascending while the target covariance eigenvalues are sorted descending, R_K = U_Q is the orthogonal matrix that minimizes the surrogate \tilde{\mathcal{L}}_K. The proof uses a rearrangement-inequality argument: since λ₁ ≥ ··· ≥ λ_d and μ₁ ≤ ··· ≤ μ_d (residual variances), the minimum of Σ μ_i λ_{σ(i)} over permutations σ is achieved by the identity permutation, which pairs the largest eigenvalues with the smallest residual variances. This requires the channels to be in eigenvalue order, which is exactly what U_Q achieves.

What goes wrong if U_Q is omitted (i.e., using pure Hadamard without eigenbasis pre-rotation): The Hadamard transform mixes all channels equally, but it does not align the coordinate axes with the importance hierarchy. The importance metric R_K^⊤ C_Q R_K after pure Hadamard is not diagonal, and its diagonal entries are not uniform. Appendix A.4 measures this on layer 10 of Qwen3-4B-Thinking: the ratio max_i (R_K^⊤ C_Q R_K)_{ii} / mean_i(·) is 1.72 for pure Hadamard versus 1.00 for OSCAR (with U_Q pre-rotation). This means pure Hadamard leaves some channels 72% more important than average, so quantization noise on those channels is disproportionately damaging. Moreover, C_Q and K^⊤ K have uncorrelated eigenbases (alignment ≈ 0.09), so U_Q does not accidentally serve a dual purpose.

A critical empirical point: U_Q alone makes per-group quantization worse, not better. The worked example in Appendix A.4 shows this concretely. For a single token at layer 10 of Qwen3-4B-Thinking, the per-group dynamic range (what the INT2 min-max quantizer sees) increases from 44.81 in the raw channel basis to 37.85 after U_Q rotation, and the imbalance between the two groups of 64 channels actually worsens (range 7.16 vs. 37.85). This is because the K-variance is not aligned with the Q-variance — projecting K onto the Q-eigenbasis pushes outlier energy into low-eigenvalue directions. The per-group quantizer sees worse dynamic range, not better. This is precisely why U_Q must be followed by H_Had: the eigenbasis step achieves importance alignment (the C_Q diagonalization), but the quantization geometry must be fixed by the subsequent factors.

Analogous construction for values: U_S diagonalizes the value target covariance C_S = V^⊤ S^⊤ S V. After rotation by U_S, the j-th channel of \tilde{V} carries the projection of value vectors onto the j-th score-weighted principal direction, and eigenvalues ν_j measure how much attention-weighted value energy lies along that direction. The same rearrangement-inequality optimality argument applies.

Factor 2: H_Had — Equalize the Importance Metric Diagonally, Suppress Outliers on K and V

The Hadamard transform is applied after the eigenbasis rotation for two distinct purposes, one exact (mathematically guaranteed) and one heuristic (empirically validated).

Purpose 1 (exact): Diagonal equalization of the importance metric. Lemma 1 in Appendix A.4 states:

$\left(H_{\text{Had}}^\top \Lambda H_{\text{Had}}\right)_{ii} = \frac{1}{d} \text{tr}(\Lambda), \quad i = 1, \ldots, d$

for any diagonal matrix Λ. Applied to the post-U_Q importance metric, which is Λ_Q (diagonal because U_Q diagonalized C_Q):

$(H_{\text{Had}}^\top \Lambda_Q H_{\text{Had}})_{ii} = \frac{1}{d} \text{tr}(\Lambda_Q) = \frac{1}{d} \text{tr}(C_Q), \quad \forall i$

What this means: after composing U_Q with H_Had, every single channel has exactly the same importance weight in the rotated frame: the contribution of quantization noise on any channel to the total logit distortion is identical. The peaky eigenspectrum of C_Q (which can have λ₁ / λ_d ratios of > 1000) is completely flattened into a uniform diagonal. The worked example in Appendix A.4 confirms this exactly: max / mean = 1.00 for U_Q H_Had. This is a mathematical guarantee that depends only on Λ_Q being diagonal in the pre-Hadamard frame, which U_Q ensures.

Why Lemma 1 works: every entry of H_Had has magnitude 1/√d. For the diagonal entry (i,i) of H_Had^⊤ Λ H_Had, we have Σ_j H_Had[j,i]^2 Λ_{jj} = (1/d) Σ_j Λ_{jj} = (1/d) tr(Λ), independent of i. The sum of squared entries in each column of H_Had is exactly d · (1/d) = 1, and each squared entry is 1/d, so the weighted sum of eigenvalues is the same for every output channel.

Purpose 2 (heuristic): Outlier suppression on the K and V distributions. After U_Q, each row of the rotated keys is a projection of the original key vector onto the Q-eigenbasis. As shown in the empirical example, this concentrates outlier energy into a few channels (the low-eigenvalue tail of U_Q). Multiplying by H_Had produces Walsh-signed sums of all 128 eigen-projections, each scaled by 1/√d. A per-token outlier on a single eigen-direction is redistributed across all 128 output channels with magnitude proportional to 1/√d, shrinking the per-channel max by roughly the same factor. The worked example shows the per-group dynamic range collapsing from 37.85 (after U_Q) to 13.58 (after U_Q H_Had) — a 2.8× reduction. The per-group min-max quantizer then sees a much tighter range and can assign INT2 levels at finer granularity.

What goes wrong if H_Had is omitted: the ablation in Table 4 shows that removing H_Had (keeping only U_Q · P_br) drops mean accuracy from 70.01 to 51.74 on Qwen3-8B — a 18.27-point gap. The U_Q rotation alone makes per-group quantization geometry worse (as shown by the dynamic range example), so the Hadamard mixing is essential for making the quantizer's job feasible. Additionally, without H_Had, the importance metric remains peaked (max / mean ≈ 46.9 in the worked example), meaning a few channels are 47× more important than average.

Why the factors must compose as U_Q H_Had rather than H_Had U_Q: if the Hadamard were applied first, it would mix the raw K channels, but the resulting matrix H_Had^⊤ C_Q H_Had would not be diagonal because C_Q is not diagonal in the channel basis. Lemma 1 would not apply, and the importance metric would not be uniform. The specific order ensures that C_Q is first diagonalized (by U_Q), then the diagonal is equalized (by H_Had).

Factor 3: P_br — Balance Importance Across Quantization Groups

The per-group INT2 quantizer (Section 3.4.4) operates on contiguous blocks of G_K channels at a time (default G_K = 64, producing 2 groups per head for d = 128). After U_Q H_Had, the per-channel importance is exactly uniform (Lemma 1), but the distribution of the underlying eigenvalue spectrum across groups matters for quantization geometry. If one group contains channels dominated by the top few eigenvectors (which carry most of the signal energy), that group will have a larger dynamic range and coarser quantization than a group whose channels span the full range of eigenvalues.

What P_br does: it is a bit-reversal permutation. Let σ be the permutation that sorts eigenvalues descending (λ_{σ(0)} ≥ λ_{σ(1)} ≥ ···), and let β be the bit-reversal permutation on {0, 1, ..., d-1}. The permutation matrix P_K places the eigenvector with the k-th largest eigenvalue at position β(k). For d = 128, this yields the placement: top-1 → 0, top-2 → 64, top-3 → 32, top-4 → 96, top-5–8 → 16, 80, 48, 112, and so on.

The defining property: for any power-of-two group size G | d, the top-d/G eigenvectors land in d/G distinct groups, exactly one per group. This recursive balance holds at every coarser binary level. With G_K = 64 (two groups), the top-2 eigenvectors split into one per group. With G_K = 32 (four groups), the top-4 distribute one-per-group. This ensures that no single group inherits a disproportionate share of the high-energy eigenchannels, making the per-group dynamic range more uniform across groups.

Empirical effect: in the worked example at G_K = 64, P_br reduces the mean per-group max-min across 8000 tokens from 13.40 (U_Q H_Had) to 11.58 (OSCAR), translating to a 1.23× reduction in tr(E_K). The effect is significant but smaller than the U_Q and H_Had steps; Table 4 confirms this: removing P_br drops mean accuracy from 70.01 to 68.00, a 2.01-point gap that is real but less dramatic than removing U_Q (to 32.82) or removing H_Had (to 51.74).

Degenerate case: when G_K = d (one group per head). In this case, P_br is just a column permutation within the single group and has no effect on quantization — the set of values in the group is invariant under permutation. OSCAR then collapses to U_Q H_Had. The paper uses G_K = 64 in its worked example specifically to make the PBR effect non-trivial.

Why the three factors compose in this specific order:

1. U_Q first: diagonalizes C_Q into Λ_Q, creating the importance-aligned coordinate system. Applying it after the Hadamard would destroy the diagonalization.
2. H_Had second: equalizes the diagonal of Λ_Q to a constant via Lemma 1, which requires Λ_Q to be diagonal — a property guaranteed by step 1. It simultaneously suppresses per-token outliers on K.
3. P_br last: reorders channels so that the eigenvalue hierarchy is balanced across quantization groups. Since it is a permutation, it commutes with per-group quantization only in how channels are assigned to groups; applying it before H_Had would randomize the diagonal of Λ_Q before equalization, preventing Lemma 1 from firing.

---

3.4.3 Optimality Under the Frozen-Error Surrogate (Theorem 1)

The paper provides a theoretical justification that the heuristic covariance targets C_Q and C_S are optimal under a specific simplifying assumption. The theorem (Appendix A.6) is stated within a frozen-error framework, and its proof is a relatively straightforward application of the rearrangement inequality.

The frozen-error surrogate for keys. Starting from the empirical logit distortion for a single query sequence:

$\ell_K(R_K) = \sum_{i=1}^N \sum_{j=1}^i \left(q_i (k_j - \hat{k}_j)^\top\right)^2$

where \hat{k}_j = Q(k_j R_K) R_K^\top and the inner sum runs over causal positions j ≤ i. The paper approximates the position-dependent query covariance C_j = Σ_{i≥j} q_i^⊤ q_i by the global average C_Q = (1/N) Σ_{i=1}^N q_i^⊤ q_i, yielding the simpler objective (up to constant factor 1/N):

$\tilde{\mathcal{L}}_K(R_K) = \text{tr}\left(R_K^\top C_Q R_K \; E_K\right), \quad R_K^\top R_K = I_d$

where E_K = Σ_{j=1}^N e_j^{(K)}(R_K)^\top e_j^{(K)}(R_K) is the covariance of the quantization residuals e_j^{(K)}(R_K) = Q(k_j R_K) - k_j R_K in the rotated frame. The key simplification — and the assumption that makes the theorem tractable — is that E_K is treated as frozen, meaning independent of R_K. This is an approximation: in reality, the quantization residuals depend on the distribution of K R_K, which changes with R_K. The assumption is reasonable when the per-channel quantization errors are dominated by the per-group min-max scaling behavior, which depends primarily on the dynamic range within each group rather than on the specific orientation.

Theorem statement (key side). Let C_Q = U_Q Λ_Q U_Q^⊤ with Λ_Q = diag(λ₁, ..., λ_d), λ₁ ≥ ··· ≥ λ_d ≥ 0. Assume the frozen residual covariance is diagonal in the ambient basis: E_K = diag(μ₁, ..., μ_d) with μ₁ ≤ ··· ≤ μ_d. Then R_K = U_Q is a minimizer of \tilde{\mathcal{L}}_K(R_K) subject to R_K^⊤ R_K = I_d.

What the theorem physically assumes:

• Diagonal E_K: the quantization errors in different channels of the rotated space are uncorrelated. This is plausible if the per-group quantizer acts independently on each channel (which it approximately does, since groups partition channels and each channel's quantization is determined by its own value relative to the group min/max).
• Sorted μ_i: the residual variances μ_i are ordered ascending, meaning we have some channels with small quantization error and some with large. This ordering is without loss of generality — we can always reorder the diagonal entries.
• Sorted λ_i: the target covariance eigenvalues are sorted descending, with the convention that u_1 is the direction of largest query energy. This ordering is also without loss of generality.

Proof sketch. The proof proceeds in three steps:

1. Parameterize the rotation. Any orthogonal R_K can be written as R_K = U_Q Z where Z^⊤ Z = I_d. Substituting into the surrogate: \tilde{\mathcal{L}}_K = tr(Z^⊤ Λ_Q Z E_K).
2. Express as a linear function over doubly stochastic matrices. Writing Z = (z_{ij}), the i-th diagonal entry of Z^⊤ Λ_Q Z is Σ_{j=1}^d λ_j z_{ji}^2. Therefore \tilde{\mathcal{L}}_K = Σ_{i=1}^d Σ_{j=1}^d μ_i λ_j z_{ji}^2. The matrix W = (w_{ji}) with w_{ji} = z_{ji}^2 is doubly stochastic (rows and columns sum to 1, all entries non-negative) because Z is orthogonal. The objective is a linear function over the Birkhoff polytope (the set of doubly stochastic matrices), so the minimum is attained at an extreme point — a permutation matrix.
3. Apply the rearrangement inequality. The objective reduces to min_{Π ∈ P_d} Σ_{i=1}^d μ_i λ_{Π(i)}. Since λ₁ ≥ ··· ≥ λ_d and μ₁ ≤ ··· ≤ μ_d, the rearrangement inequality states that the sum Σ μ_i λ_{Π(i)} is minimized when Π pairs the largest λ's with the smallest μ's — that is, the identity permutation Π(i) = i. Therefore Z = I_d is a minimizer, and R_K = U_Q.

What the theorem provides and what it does not:

• Provides: a formal justification that U_Q is the optimal orthogonal basis for minimizing the frozen-error surrogate, given the diagonal-residual assumption. This explains why the heuristic of diagonalizing C_Q is mathematically principled rather than merely intuitive.
• Does not provide: a guarantee of optimality for the full autoregressive decoding process. The frozen-residual assumption is a significant simplification; in reality, E_K depends on R_K through the quantization scale computation, and the theorem does not account for the Hadamard and permutation factors (which are motivated by quantization geometry rather than the surrogate). The theorem also does not address the value-side objective or the interaction between key and value quantization. The paper acknowledges this limitation explicitly in Appendix D: "it is not yet a full proof of optimality for the complete autoregressive decoding process."

The value-side theorem is identical in structure, replacing C_Q with C_S = U_S Λ_S U_S^⊤, E_K with E_V = diag(η₁, ..., η_d) (sorted ascending), and concluding that R_V = U_S is optimal.

Why this theorem matters despite its limitations: it provides the intellectual justification for the paper's central design decision — choosing Q^⊤ Q and V^⊤ S^⊤ S V as the rotation targets rather than K^⊤ K and V^⊤ V. The theorem shows that, under the frozen-error simplification, the optimal basis is the eigenbasis of the target covariance. While the real system includes additional factors and operates under weaker assumptions, the theorem establishes a principled starting point that is verified empirically in the ablation (Table 4: Q^⊤ Q target outperforms K^⊤ K target by 38.89 points).

Connection to the Huber loss and PCA optimality. Although the paper does not make this connection explicit, the surrogate tr(R^⊤ C R E) with diagonal E is structurally similar to a weighted PCA objective, where the weighting is the residual variance. The theorem can be interpreted as: when the quantization noise is heteroscedastic (different variance per channel, as is typical with per-group min-max quantization), the optimal compression basis is the one that aligns high-importance directions (large λ) with low-noise channels (small μ). This is a generalization of standard PCA, which assumes homoscedastic noise (E ∝ I) and is optimal for mean-squared reconstruction error.

---

3.4.4 Scale Determination and Per-Token Clipping

With the rotations R_K and R_V fixed from calibration, the online serving path applies quantization in three steps per token: rotate into the OSCAR basis, clip outliers, and quantize with dynamic per-token per-group min-max asymmetric INT2 scaling. The paper's quantization mechanics (detailed in Appendix A.5) follow standard post-training quantization practice with specific adaptations for the KV-cache serving setting.

Step 1: Rotation into the OSCAR basis. For each new key and value token at position t:

$\tilde{k}_t = k_t R_K \in \mathbb{R}^{1 \times d}, \quad \tilde{v}_t = v_t R_V \in \mathbb{R}^{1 \times d}$

This is a simple matrix-vector multiply. The rotations R_K and R_V are fixed d × d orthogonal matrices loaded from the calibration output. No online adaptation occurs.

Step 2: Per-token percentile clipping. Before quantization, each token's rotated vector is clipped independently to control outlying values that would otherwise inflate the quantization range:

$\tau_t^{(K)} = \text{quantile}_{\rho_K}\left(\{|\tilde{k}_{t,c}|\}_{c=1}^d\right)$

$\tilde{k}_t^{\text{clip}} = \text{clip}\left(\tilde{k}_t, -\tau_t^{(K)}, \tau_t^{(K)}\right)$

where ρ_K ∈ (0, 1] is the clip ratio hyperparameter. The default values are ρ_K = 0.96 for keys and ρ_V = 0.92 for values, estimated per layer from rotated calibration activations rather than tuned on downstream tasks.

What clipping does: for each token, it computes the ρ_K-th percentile of the absolute values across the d channels and clamps all values to [-τ, τ]. Values outside this range are set to the bound. This protects against a single channel having an extreme value (say, 30× the median) that would force the quantization scale to be very coarse for all channels in that group.

Why ρ_K = 0.96 and ρ_V = 0.92: these values are derived from the calibration set, not hand-tuned. The paper reports (Section 5.1, calibration paragraph) that the calibration-derived clip thresholds are close to the optimal grid point: on Qwen3-4B-Thinking, the calibration point (c_K, c_V) = (0.96, 0.92) achieves mean accuracy 71.86, within 0.15 points of the best grid point (70.74 at (0.92, 0.98)) in the sweep of Table 6. The fact that values receive more aggressive clipping (lower ρ) than keys is consistent with the finding in prior work (Hariri et al., 2025) that keys are generally more sensitive to quantization than values.

Step 3: Per-token per-group asymmetric affine INT2 quantization. The clipped rotated vector is partitioned into groups of G_K consecutive channels (for keys) or G_V consecutive channels (for values). The default group size is 128 for both, though the paper also reports results with group sizes 64 and 32. For each token t and each group g:

$a_{t,g}^{(K)} = \min_c \tilde{k}_{t,g,c}^{\text{clip}}, \quad b_{t,g}^{(K)} = \max_c \tilde{k}_{t,g,c}^{\text{clip}}$

$s_{t,g}^{(K)} = \frac{b_{t,g}^{(K)} - a_{t,g}^{(K)}}{q_{\max}}, \quad z_{t,g}^{(K)} = -\frac{a_{t,g}^{(K)}}{s_{t,g}^{(K)}}$

where q_max = 2^b - 1 = 3 for b = 2 bits. The quantized integer representation is:

$Q^+\left(\tilde{k}_{t,g}^{\text{clip}}\right) = \text{clip}\left(\left\lfloor \frac{\tilde{k}_{t,g}^{\text{clip}}}{s_{t,g}^{(K)}} + z_{t,g}^{(K)} \right\rceil, \; 0, \; q_{\max}\right) \in \{0, 1, 2, 3\}^{G_K}$

and the dequantized floating-point reconstruction is:

$Q\left(\tilde{k}_{t,g}^{\text{clip}}\right) = s_{t,g}^{(K)} \cdot \left(Q^+\left(\tilde{k}_{t,g}^{\text{clip}}\right) - z_{t,g}^{(K)}\right) \in \mathbb{R}^{G_K}$

What this quantization scheme does:

• Asymmetric: the zero point z is not constrained to be 0, which means the representable values {0, s, 2s, 3s} (before adding the zero-point offset) can cover the full observed range [a, b] without wasting levels on the negative side if the distribution is skewed positive. This is important because rotated KV activations can be asymmetric.
• Affine: the mapping from floating-point to integer is a linear transformation (scale and shift) followed by rounding. This is the standard uniform quantization scheme.
• Per-token, per-group: the scale s and zero point z are computed dynamically for each token and each group of G_K channels independently. This means a token with a small dynamic range in one group gets fine-grained quantization (small s), while a token with a large dynamic range in another group gets coarser quantization (large s). The dynamic range is determined by the clipping step, so the clip thresholds indirectly control the quantization granularity.
• INT2 packing: for storage, the four 2-bit values per group are packed into a single byte (4 × 2 bits = 8 bits). The paper reports effective bits per element (BPE) at 128k context length including the BF16 scale and zero point metadata (2 bytes per group of 128 elements = 0.125 extra bits per element) plus the BF16 sink and recent tokens (64 + 256 = 320 tokens out of 131072 total, contributing approximately 0.02 extra BPE), for a total of approximately 2.28 BPE.

Step 4: Inverse rotation for attention. During decode, when the quantized keys and values are read from the INT2 cache pages, they are first dequantized in the rotated frame (applying the stored s and z to unpack the 2-bit integers), then inverse-rotated back to the original coordinate system:

$\hat{k}_t = Q\left(\tilde{k}_t^{\text{clip}}\right) R_K^\top \in \mathbb{R}^{1 \times d}, \quad \hat{v}_t = Q\left(\tilde{v}_t^{\text{clip}}\right) R_V^\top \in \mathbb{R}^{1 \times d}$

These reconstructed vectors are what the attention computation actually consumes.

Mixed-precision sink and recent windows. Not all tokens are quantized. The paper's KV cache layout (Section 4) preserves two short high-precision windows:

• Sink tokens: the first S₀ = 64 tokens, which behave as attention sinks (Xiao et al., 2024) — the model learns to dump attention mass into these early positions as a mechanism for ignoring irrelevant context, making them disproportionately important for attention score normalization.
• Recent tokens: the most recent W = 256 tokens before the current decoding position, which are the most likely to be attended to by the current query due to recency bias.

The total BF16-protected fraction at 128k context length is (S₀ + W) / 128k = (64 + 256) / 131072 = 0.24% of tokens, contributing a negligible memory overhead while providing substantial accuracy protection. The ablation in Table 5 shows that (0, 0) (no protected tokens) yields 0% accuracy on Qwen3-4B-Thinking — the model collapses completely — while (64, 256) reaches 71.86 mean accuracy. Larger windows (128, 512) and (256, 1024) provide diminishing returns (72.96 and 73.08, respectively) at increasing BF16 memory cost.

---

3.4.5 Online Serving Integration and Kernel Design

The paper's production deployment (Section 4, Appendix B Algorithm 1) integrates OSCAR into the SGLang serving stack (Zheng et al., 2024) with three design principles: (1) preserve standard paged-attention memory layout for the quantized cache, (2) fuse quantization/dequantization operations into existing Triton decode kernels to avoid extra memory traffic, and (3) absorb the value rotation into the output projection weight to reduce computation.

KV cache layout. At logical cache position t during decoding, the cache consists of three contiguous segments:

$\underbrace{[1, S_0]}_{\text{BF16 sink}} \;\|\; \underbrace{[S_0 + 1, t - W]}_{\text{INT2 history}} \;\|\; \underbrace{[t - W + 1, t]}_{\text{BF16 recent}}$

The BF16 segments store unrotated, unquantized key and value vectors in standard FP16 format. The INT2 history segment stores rotated, clipped, and quantized vectors packed 4× per byte (four 2-bit values per byte), plus per-group scale and zero-point metadata (2 bytes per group of 128 elements).

Prefill kernel. During prefill (processing the input prompt), a fused Triton kernel writes each token's KV rows through the full OSCAR pipeline in a single GPU kernel launch:

1. Compute \tilde{k}_t ← k_t R_K and \tilde{v}_t ← v_t R_V (matrix-vector multiply).
2. Compute clip threshold τ_t^{(K)} ← quantile_{ρ_K}(|\tilde{k}_t|) via an efficient percentile approximation.
3. Clip: \tilde{k}_t^{\text{clip}} ← clip(\tilde{k}_t, -τ_t^{(K)}, τ_t^{(K)}).
4. Compute per-group min/max, scale, and zero point for INT2.
5. Quantize and pack: k_t^+ ← Q_2^+(\tilde{k}_t^{\text{clip}}) into 2-bit integers, pack 4 per byte.
6. Write packed bytes and scale/zero metadata to the paged INT2 cache.

Tokens in the sink and recent windows are written as BF16 to separate cache regions. The paper notes that this is a single fused kernel, so the rotated activations never leave the GPU register file or shared memory — they are produced, clipped, quantized, and packed without intermediate writes to global memory.

Decode attention kernel. During autoregressive decoding, each new token triggers an attention computation over the mixed-precision cache. The paper's fused Triton kernel (Section 4, "Decoding Attention Kernel") works as follows:

1. Cache index partitioning: the kernel receives the logical cache indices for the current request. It partitions them into three ranges: BF16 sink indices, INT2 history indices, and BF16 recent indices. The INT2 segment typically accounts for >99% of the indices at long context.
2. INT2 segment processing (new kernel): a specialized kernel reads the packed INT2 bytes, unpacks them to 2-bit integers, dequantizes to FP16 using the stored per-group scales and zero points (\hat{\tilde{k}} = s · (\text{unpacked} - z)), then applies the inverse rotation \hat{k} ← \hat{\tilde{k}} R_K^⊤ to recover the approximate key vector in the original coordinate system. The same process applies to values.
3. BF16 segment processing (existing kernel): the standard FlashAttention decode kernel processes the sink and recent BF16 cache segments without any inverse rotation (they were stored unrotated).
4. Partial attention merge: the existing FlashDecoding two-kernel design (Dao et al., 2023) is reused. The first kernel computes partial attention logits and value-weighted sums for each segment; the second kernel merges these partial results with online softmax rescaling. OSCAR introduces an additional kernel launch for the INT2 segment but piggybacks on the existing merge kernel, so the total kernel count increases only slightly.

Why this is efficient: the INT2 segment is processed with dramatically reduced memory bandwidth — reading 2-bit values instead of 16-bit values reduces KV-cache memory traffic by approximately 8×. The dequantization and inverse rotation are compute-bound operations that run on the GPU's tensor cores or CUDA cores while the memory system is less saturated. Table 8 (per-decode kernel profiling on Qwen3-8B) shows this concretely: at batch size 1, BF16 attention takes 3.8 ms (37.9% of total decode time) while OSCAR attention takes 1.3 ms (16.8%) — a 2.9× reduction in attention latency that directly reflects the reduced memory bandwidth demand. The quantization overhead (0.5 ms, 5.9% of total) is small relative to the attention savings.

Value rotation absorption into the output projection. An important optimization for latency: the paper absorbs the value inverse rotation R_V^⊤ into the attention output projection weight W_O. Recall the attention output computation:

$O = \text{softmax}\left(\frac{Q K^⊤}{\sqrt{d}}\right) V W_O$

With quantized values, this becomes:

$\hat{O} = \text{softmax}\left(\frac{Q \hat{K}^⊤}{\sqrt{d}}\right) \hat{V} W_O = \text{softmax}\left(\frac{Q (Q(K R_K) R_K^⊤)^⊤}{\sqrt{d}}\right) Q(V R_V) R_V^⊤ W_O$

The value inverse rotation R_V^⊤ is a fixed matrix that can be pre-composed with W_O offline: W_O' = R_V^⊤ W_O. This eliminates the value inverse rotation at decode time entirely — the dequantized values in the rotated frame \hat{\tilde{V}} are used directly in the attention-weighted sum, and the output is computed as \hat{O} = S \hat{\tilde{V}} W_O'. This absorption is possible because W_O is a learned weight matrix, so pre-multiplying it by R_V^⊤ simply produces an equivalent reparameterization of the output projection. Keys cannot be absorbed in this way because the key rotation affects the attention logits Q K^⊤, which involve the query-key interaction, not just a post-hoc projection.

Window management during decoding. As decoding advances, the oldest token in the recent window must be demoted to the INT2 history to maintain the W = 256 recent-window invariant. This demotion is handled by a fused Triton kernel that:

1. Reads the oldest recent-window token's BF16 key and value from the recent cache.
2. Applies the same rotate–clip–quantize–pack pipeline as the prefill kernel.
3. Writes the packed INT2 bytes to the history cache page.
4. The token's BF16 entry in the recent cache is freed for the next new token.

This demotion happens once per decode step (since each step produces exactly one new token, and the window slides by one), so the overhead is amortized over the decode step's other computations.

Prefix cache compatibility. Because OSCAR's quantized representation is a uniform dense INT2 layout with standard paged-attention page sizes, it is fully compatible with SGLang's radix tree-based prefix caching (Zheng et al., 2024). When multiple requests share a common prompt prefix, the quantized KV pages for that prefix are physically shared and reused. The paper's Figure 6 shows that prefix-cache hit ratio significantly improves throughput: going from 0% to ~100% cache hits approximately doubles per-GPU throughput at batch size 32 on Qwen3-4B-Thinking and GLM-4.7-FP8, and OSCAR tracks the efficiency frontier across all cache regimes.

Kernel profiling data (Table 8) contextualized. The per-step decode latency breakdown on Qwen3-8B at batch size 1: BF16 total is 9.9 ms (GEMM 5.6 ms, Attention 3.8 ms, Other 0.6 ms), while OSCAR total is 8.0 ms (GEMM 5.6 ms, Attention 1.3 ms, Quant 0.5 ms, Other 0.6 ms). The 1.9 ms savings comes entirely from attention memory bandwidth reduction; the quant overhead (0.5 ms) is small and the GEMM (the model's feedforward computation) is unchanged. At batch size 32, the attention dominates: BF16 attention is 17.0 ms (72.7% of 23.4 ms total) while OSCAR attention is 8.5 ms (54.6% of 15.5 ms total) — a 2.0× attention speedup that translates to a 1.51× overall decode speedup. At batch size 128, BF16 runs out of memory, while OSCAR handles it with 31.9 ms attention time.

Memory and batch scalability (Figure 4, right, and Figure 5). The 8× KV-cache compression enables substantially larger batch sizes under the same memory budget. On Qwen3-4B-Thinking with 100k-token inputs, OSCAR supports up to 28 concurrent requests on a single H100 while BF16 runs out of memory at smaller batch sizes. Job-level throughput (tokens/s) continues to scale with batch size for OSCAR, reaching 6.17× BF16 at batch size 32 on Qwen3-4B-Thinking and 7.83× on GLM-4.7-FP8. The paper emphasizes that this throughput improvement comes from two mechanisms: (1) each decode step is faster due to reduced KV bandwidth (the per-step speedup), and (2) larger batches are feasible, which increases GPU utilization (the batch-size scaling).

Calibration cost and one-time processing. The offline calibration pass (Section 5.1) processes a single small dataset (default: 8878 tokens from GPQA-Diamond MMLU-style prompts, approximately 8k tokens) and dumps per-layer Q, K, V, and S activations. The eigendecompositions are computed on CPU for d × d matrices where d = 128 (head dimension), so the computational cost is trivial (a few seconds per layer). The clip thresholds are estimated from the rotated calibration activations as simple percentiles. All parameters are reused across all benchmarks and tasks without re-calibration — the same rotations and thresholds serve for AIME25, GPQA, HumanEval, LiveCodeBench, MATH500, and RULER. The calibration data regime study (Table 7) shows that the method is robust to calibration volume and domain: varying from 2k to 32k tokens and across MMLU, WikiText, and GPQA-Diamond domains produces mean accuracies within 2.73 points of each other (68.28 to 71.01), and the default 8k GPQA-Diamond setting (71.01) is close to the best setting (70.88 at 16k MMLU).

4. Key Insights and Innovations

Innovation 1: Attention-Consumed Covariance as the Correct Target for KV-Cache Rotation

The paper's deepest conceptual contribution is reframing what it means for a rotation to be "optimal" for KV-cache quantization. Prior rotation-based methods — exemplified by QuaRot (Ashkboos et al., 2024) — implicitly optimize raw Euclidean reconstruction error. They apply data-oblivious Hadamard transforms whose purpose is to flatten activation distributions so that a uniform quantizer incurs less mean-squared error when reconstructing K and V. This is the natural default: if you're going to quantize a tensor, you want the quantized version to be as close as possible to the original in L2 distance. Every post-training quantization method from GPTQ to SmoothQuant operates on some version of this principle.

OSCAR argues that this default is wrong for KV caches in a specific, provable way. The downstream attention computation does not consume K and V through their Euclidean reconstruction error. It consumes keys through the quadratic form Q K^⊤ and values through the attention-weighted aggregation S V. A rotation that minimizes ∥K − Q(K)∥_F^2 can produce arbitrarily large attention-logit errors if the quantization noise lands in directions where queries have large energy. Conversely, a rotation that allows larger raw reconstruction error in some directions but steers the noise into query-null directions can achieve much better attention fidelity.

This is not an incremental refinement of existing rotation methods. It is a fundamental shift in the optimization target. The paper formalizes this through the trace identities in Section 2: the key-side logit distortion is tr((K − K̂) Q^⊤ Q (K − K̂)^⊤), and the value-side output distortion is tr((V − V̂)^⊤ S^⊤ S (V − V̂)). These identities are mathematically trivial (cyclic permutation of the trace), but their interpretation as design principles — that the rotation should diagonalize Q^⊤ Q and V^⊤ S^⊤ S V, not K^⊤ K and V^⊤ V — is what distinguishes OSCAR conceptually. Prior work in the covariance-aware compression literature (GPTQ, SVD-LLM, CorDA) used Hessian or activation covariance to guide weight compression, but nobody had extended this principle to the specific quadratic forms that arise in attention's consumption of KV caches.

The ablation in Table 4 crystallizes how much this matters. Replacing OSCAR's Q^⊤ Q / V^⊤ S^⊤ S V PCA targets with raw K^⊤ K / V^⊤ V PCA targets — keeping everything else identical (same Hadamard, same bit-reversal, same quantization scheme, same sink/recent windows) — drops mean accuracy on Qwen3-8B from 70.01 to 31.12, a 38.89-point gap. This is not a small refinement; it is the difference between functional and collapsed INT2 quantization. The empirical confirmation that Q^⊤ Q and K^⊤ K have nearly uncorrelated eigenbases (alignment ≈ 0.09, indistinguishable from random at d = 128) drives home that this is not a minor detail — the two targets point in genuinely different directions, and choosing the right one is the primary determinant of success.

This innovation also resolves a tension in the prior literature. Why do QuaRot-style Hadamard rotations work well at 4 bits but collapse at 2 bits? The paper's framework provides an answer: at 4 bits, the quantization error is small enough that even a suboptimal rotation basis keeps the attention error within acceptable bounds. At 2 bits, with only four quantization levels, the error is so large that the basis matters decisively — the noise must be steered into low-importance directions, and a data-oblivious transform cannot do this steering because it does not know which directions are low-importance. The framework thus predicts a sharp phase transition in the required sophistication of the rotation as precision decreases, which is exactly what the experiments show: QuaRot-INT2 collapses to near-zero accuracy across all benchmarks, while OSCAR's attention-aware rotation recovers most of the BF16 accuracy.

---

Innovation 2: Decomposing Rotation into Three Independent Geometric Objectives with Guaranteed Properties

Beyond choosing the right target covariance, the paper's second conceptual contribution is recognizing that a single orthogonal transform cannot simultaneously satisfy all the requirements of INT2 quantization. Prior rotation-based methods (QuaRot, SpinQuant) applied a single fixed matrix and evaluated it holistically. OSCAR decomposes the problem into three separable geometric objectives — importance alignment, importance equalization, and cross-group importance balancing — and constructs a composed rotation R = U · H_Had · P_br where each factor addresses exactly one objective with a specific mathematical guarantee.

This decomposition is intellectually distinctive because it demonstrates why pure learned rotations (SpinQuant) or pure Hadamard transforms (QuaRot) are insufficient at extreme precision. A learned rotation optimized end-to-end for quantization error can discover importance alignment if the training signal reflects downstream attention error, but the paper's empirical evidence (SpinQuant collapses at INT2, per Appendix C) suggests that the optimization landscape is too difficult at 2 bits for end-to-end learning to reliably find the right basis. A pure Hadamard transform provides importance equalization (in the sense of uniform channel energy) and outlier suppression, but it cannot provide importance alignment because it is data-oblivious. A pure eigenbasis rotation provides importance alignment but actually worsens quantization geometry — the worked example in Appendix A.4 shows per-group dynamic range increasing after U_Q rotation because the K-variance gets pushed into low-eigenvalue directions. Only the composition of all three factors, in the specific order that allows each to do its job without interfering with the others, achieves both accuracy and deployability.

The guarantees are worth articulating precisely because they explain why the composition works rather than merely that it works. Lemma 1 (Appendix A.4) provides an exact guarantee: after U_Q diagonalizes C_Q into Λ_Q, the Hadamard conjugate H_Had^⊤ Λ_Q H_Had has a uniform diagonal equal to tr(C_Q)/d for every channel. This is not an empirical observation or a heuristic — it is a mathematical theorem that follows from the property that all entries of H_Had have equal magnitude. This guarantee means that, regardless of how peaky the query covariance spectrum is (and the paper shows λ₁/λ_d ratios can exceed 1000), after the U_Q H_Had composition, every channel contributes identically to the importance-weighted error. The bit-reversal permutation then provides a recursive balancing guarantee across quantization groups: at any power-of-two group size, the top eigenvectors are distributed one-per-group, ensuring that no single group inherits a disproportionate share of the signal energy.

This three-factor decomposition is not an incremental improvement on existing rotations — it is a fundamentally different way of thinking about what a rotation for quantization must achieve. Prior work treated rotation design as a single optimization problem: find a matrix that makes the quantized output close to the original. OSCAR reframes it as three independent subproblems — align with importance (linear algebra), equalize importance (spectral property of Hadamard), balance across groups (combinatorial permutation) — each with its own solvable structure. This modularity makes the method interpretable (each factor can be ablated and its contribution measured, as in Table 4) and robust (the factors do not interfere because they operate on orthogonal properties of the transformed distribution).

---

Innovation 3: Verifier-Free Calibration — One Pass, All Benchmarks, No Task-Specific Tuning

A practical but conceptually important contribution is OSCAR's calibration methodology. The entire system — two rotation matrices per layer, a pair of clip thresholds, and a quantization group size — is determined from a single forward pass over 8k tokens of MMLU-style calibration data. There is no iterative optimization, no gradient-based learning, no validation-loop hyperparameter tuning, and no per-task adaptation. The same calibration parameters are used for GPQA (graduate-level science QA), HumanEval (code generation), LiveCodeBench (contemporaneous coding problems), AIME25 (competition math), MATH500 (diverse math problems), and RULER (long-context retrieval at up to 128k tokens).

This matters because it establishes that attention-aware covariance structure is a stable, generalizable property of the model's attention patterns, not a brittle artifact of calibration data. The calibration data regime study (Table 7) supports this: varying calibration from 2k to 32k tokens, and across MMLU, WikiText, and GPQA-Diamond domains, produces mean accuracy varying from 68.28 to 71.01 — a range of only 2.73 points across an 8× calibration volume range and three distinct text domains. Even 2k tokens of MMLU prompts (a tiny calibration set by modern standards) yields 68.28 mean accuracy, recovering over 90% of the best setting's performance. The clip threshold grid sweep (Table 6) confirms that the calibration-derived (c_K, c_V) = (0.96, 0.92) is within 0.15 points of the best grid point on Qwen3-4B-Thinking, meaning the offline percentile estimation is essentially optimal.

This is not just a convenience feature — it has methodological significance. Many post-training quantization methods (GPTQ, AWQ, SpinQuant) require careful calibration set selection and can be sensitive to domain mismatch between calibration and evaluation. The fact that OSCAR's covariance targets — derived from attention's own computation graph rather than from external labels or loss functions — transfer robustly across tasks suggests that these targets capture something fundamental about how the model routes information through attention, not something specific to the surface statistics of the calibration text. The paper does not make this claim explicitly, but it is a natural interpretation of the robustness results.

---

Innovation 4: Verifier Over-Optimization as the Limiting Factor for Test-Time Compute, Not Search Algorithm Design

A subtler contribution emerges from the paper's negative results and ablation patterns. The paper shows that even with optimal importance-aligned rotations, INT2 quantization still requires (1) BF16 protection for sink and recent tokens, (2) per-token percentile clipping, and (3) asymmetric quantization with per-group dynamic scaling. Remove any of these, and accuracy collapses — Table 5 shows (S₀, R) = (0, 0) yields zero accuracy, and the ablation in Table 4 shows that removing clipping and sink/recent drops the "no rotation" configuration to 4.23 mean accuracy. This suggests that rotation alone, even with the optimal covariance target, is not sufficient for INT2 — the quantization infrastructure (mixed precision, clipping, dynamic scaling) is equally essential.

This is conceptually significant because it mirrors a finding from a different literature. In the test-time compute scaling literature (Snell et al., 2024), the primary bottleneck is not the search algorithm but the quality of the verifier — beam search and lookahead search over-optimize the verifier signal at the expense of true correctness, and compute-optimal allocation works by routing easy problems away from aggressive optimization. OSCAR's design embodies a structurally similar insight for KV-cache quantization: the primary bottleneck is not the rotation target or the quantization algorithm individually, but their interaction. The rotation target determines where the noise goes (steering it into low-importance directions), and the quantization infrastructure determines how much noise there is (clipping outliers, protecting critical tokens, adapting scales per group). Both are necessary; neither is sufficient alone. The paper's key design decision — pairing attention-aware rotations with standard quantization infrastructure rather than inventing a new quantizer — reflects this diagnosis. The innovation is recognizing that the rotation target and the quantization scheme are complementary, separable concerns, and that improving the target (the contribution of this paper) is orthogonal to improving the quantizer (a separate research direction).

Furthermore, the paper's approach to mixed precision — protecting a fixed number of sink and recent tokens in BF16 rather than adaptively selecting which tokens to protect — is a deliberate simplification motivated by deployability. Adaptive token-level precision assignment (as in KIVI, Kitty, or ZipCache) requires per-token metadata and complicates the memory layout. OSCAR's fixed S₀ = 64, W = 256 windows are a heuristic that is "dumb" in the sense of being data-independent, but they are compatible with paged-attention and fused kernels. The fact that this simple heuristic works as well as it does (Table 5 shows diminishing returns beyond (64, 256)) suggests that the attention-sink and recency-bias phenomena are robust enough across models and tasks that a fixed window is sufficient. This is a negative result with positive implications: it means you don't need adaptive token-level precision, which would break the uniform dense representation that makes deployment practical.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmarks are five reasoning and coding tasks: AIME25 (competition mathematics, 30 problems), GPQA-Diamond (graduate-level science QA, 198 problems), HumanEval (code generation, 164 problems), LiveCodeBench v6 (contemporaneous coding problems, 171 evaluation instances), and MATH500 (diverse math problems, 500 problems). Long-context robustness is tested with RULER-NIAH (retrieval with needle-in-a-haystack) from 4k to 128k context lengths. All accuracy evaluations use a maximum generation length of 32,768 tokens and run end-to-end inside SGLang with the same execution path as the system measurements. LiveCodeBench v6 is evaluated with the same 32k cap, which truncates some long code-generation outputs, so reported LCB numbers are lower than they would be under a longer generation budget.

• Base model(s). Four model configurations are evaluated to cover different scales and architectures: Qwen3-4B-Thinking-2507 (a small reasoning model with thinking mode), Qwen3-8B (a mid-sized dense model), Qwen3-32B (a larger dense model), and GLM-4.7-FP8 (a frontier-scale 358B-parameter model with FP8 weights). These span from 4B to 358B parameters, allowing the paper to test whether INT2 robustness scales with model size — a question the results answer affirmatively, as larger models prove increasingly resilient to aggressive quantization.

• Metrics. The primary metric is task-specific accuracy computed as the fraction of problems for which the model's generated answer matches the ground truth. For reasoning benchmarks (AIME25, GPQA, MATH500), the generated text is extracted and compared against the reference answer using standard answer-parsing and grading functions. For code benchmarks (HumanEval, LiveCodeBench), generated code is executed against test cases and scored on pass/fail. For RULER-NIAH, accuracy is the fraction of retrieval queries where the model correctly extracts the needle from the long context. For throughput experiments, the paper reports per-user tokens per second (U, tok/s) and per-GPU tokens per second (G, tok/s). For kernel profiling, per-step decode latency is reported in milliseconds. The paper also reports KL divergence between the FP16 attention distribution and the quantized attention distribution as a diagnostic metric (Figure 3), computed as KL(p_FP16 ∥ p_q) over teacher-forced decode steps.

• Baselines. The paper uses two groups of baselines. Group A (channel-wise methods, Table 1 only): KIVI-KV2 (Liu et al., 2024, 2-bit asymmetric KV quantization with per-channel scales), KIVI-KV2* (a variant), and Kitty (Xia et al., 2025, 2-bit KV quantization with channel-wise precision boost). These methods require residual buffers, channel-wise scales, or custom page layouts; the paper reports their published 32K-generation AIME25 numbers but does not re-implement them because "we do not have paged/fused kernels for them at 32K generation length." Group B (rotation-based methods, full 32K evaluation): BF16/FP16 (the uncompressed upper bound), Naive INT2 (direct 2-bit quantization without rotation or clipping), Saw-INT4 (Jia et al., 2026, block-diagonal Hadamard rotation at 4 bits), QuaRot-INT2 (Ashkboos et al., 2024, Hadamard-only rotation at 2 bits), and TurboQuant (Zandieh et al., 2025, data-oblivious online vector quantization with random rotations, evaluated using the official vLLM implementation from PR #38479 with mixed precision disabled for fairness). Effective bits per KV element (BPE) are reported for all methods at 128k context length, counting the INT payload plus BF16 scale/zero metadata (group size 128) plus BF16 initial tokens where applicable.

• Generation budget / compute accounting. All INT2 methods are compared at the same bit width (2 bits for the quantized portion) rather than the same FLOP count, since the primary claimed benefit of INT2 is memory reduction. The key budget metric is BPE (bits per KV element), which accounts for the INT2 payload (2 bits), the BF16 scale and zero point metadata (2 bytes per group of 128 elements ≈ 0.125 extra bits per element), and the BF16 sink/recent tokens (64 + 256 = 320 out of 131,072 tokens at 128k context ≈ 0.02 extra bits per element), for a total of approximately 2.28 BPE for OSCAR. Throughput comparisons are at fixed context lengths (30k, 60k, 100k tokens) and batch sizes (1, 8, 32), with all methods running in the same SGLang serving framework on the same hardware. The paper also reports FLOPs-matched comparisons indirectly through throughput — higher throughput at the same accuracy means fewer GPU-hours per query.

• Cross-validation / statistical protocol. All Qwen-family accuracy results are reported as mean ± standard deviation over 5 independent runs with different random seeds (temperature T = 0.6, top-p = 0.95, top-k = 20). GLM-4.7-FP8 results use 3 independent runs (T = 1.0, same top-p/top-k). TurboQuant entries are single-run results because "its vLLM path is too slow for repeated 32K evaluations under our time budget." The calibration parameters (rotation matrices and clip thresholds) are estimated once from a small MMLU-style calibration set of 8k tokens and reused for all benchmarks without task-specific adaptation. RULER results use 3 seeds except GLM-4.7-FP8 (1 run, as the model is "very stable"). No cross-validation over calibration data splits is performed — the calibration set is fixed, and generalization is tested by evaluating on held-out benchmarks. Since the calibration set contains no examples from any evaluation benchmark, this tests out-of-domain generalization of the estimated covariance structure.

---

Main Quantitative Results

Accuracy at 32K Generation Across Models and Benchmarks

The central accuracy results are in Table 2. The headline finding is that OSCAR is the only near-2-bit method that remains close to the BF16 accuracy frontier under 32K-generation evaluation across all four model configurations.

On Qwen3-4B-Thinking-2507, BF16 achieves 75.64 mean accuracy across the five benchmarks. OSCAR at 2.28 BPE achieves 71.86 mean accuracy, a drop of only 3.78 points. In contrast, QuaRot-INT2 collapses to 1.40 mean accuracy (drop of 74.24 points), Naive INT2 yields 0.00 (complete collapse), and TurboQuant without mixed precision yields 31.74 (drop of 43.90 points). Saw-INT4 achieves 73.11 mean accuracy but uses 4.25 BPE — nearly twice the bit budget. Breaking down by benchmark: OSCAR achieves 64.95 on GPQA (vs. 67.27 BF16), 92.24 on HumanEval (vs. 94.05), 45.38 on LiveCodeBench v6 (vs. 48.66), 64.00 on AIME25 (vs. 74.67), and 92.75 on MATH500 (vs. 93.55). The largest gap is on AIME25 (−10.67 points from BF16), consistent with the observation that reasoning-intensive tasks expose more accumulated quantization error over long generation.

On Qwen3-8B, the pattern is similar but the gap shrinks. BF16 mean is 70.84. OSCAR at 2.28 BPE achieves 69.42, a drop of only 1.42 points. QuaRot-INT2 drops to 10.14 (−60.70 points). Saw-INT4 at 4.25 BPE achieves 69.97 (−0.87 points) — slightly better than OSCAR but at nearly 2× the bit budget. Notably, QuaRot-INT2 on Qwen3-8B does not collapse as completely as on Qwen3-4B-Thinking (10.14 vs. 1.40 mean accuracy), suggesting the larger base model is somewhat more robust to quantization noise, though still far from usable.

On Qwen3-32B and GLM-4.7-FP8 (358B), INT2 quantization becomes dramatically easier. On Qwen3-32B, BF16 achieves 74.19 mean accuracy. OSCAR at 2.28 BPE achieves 74.17 — essentially tied with BF16 (drop of only 0.02 points). Saw-INT4 at 4.25 BPE achieves 74.43 (+0.24 points, marginally above BF16). Even TurboQuant holds at 71.99 (−2.20 points). On GLM-4.7-FP8 (358B), BF16 achieves 77.89. OSCAR achieves 78.16 (+0.27 points, marginally above BF16). Saw-INT4 achieves 77.95 (+0.06). TurboQuant achieves 78.15 (+0.26). Even Naive INT2 achieves 60.49 (−17.40), and QuaRot-INT2 achieves 75.14 (−2.75). At these model scales, INT2 quantization is nearly lossless regardless of method — the model's representations are sufficiently robust that aggressive compression causes minimal downstream accuracy degradation.

Key observation on model scale: The paper demonstrates a clear trend that INT2 robustness increases with model size. The BF16-to-OSCAR gap shrinks from −3.78 (4B) to −1.42 (8B) to −0.02 (32B) to +0.27 (358B). This is not merely a consequence of larger models having more capacity to absorb noise — it may reflect that larger models learn more disentangled representations where attention-covariance structure is more sharply defined, making the PCA-based rotation more effective. The paper does not investigate this mechanism, but the empirical trend is unambiguous.

Long-Context Robustness on RULER-NIAH

Table 3 and Figure 3 evaluate how quantized KV caches hold up as context length grows from 4k to 128k tokens. The task is RULER-NIAH (needle-in-a-haystack retrieval), which explicitly tests whether quantized history tokens can be attended to correctly at long range. Each number is averaged over 3 seeds.

On Qwen3-4B-Thinking-2507, BF16 maintains 100% accuracy at 4k and 8k, dropping gradually to 85.3% at 64k and 81.0% at 128k — the model itself struggles with very long retrieval even without quantization. QuaRot-INT2 collapses to near-zero at all lengths: 0% at 4k, 0.8% at 8k, 0% at 16k and 32k, with a spurious 15.6% at 64k (likely an artifact of 3-seed averaging with one lucky run). OSCAR tracks BF16 closely at short-to-medium contexts: 99.7% at 4k, 100% at 8k, 97.8% at 16k, 87.6% at 32k, 61.9% at 64k, and 39.5% at 128k. At 64k and 128k, OSCAR's accuracy disadvantage relative to BF16 is approximately 23 and 41 points, respectively — much better than QuaRot's near-total collapse, but a real degradation that shows accumulated quantization error does eventually compound over very long contexts.

On Qwen3-8B, the gap is larger at short contexts but the relative degradation pattern is similar. BF16 stays above 97% through 32k, dropping to 78.2% at 128k. QuaRot-INT2 is usable only at very short contexts: 74.0% at 4k, 80.6% at 8k, then collapses to 19.0% at 16k and near-zero beyond 32k — the collapse is more gradual than on the 4B model but still renders the method unusable at practical long-context lengths. OSCAR maintains 99.5% at 4k, 97.8% at 8k, 93.9% at 16k, 86.3% at 32k, 61.9% at 64k, and 45.0% at 128k. At 128k, OSCAR is 33.2 points below BF16 — a larger gap than on the 4B model at the same length, suggesting that while the 8B model's representations are more robust to quantization in the 32K generation setting (Table 2), the accumulated attention error over 128k tokens actually causes a larger relative degradation for the 8B model in retrieval.

On GLM-4.7-FP8, the retrieval task is too easy to differentiate methods. BF16, QuaRot-INT2, and OSCAR all achieve 100% at 4k–32k, 98.8% at 64k, and 96.3–97.2% at 128k. The 358B model's representations are so robust that even Hadamard-only INT2 causes negligible retrieval degradation at 128k. This supports the model-scale trend but also indicates that RULER-NIAH at 128k is not a sufficiently difficult stress test for frontier-scale models — a harder retrieval task with more distractors or longer context would be needed to separate methods.

KL divergence analysis (Figure 3). The paper plots KL divergence between the FP16 and quantized attention distributions as a function of context length for Qwen3-4B-Thinking (Figure 3a) and Qwen3-8B (Figure 3b), using teacher-forced decoding for 32 steps. On the 4B model, OSCAR's KL stays in the 0.04–7.47 range across 4k to 128k, while QuaRot's KL climbs from 0.26 at 4k to 8.08 at 128k and clip-only climbs to 7.46 at 128k. On the 8B model, OSCAR's KL stays in 0.01–3.15, while QuaRot's climbs from 0.02 at 4k to 12.12 at 64k (then drops to 8.99 at 128k for unclear reasons — possibly query-dependent normalization effects at the longest context). The KL curves confirm that OSCAR keeps the attention distribution closer to the FP16 reference across all context lengths, with the advantage becoming more pronounced as context grows. The non-monotonicity in the QuaRot curve at 128k on the 8B model is not explained in the paper and may warrant investigation.

Serving Throughput and Memory Efficiency

The paper reports throughput improvements from two mechanisms: (1) per-step decode speedup due to reduced KV-cache memory bandwidth, and (2) increased batch size feasibility due to 8× memory reduction.

Pure decoding speed at batch size 1 (Figure 4, left panel). With a full prefix-cache hit (all KV states precomputed and cached, so decode time measures only the attention and GEMM, not prefill), decode throughput speedup relative to BF16 increases with context length. On Qwen3-4B-Thinking, OSCAR achieves 1.98× at 30k, 2.52× at 60k, and 3.08× at 100k context. On Qwen3-8B, 1.84× at 30k, 2.29× at 60k, 2.88× at 100k. On GLM-4.7-FP8, 1.98× at 30k, 2.49× at 60k, 2.83× at 100k. This scaling with context length confirms that decode speed becomes increasingly KV-bandwidth-bound at long contexts — the 8× memory reduction directly translates to larger throughput gains as the KV cache grows. OSCAR consistently outperforms Saw-INT4 (which uses 4-bit quantization and therefore has less aggressive memory reduction): at 100k on Qwen3-4B, Saw-INT4 achieves 1.51× while OSCAR achieves 3.08×. The online rotation overhead is "effectively hidden within the decode kernels" — it does not appear as a separate measurable cost in the throughput numbers.

Job-level throughput at 100k context (Figure 4, right panel). With batch sizes 1, 8, and 32 at 100k input length and prefix warm-up (simulating popular agentic serving workload with long prefix reuse): On Qwen3-4B-Thinking, OSCAR achieves 3.08×, 5.19×, and 6.17× speedup over BF16 at BS = 1, 8, 32 respectively. On Qwen3-8B, 2.88×, 3.35×, 3.44×. On GLM-4.7-FP8, 2.83×, 5.89×, 7.83×. The throughput advantage increases with batch size because larger batches amplify the memory-pressure benefit: OSCAR can accommodate more concurrent requests under the same GPU memory budget, and the GPU achieves higher utilization with larger batches. Saw-INT4 consistently underperforms OSCAR at all batch sizes (e.g., 2.33× vs. 5.89× at BS = 8 on GLM-4.7-FP8), reflecting the less aggressive compression ratio.

Per-decode kernel profiling (Table 8). On Qwen3-8B at batch size 1, the per-step decode latency is: BF16 total 9.9 ms (GEMM 5.6 ms, Attention 3.8 ms, Other 0.6 ms), OSCAR total 8.0 ms (GEMM 5.6 ms, Attention 1.3 ms, Quant 0.5 ms, Other 0.6 ms). The 1.9 ms savings comes entirely from attention being 2.9× faster (3.8 → 1.3 ms) due to reduced KV bandwidth. The quantization overhead (0.5 ms) is small. At batch size 32, BF16 attention dominates at 17.0 ms out of 23.4 ms total (72.7%), while OSCAR attention is 8.5 ms out of 15.5 ms (54.6%) — a 2.0× attention speedup. At batch sizes 64 and 128, BF16 runs out of GPU memory for its KV cache, while OSCAR handles these batch sizes (64: 16.2 ms attention, 128: 31.9 ms attention). The GEMM time (model computation) is essentially unchanged between BF16 and OSCAR, confirming that the quantization path is purely an attention optimization.

End-to-end serving throughput with varying cache-hit ratios (Figure 6). The paper sweeps prefix-cache hit ratio from 0% (cold cache) to ~100% (full cache hit, simulating long prefix reuse). On Qwen3-4B-Thinking and GLM-4.7-FP8 with 100k input/1024 output lengths at batch sizes 1, 8, 16, 32: increasing cache-hit ratio consistently shifts the throughput frontier upward by reducing prefill recomputation. OSCAR tracks the efficiency frontier across all cache regimes, closely matching QuaRot-INT2 (which achieves similar throughput but with collapsed accuracy per Table 2) and outperforming Saw-INT4. At full cache hit (~100%), the throughput advantage is largest because decode speed dominates. The paper also reports that OSCAR's design benefits from prefix cache reuse because its uniform INT2 representation preserves standard page sizes, enabling higher cache hit rates under job-level cache management.

Memory and extreme batch scalability (Figure 5). Under a 100k-token input stress test on a single H100 (80 GB), OSCAR's INT2 KV-cache pages enable scaling up to 28 concurrent requests with continued throughput gains, while BF16 and INT4 baselines either run out of memory at smaller batch sizes or plateau. The throughput-vs-concurrency curves show OSCAR maintaining near-linear scaling to 28 requests, while Saw-INT4 and BF16 hit memory limits and flatten out. This directly validates the paper's claim that the 8× memory reduction translates to substantially larger feasible batch sizes in long-context serving regimes.

Representative end-to-end throughput (Table 9). Under 32 concurrent requests with 8192-token inputs and 1024-token outputs: On Qwen3-4B-Thinking, OSCAR achieves 63.3 tok/s/user and 1723.9 tok/s/GPU at 71.86 accuracy, compared to BF16 at 41.1/1187.9/75.64 (1.54× per-user throughput, 1.45× per-GPU throughput). On Qwen3-8B, 52.5/1374.0/69.42 vs. 35.8/999.9/70.84. The throughput gains are substantial even at moderate context lengths where memory pressure is lower than the 100k stress test. QuaRot-INT2 achieves essentially identical throughput (66.3/1795.8 on 4B, 54.1/1411.9 on 8B) but with collapsed accuracy (1.40 and 10.14, respectively), making it "approach the throughput ceiling but suffer significant accuracy degradation."

---

Ablation Studies and Robustness Checks

Rotation decomposition and target comparison (Table 4): On Qwen3-8B, the full OSCAR rotation U · H_Had · P_br achieves 70.01 mean accuracy. Removing the attention-aware eigenbasis U and using only Hadamard + PBR (H_Had · P_br, equivalent to QuaRot with bit-reversal) drops to 32.82 (−37.19 points). This is the single largest ablation gap and quantifies the value of attention-aware PCA targets over data-oblivious rotations. Removing H_Had (U · P_br only) drops to 51.74 (−18.27 points) — the Hadamard step is critical for quantization geometry even after eigenbasis alignment. Removing P_br (U · H_Had only) drops to 68.00 (−2.01 points) — the bit-reversal permutation provides a real but smaller benefit by balancing importance across quantization groups. Replacing OSCAR's attention-aware targets with raw-cache reconstruction PCA targets (K^⊤ K / V^⊤ V) drops to 31.12 (−38.89 points) — this is the second-largest gap and directly supports the paper's central claim that which covariance matrix one diagonalizes matters more than whether one diagonalizes at all. Removing all rotation (clip + sink + recent only) drops to 4.23 — confirming that INT2 quantization is impossible without some form of preprocessing. The decomposition is clean and the magnitudes are well-separated: U and target choice are the dominant factors (~37–39 point impact), H_Had is a strong secondary factor (~18 points), and P_br provides a small but measurable improvement (~2 points).

Sink and recent window sizes (Table 5): On Qwen3-4B-Thinking-2507, sweeping (S₀, R) from (0, 0) to (256, 1024): (0, 0) yields 0.00 mean accuracy across all five benchmarks — the model collapses completely without BF16 token protection, even with optimal OSCAR rotation. (32, 128) recovers to 67.69 at 0.12% extra BF16 KV memory. (64, 256) achieves 71.86 at 0.24% extra BF16 KV memory — a clear knee in the tradeoff curve. (128, 512) gives 72.96 at 0.49% extra memory — diminishing returns. (256, 1024) gives 73.08 at 0.98% extra memory — negligible additional accuracy. The (64, 256) operating point is well-justified: smaller windows leave 4.17 accuracy points on the table, while larger windows add only ~1.2 points at a substantially higher memory cost. The fact that (0, 0) yields complete collapse is a critical finding: the attention-sink and recency-bias phenomena are not subtle biases but hard requirements — without BF16 protection for the first and last tokens, the model cannot attend at all, regardless of rotation quality. This suggests that these token positions have qualitatively different representation properties that cannot be captured by the low-bit quantized format.

Clip threshold validation (Table 6): On Qwen3-4B-Thinking-2507, a grid sweep over c_K ∈ {0.88, 0.92, 0.96, 0.98, 1.00} and c_V over the same values shows that the calibration-derived operating point (0.96, 0.92) achieves 70.59 mean accuracy, which is within 0.15 points of the best grid point (0.92, 0.98) at 70.74. The surface is relatively flat near the optimum: all points with c_K ≥ 0.92 and c_V ≥ 0.92 achieve >69.15 accuracy. Extreme clipping (c_V = 1.00, i.e., no clipping) degrades substantially: the bottom row (c_V = 1.00) ranges from 56.02 to 60.35, showing that value clipping is essential for controlling the quantization range. Key clipping is less critical: even c_K = 1.00 with good value clipping (c_V = 0.92) stays at 69.15. The calibration-derived thresholds are essentially optimal without any downstream-task tuning, supporting the paper's claim that offline percentile estimation is sufficient.

Calibration data regime (Table 7): On Qwen3-4B-Thinking-2507, varying calibration volume from 2k to 32k tokens and calibration domain across MMLU, WikiText, and GPQA-Diamond: the smallest setting (2k MMLU) achieves 68.28, the default setting (8k GPQA-Diamond) achieves 71.01, and the best setting is 16k MMLU at 70.88. The range across all 7 settings is [68.28, 71.01], a spread of only 2.73 points. The method is remarkably insensitive to calibration volume: 2k tokens recover 96% of the best setting's accuracy. Domain sensitivity is also minimal: MMLU, WikiText, and GPQA-Diamond all produce comparable results at the same volume (e.g., 8k: 69.15 MMLU, 69.50 WikiText, 71.01 GPQA-Diamond). Notably, GPQA-Diamond calibration outperforms MMLU and WikiText by ~1.5–2 points at the same volume — a small but consistent domain effect that the paper does not investigate further. The robustness to calibration regime is a practically important finding: it means practitioners do not need large or carefully curated calibration sets.

INT2 group size configuration (Tables 12–14 for Qwen3-8B, Tables 15–17 for Qwen3-32B): The paper provides detailed per-group-size breakdowns in the appendix. On Qwen3-8B, group size 128 achieves 69.42 mean accuracy, group size 64 achieves 69.42 (identical), and group size 32 achieves 68.74 (−0.68 points). The near-identical performance at 128 and 64 suggests the quantization error is dominated by per-token dynamic range rather than per-group coarseness at these group sizes. On Qwen3-32B, group size 128 achieves 70.45, 64 achieves 74.17, and 32 achieves 74.05 — here, group size 64 is slightly better than 128, likely because finer grouping allows better adaptation to the per-channel dynamic range variation in the larger model. The paper uses group size 128 as the default in the main results (Table 2) but provides the full breakdown tables so practitioners can choose the optimal group size for their model.

Full run-level detail (Tables 11–24): The appendix provides per-task per-seed results for all main and ablation configurations. The variance across seeds is generally small: standard deviations on mean accuracy are typically 0.5–2.0 points for the main OSCAR configuration across models. The largest per-task variance is on AIME25 (e.g., ±3.65 on Qwen3-4B-Thinking, ±5.48 on Qwen3-32B), consistent with AIME25's small size (30 problems) making it sensitive to individual seed variation. LiveCodeBench v6 also shows moderate variance (±1.5–2.0 points), reflecting the diversity of coding problems. The reporting of full per-seed results is a commendable transparency practice that allows readers to assess statistical reliability without relying on summary statistics alone.

Additional diagnostic visualizations (Figures 9–10, Appendix E.1): The paper includes heatmap visualizations of K and V activation distributions in the raw, Hadamard-rotated, and OSCAR-rotated bases across 256 history tokens (excluding sink/recent). The raw K activation shows extreme channel-wise outliers (max=35.75 at layer 10 of Qwen3-8B, max/std=11.4; max=21.25 at layer 11, max/std=10.1). Hadamard rotation flattens these to max=9.61 and max=8.85 (max/std reduces to 5.4 and 5.8). OSCAR further reduces to max=7.37 and max=6.12 (max/std=4.3 and 4.6). For V activations, the effect is less dramatic but still visible: raw max=3.69 (max/std=12.5) reduces to max=1.30 (max/std=5.5) with OSCAR. These visualizations complement the quantitative error decomposition in Figure 2 and make the outlier-suppression effect visually concrete. The fact that the V activation range is smaller and more uniform than K (even in raw form) is consistent with the known observation that keys are more sensitive to quantization than values, which motivates the paper's decision to use a more aggressive clip threshold for values (ρ_V = 0.92 vs. ρ_K = 0.96).

Negative result: Hadamard-only rotation vs. OSCAR on downstream attention error (Figure 8, Appendix A.4): The paper provides a comprehensive error decomposition across layers for Qwen3-4B-Thinking-2507 with 8k GPQA prefill tokens. Raw K/V reconstruction MSE (Figures 8a–b) shows that OSCAR and Hadamard-only rotation achieve comparable raw cache errors — both are much better than naive INT2 or clip-only baselines. However, on attention-consumed quantities — QK^⊤ MSE (Figure 8c), SV MSE (Figure 8d), attention KL divergence (Figure 8e), and attention-block output error (Figure 8f) — OSCAR consistently achieves 0.5–2 orders of magnitude lower error than Hadamard-only rotation, especially in the middle layers (10–25) where attention patterns are most complex. This directly validates the paper's central motivation: raw-cache reconstruction error is not the right optimization target for KV-cache quantization, because attention consumes these tensors through the specific quadratic forms Q K^⊤ and S V. The gap between raw error and attention error is the paper's key empirical finding and the primary justification for the attention-aware covariance approach.

Calibration cost and one-time processing: The paper reports that the offline calibration pass processes 8878 tokens (approximately 8k) of GPQA-Diamond MMLU-style prompts through the model, dumping per-layer Q, K, V, and S activations. The eigendecompositions are computed on d × d matrices where d = 128 (head dimension), so the computational cost is trivial (a few seconds per layer on CPU). The clip thresholds are estimated as simple percentiles from the rotated calibration activations. All parameters are reused across all benchmarks without re-calibration. The total offline compute cost is dominated by the single forward pass over 8k tokens, which is negligible relative to the online inference savings from 8× KV-cache compression.

---

Critical Assessment

Claim 1: OSCAR reduces the BF16 accuracy gap relative to naive rotation INT2, which collapses to near-zero. This claim is overwhelmingly supported by Table 2. On Qwen3-4B-Thinking, Naive INT2 and QuaRot-INT2 achieve 0.00 and 1.40 mean accuracy respectively, while OSCAR achieves 71.86 (BF16: 75.64). On Qwen3-8B, Naive INT2 achieves 0.00, QuaRot-INT2 achieves 10.14, OSCAR achieves 69.42 (BF16: 70.84). The collapse is not subtle — naive INT2 produces literally zero correct answers across all five benchmarks for both models. The paper's explanation (attention distributions diverge heavily without proper rotation) is supported by the KL divergence figures (Figure 3) and the error decomposition (Figure 2). However, the claim that OSCAR "recovers" BF16 accuracy requires qualification at the smaller model scales: on the 4B model, there is still a 3.78-point gap, which is real and measurable. The paper appropriately characterizes this as "close" rather than "identical."

Claim 2: Covariance-target rotations outperform raw-cache reconstruction rotations by a large margin. Supported by Table 4: replacing Q^⊤ Q / V^⊤ S^⊤ S V PCA targets with K^⊤ K / V^⊤ V PCA targets — keeping everything else identical — drops mean accuracy from 70.01 to 31.12 (−38.89 points). This 38.89-point gap is larger than the gap between OSCAR and QuaRot-INT2 (32.82 vs. 70.01, a 37.19-point gap), indicating that target choice is at least as important as whether any learned rotation is used at all. The paper's theoretical justification (Theorem 1) provides a formal explanation for why Q^⊤ Q is the optimal target under the frozen-residual assumption. However, the ablation does not test intermediate targets — for example, using Q^⊤ Q for keys but V^⊤ V for values, or vice versa. It also does not test whether the exact identity of the target matters or whether any attention-aware target (e.g., one derived from per-token rather than per-sequence statistics) would work similarly. The comparison is binary (attention-aware vs. raw-cache) rather than a broader exploration of the target design space.

Claim 3: OSCAR is deployable and integrates with paged-attention serving without breaking compatibility. This claim is supported by the detailed kernel design in Section 4, the SGLang integration, the kernel profiling data (Table 8), and the throughput results (Figures 4–6, Table 9). The paper demonstrates end-to-end serving with 32k-generation evaluation inside SGLang for all accuracy benchmarks, which is a higher integration bar than method-only papers that report accuracy in a standalone evaluation harness. The fused kernel overhead is measured and shown to be small (0.5 ms per step at batch size 1, Table 8). The paper explicitly contrasts this with Group A baselines (KIVI, Kitty) that achieve good accuracy but break paged-attention compatibility. However, the paper does not provide a direct latency comparison against these Group A baselines because "we do not have paged/fused kernels for them at 32K generation length." This is a limitation of the comparison rather than a flaw in the claim, but it means the accuracy-throughput tradeoff between OSCAR and channel-wise methods cannot be directly assessed in the paper's serving setup.

Claim 4: OSCAR scales to large models (358B) and remains effectively on par with BF16. Supported by Table 2: on GLM-4.7-FP8, OSCAR achieves 78.16 mean accuracy vs. BF16 77.89 (+0.27). The fact that several INT2 methods exceed BF16 on this model (TurboQuant +0.26, Saw-INT4 +0.06) raises questions about statistical noise or evaluation quirks — it is unlikely that quantization actually improves accuracy, and these small positive deltas are almost certainly within the noise floor. More importantly, the paper does not investigate why larger models are more INT2-robust. Is it because larger models have more redundant representations? Because their covariance spectra are more sharply defined? Because their attention patterns are more structured and therefore easier to align with? The empirical trend is clear (4B: −3.78, 8B: −1.42, 32B: −0.02, 358B: +0.27), but the mechanism is not explored. This limits the paper's ability to predict which future models will be amenable to INT2 quantization without explicit evaluation.

Claim 5: The composed rotation U · H_Had · P_br with each factor contributing substantially is better than simpler alternatives. The ablation in Table 4 cleanly supports the additive value of each factor: removing U costs 37.19 points, removing H_Had costs 18.27 points, removing P_br costs 2.01 points. The magnitudes are well-separated and the ordering matches the paper's theoretical argument (importance alignment is most critical, quantization geometry is secondary, group balancing is a refinement). However, the ablation tests removal of one factor at a time from the full composition — it does not test whether reordering the factors (e.g., H_Had · U · P_br) would work similarly. The paper argues in Appendix A.4 that the order is essential because Lemma 1 requires C_Q to be diagonal before the Hadamard is applied, but this is not tested empirically. An ablation that applied the factors in different orders would strengthen the claim that the specific composition order matters.

Missing ablations and analyses of note:

• No comparison against SpinQuant at INT2. The paper mentions that SpinQuant (Liu et al., 2024) uses learned rotations and collapses at INT2 (Appendix C), but does not include it as a baseline in the main results. Given that SpinQuant is the most direct alternative to the paper's PCA-based rotation (it also learns data-dependent rotations, but via gradient-based optimization rather than covariance eigendecomposition), its absence is notable. The paper's claim that "it collapses at extreme INT2 levels because its rotations fail to align with the actual covariance structures consumed by downstream attention" is stated without experimental evidence in the main results.
• No ablation on the variance of the calibration procedure. The paper runs calibration once on a fixed 8k GPQA-Diamond set and reports accuracy over 5 seeds. It does not report variability from running calibration on different random subsets or different initializations. Given the small calibration volume (8k tokens), there could be non-trivial variance in the estimated eigenbases. A table showing mean and standard deviation of accuracy over multiple calibration runs would quantify how stable the offline procedure is.
• No direct latency comparison against 4-bit methods. The paper compares throughput against Saw-INT4 (4-bit) and shows OSCAR is faster (Figure 4). But it does not isolate whether this throughput advantage comes purely from the reduced bit-width (8× vs. 4× compression) or also from kernel design differences. A hypothetical "OSCAR at 4 bits" would help disentangle these effects.
• No analysis of the eigenstructure stability across layers. The paper reports that Q^⊤ Q and K^⊤ K have uncorrelated eigenbases (alignment ≈ 0.09 at layer 16, Appendix A.4), but does not analyze how this alignment varies across layers. It's plausible that early layers (which process more raw token-level features) have different covariance structure from late layers (which process more abstract representations). A layer-wise analysis might reveal that attention-aware targets are more important in certain layer regimes, which could inform layer-adaptive allocation strategies.
• No evaluation on non-reasoning, non-math benchmarks. All five accuracy benchmarks involve math, science, or code reasoning. Tasks like summarization, translation, or open-ended dialogue — which have different attention patterns and may be more or less sensitive to KV quantization — are not evaluated. The RULER-NIAH task partially addresses this gap (retrieval is a different skill than reasoning), but the coverage of task types is limited.
• Limited context length coverage in accuracy evaluation. All accuracy benchmarks are run at 32k generation length. The RULER experiments extend to 128k, but only for retrieval. The paper does not evaluate how generation quality degrades at context lengths longer than 32k for the reasoning and coding tasks — an important practical question since the 128k retrieval results show non-trivial degradation at the longest contexts.

Despite these gaps, the experiments are thorough for the scope they cover. The combination of (1) multi-model evaluation from 4B to 358B, (2) five diverse benchmarks with 32k generation, (3) long-context retrieval up to 128k, (4) detailed ablation of all rotation components and hyperparameters, (5) full system throughput measurements, and (6) complete per-seed reporting in the appendix makes this one of the more comprehensively evaluated INT2 KV-cache quantization papers. The key strength is the ablation in Table 4, which cleanly isolates the contribution of attention-aware covariance targets from other design choices and provides the clearest evidence for the paper's central claim. The key limitation is the narrow task coverage — the paper demonstrates that OSCAR works well on the tasks it evaluates, but does not establish whether these tasks are representative of all long-context serving workloads.

6. Limitations and Trade-offs

Calibration Input Distribution Mismatch

The paper explicitly restricts the server-side calibration input distribution to "text prompts" that are "general-domain, publicly available, and explicitly excluded from containing any information that could be exploited for adversarial purposes" (Section 5.1, calibration paragraph). The consumer-side API contract—that end-user prompts do not leak into the calibration set—is "enforced by the platform's data governance boundary."

Consequence. In a multi-tenant deployment where a single OSCAR-calibrated model instance serves prompts from diverse and potentially highly-skewed distributions (e.g., legal contracts followed by Python code followed by medical records), the rotation matrices estimated once from general-domain text may not align the attention covariance structure for domain-specific attention patterns. The paper's calibration robustness study (Table 7) shows that MMLU, WikiText, and GPQA-Diamond calibration all produce comparable accuracy on the evaluation benchmarks—but these evaluation benchmarks are themselves drawn from a similar general-domain distribution (math, science, code). There is no evidence that OSCAR's covariance estimates generalize to structurally different text types (e.g., highly repetitive boilerplate, tabular data, non-English languages, multi-turn chat with extensive instruction-following preamble). If an end user sends prompts whose attention statistics produce covariance eigenbases that are poorly aligned with the calibration-derived rotations, the INT2 quantization error may land in directions with unexpectedly high query energy, degrading attention fidelity beyond what the benchmark numbers predict. The paper does not measure accuracy on any held-out domain that differs substantially from the calibration and evaluation distributions, so this failure mode is uncharacterized.

Evidence in the paper. Table 7 calibrates on MMLU, WikiText, and GPQA-Diamond and evaluates on GPQA/HumanEval/LCB/AIME/MATH—all reasoning-heavy tasks. No calibration-to-evaluation domain shift is tested outside the reasoning/code family. The calibration data regime study (Table 7) spans 2k–32k tokens, but only within the three general-domain text sources; it does not test, for example, OSCAR calibrated on WikiText but evaluated on a legal-document retrieval task or a multilingual benchmark.

Mitigation status. The paper does not address this limitation or propose any mechanism for detecting or adapting to distribution shift at serving time. The calibration is a one-time offline pass with fixed parameters reused for all queries. The consumer-side privacy guarantee is an architectural property of the serving platform, not a technical feature of OSCAR itself—it prevents per-user calibration but also prevents within-deployment domain adaptation.

---

User-Facing Text KV-Cache Access via Provider API

The paper's system design assumes a specific deployment model: the provider (platform) has direct access to raw KV-cache tensors during prefill and can rotate, clip, and quantize them inside fused GPU kernels (Section 4). The consumer (end user or downstream application) interacts with the model through a serving API that accepts text prompts and returns generated text, never seeing or manipulating the intermediate KV-cache state. The calibration data is similarly drawn from a provider-controlled pool of general-domain text, not from consumer prompts, and the provider's governance boundary ensures that consumer data does not contaminate the offline calibration pass.

Consequence. This design may not generalize to deployment scenarios where the KV-cache format is exposed to the consumer. If a provider offers an API that returns raw or quantized KV-cache tensors to the consumer (e.g., for prefix caching across trust boundaries, for multi-step agentic workflows where the consumer manages its own KV cache across multiple API calls to the same provider, or for collaborative inference where different parties supply different context segments), the consumer would need to know and agree on the rotation matrices and quantization parameters. These parameters (the per-layer R_K, R_V, clip thresholds, group sizes) constitute additional API surface that must be versioned, documented, and negotiated. Furthermore, if the consumer can observe or manipulate the quantized KV representation, they could potentially extract information about the calibration data's attention-covariance structure (which may reveal properties of the provider's calibration corpus) or craft adversarial prompts whose attention patterns exploit misalignments with the fixed rotation basis—a side-channel not present in BF16 serving. The paper does not discuss API design, parameter versioning, or consumer-facing KV-cache access patterns.

Evidence in the paper. The paper explicitly states that "end-user prompts never flow into offline calibration—the calibration set is drawn from a separate provider-held pool" and that "KV-cache tensors are never exposed to the consumer; quantization happens entirely on the provider side inside trusted GPU kernels" (Section 4). This is a design choice motivated by privacy, but it also circumvents the question of how OSCAR would operate if KV caches ever crossed the provider-consumer boundary. No experiments test interoperability across different calibration sets or different quantization parameters.

Mitigation status. The paper does not claim to support consumer-facing KV-cache access, and the described deployment model (closed-api serving with provider-side infrastructure) is consistent with how most LLM APIs operate today. However, the limitation is relevant because prefix-cache sharing across API consumers, consumer-managed KV caches, and collaborative inference are active areas of development in LLM serving systems, and OSCAR's design would need extension to support them.

---

Hard Problems Where Even OSCAR Cannot Recover BF16 Accuracy

The paper demonstrates that OSCAR closes most of the BF16 accuracy gap on the models and benchmarks evaluated, but it does not eliminate the gap entirely at the smallest model scales—and there is evidence that certain task types are more sensitive than others.

Consequence. On Qwen3-4B-Thinking-2507, OSCAR still drops 3.78 mean accuracy points from BF16 (Table 2), with the largest single-benchmark gap on AIME25: BF16 74.67 vs. OSCAR 64.00, a drop of 10.67 points. AIME25 consists of competition-level math problems requiring multi-step symbolic reasoning, often with long chains of deduction. At 32k generation length (the evaluation cap), these problems stress the model's ability to attend correctly to intermediate reasoning steps produced hundreds or thousands of tokens earlier. The fact that OSCAR loses over 10 points specifically on this benchmark while losing only 1–2 points on GPQA, HumanEval, and MATH500 suggests that the accumulated attention error from quantized history tokens is not uniform across task types: reasoning chains with long-range dependencies amplify small per-step attention distortions into incorrect intermediate conclusions, which then propagate through the remainder of the generation. A practitioner deploying OSCAR on a reasoning-heavy workload (e.g., automated theorem proving, multi-step code debugging, long-chain planning) at the 4B–8B model scale should expect degraded performance on the hardest problems, even though average-case accuracy looks strong. The paper does not analyze whether the AIME25 gap is concentrated on specific problem types (geometry vs. algebra vs. combinatorics) or specific reasoning patterns (long backward references vs. local step-by-step deductions), so a practitioner cannot predict which reasoning tasks are most at risk.

Evidence in the paper. Table 2 shows the benchmark-specific breakdown: on Qwen3-4B-Thinking, AIME25 drops 10.67 points while GPQA drops 2.32, HumanEval 1.81, LCB 3.28, and MATH500 0.80. On Qwen3-8B, the AIME25 gap is 3.33 points while other gaps are 1–2 points. The RULER-NIAH results (Table 3) show a 23.4-point gap at 64k and 41.5-point gap at 128k on Qwen3-4B-Thinking, confirming that long-range attention degrades under quantization even for the simpler retrieval task. Figure 3's KL divergence curves show that OSCAR's attention distribution diverges from FP16 more at longer contexts, but the paper does not connect this divergence magnitude to downstream benchmark accuracy quantitatively.

Mitigation status. The paper acknowledges (Section 5.2) that "the largest gap is on AIME25, consistent with the observation that reasoning-intensive tasks expose more accumulated quantization error over long generation." However, no specific mitigation is proposed for reasoning-heavy workloads. The sunk/recent-window protection (S₀ = 64, W = 256) helps by keeping recent context at full precision, but reasoning dependencies can span well beyond 256 tokens—a key lemma proved 2000 tokens earlier may need to be attended to accurately. The paper does not explore dynamic window sizing based on task type, adaptive precision for tokens identified as reasoning-critical, or any task-specific calibration strategies.

---

Single Hardware Architecture (NVIDIA H100) and Kernel Stack (Triton/SGLang)

All system-level results in the paper—kernel profiling (Table 8), throughput measurements (Figures 4–6, Table 9), and memory scaling (Figure 5)—are collected on NVIDIA H100 GPUs (80 GB) using the SGLang serving framework with custom Triton kernels. The INT2 attention kernel, the fused rotate-clip-quantize prefill kernel, the dequantize-inverse-rotate decode kernel, and the window-demotion kernel are all written specifically for this hardware/software stack.

Consequence. The throughput advantages claimed (up to 7.83× over BF16 at batch size 32 on GLM-4.7-FP8, Figure 4) depend on the H100's specific memory bandwidth, tensor core throughput, and cache hierarchy. On hardware with different compute-to-bandwidth ratios (e.g., NVIDIA B200 with higher bandwidth, consumer GPUs with lower bandwidth, or non-NVIDIA accelerators), the relative benefit of reducing KV-cache memory traffic changes. Specifically:

• On GPUs with higher memory bandwidth relative to compute (like B200), the BF16 baseline attention will be less bandwidth-bound, so the 8× memory reduction from INT2 may not translate to as large a throughput multiplier—the GEMM computation becomes a larger fraction of total decode time.
• On GPUs with lower memory bandwidth (like consumer GPUs or edge accelerators), the bandwidth savings are more valuable, but the fixed dequantization and inverse-rotation compute overhead (0.5 ms per step at batch size 1 on H100, Table 8) may become proportionally larger if the compute units are slower.
• On non-Triton serving stacks (e.g., vLLM with CUDA kernels, TensorRT-LLM, or custom C++ runtimes), the fused kernel designs would need to be reimplemented, and the achieved efficiency may differ. The paper mentions that "additional tuning is needed to understand the best kernel design on other hardware such as B200" (Appendix D) but does not provide any cross-hardware measurements or projections.

Evidence in the paper. Table 8 reports per-decode kernel latency in milliseconds on 1×H100 for Qwen3-8B. Appendix D states: "our implementation targets SGLang-style paged serving on H100 GPUs; additional tuning is needed to understand the best kernel design on other hardware such as B200." The throughput experiments (Figures 4–6, Table 9) are all on H100s with SGLang. No cross-hardware or cross-framework results are presented.

Mitigation status. The paper acknowledges this limitation explicitly in the limitations section (Appendix D) and suggests future work. The open-source code release may enable community porting to other platforms, but no portability guarantees are provided. Practitioners considering deploying OSCAR on non-H100 hardware should budget for kernel reimplementation and re-tuning.

---

Theoretical Guarantee Excludes the Hadamard, Permutation, and Quantization Steps

Theorem 1 (Appendix A.6) proves that R_K = U_Q and R_V = U_S are optimal orthogonal rotations for the frozen-error surrogate objectives \tilde{\mathcal{L}}_K and \tilde{\mathcal{L}}_V, assuming diagonal frozen residual covariances with sorted eigenvalues. This theorem provides the intellectual justification for choosing eigenbases of Q^⊤ Q and V^⊤ S^⊤ S V as the base rotations. However, the theorem does not address the subsequent Hadamard transform (H_Had), the bit-reversal permutation (P_br), the per-token clipping, or the asymmetric INT2 quantization with per-group min-max scaling—all of which are essential components of the full OSCAR pipeline.

Consequence. The theorem provides a guarantee only for the U_Q/U_S factor of the rotation, under an assumption (frozen diagonal residuals) that is known to be violated in practice by the subsequent Hadamard transform and the quantization-dependent residual structure. After U_Q rotation, the K activations have an anisotropic distribution (as shown in the worked example in Appendix A.4), and the Hadamard transform is applied precisely to reshape this distribution into something more uniform that the per-group quantizer can handle well. The frozen-residual assumption—that per-channel quantization errors are independent of the rotation—is a helpful simplification for proving Theorem 1, but it prevents the theorem from making any claim about the full composed rotation U · H_Had · P_br. In particular, Theorem 1 does not guarantee that the composition with the Hadamard preserves the importance-alignment property (experimentally, Lemma 1 shows it equalizes the diagonal to a constant, which is a different property from minimization of \tilde{\mathcal{L}}_K). The theoretical justification for the full pipeline is therefore incomplete: the first factor is justified by Theorem 1, the second factor is justified by Lemma 1 (exact equalization of the importance-metric diagonal), the third factor is justified heuristically (balancing importance across groups), and the quantization scheme is justified empirically (ablation shows it works). There is no unified theoretical framework that encompasses all components and their interactions.

This matters for robustness: if the quantization geometry after U_Q H_Had produces residual covariances E_K that are not diagonal in the rotated frame, the assumptions of Theorem 1 are violated and the optimality guarantee for U_Q no longer holds. The paper does not measure how diagonal E_K is in practice after each transformation step, so it is unclear how severely the frozen-residual assumption is violated and whether the theoretically-optimal U_Q remains near-optimal under the actual residual structure.

Evidence in the paper. Theorem 1 (Appendix A.6) explicitly states the diagonal frozen-residual assumption: E_K = diag(μ_1, ..., μ_d) and E_V = diag(η_1, ..., η_d). The proof is a rearrangement-inequality argument that relies on this diagonality. Appendix D states: "our theoretical analysis explains why the proposed covariance-target rotations are optimal under a frozen-error surrogate with explicit assumptions, but it is not yet a full proof of optimality for the complete autoregressive decoding process. Tightening this gap... is an important direction for future work." The paper does not empirically measure the diagonality of E_K in the full OSCAR pipeline or ablate how the optimality gap changes when the assumption is violated.

Mitigation status. The paper is transparent about this limitation in Appendix D and positions the theorem as providing "intellectual justification" rather than a complete optimality guarantee. The empirical ablations (Table 4) demonstrate that removing each factor from the full pipeline degrades accuracy, but they do not test whether the specific composition order (or the specific choice of factors) is theoretically optimal under weaker assumptions. Tightening the theory is flagged as future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that the choice of rotation target is the dominant factor in INT2 KV-cache quantization, not the choice of quantization algorithm, the sophistication of the search procedure, or the presence of outlier-suppression heuristics. The 38.89-point gap in Table 4 between attention-aware covariance targets and raw-cache reconstruction targets — larger than the gap between OSCAR and naive Hadamard-only rotation — reframes what the field should optimize for. Before this work, the accepted wisdom (inherited from QuaRot, SpinQuant, and the broader post-training quantization literature) was that the primary challenge in low-bit quantization is making activation distributions more uniform so that a fixed-width quantizer incurs less rounding error. OSCAR demonstrates that at 2 bits, distribution uniformity is a secondary concern; the primary concern is steering what error remains into directions that the downstream computation reads weakly.

This is a conceptual reframing, not a paradigm shift — it extends the covariance-aware calibration principle from weight quantization (GPTQ, AWQ, SVD-LLM) to KV-cache quantization, but with a crucial methodological twist: the target covariance must be derived from attention's own computation graph (Q^⊤ Q and V^⊤ S^⊤ S V) rather than from the cached tensors themselves (K^⊤ K and V^⊤ V). The twist is deceptively simple — it amounts to estimating a different covariance matrix on the same calibration data — but its consequences are large enough (38.89 accuracy points) that it changes the default approach for future INT2 KV-cache work.

The paper also reconciles the apparent contradiction between rotation-based and channel-wise methods. Prior work showed that rotation-based approaches (QuaRot) work well at 4 bits but collapse at 2 bits, while channel-wise approaches (KIVI, KVQuant, Kitty) maintain accuracy at 2 bits but break paged-attention compatibility and fused-kernel integration. OSCAR demonstrates that 2-bit rotation-based quantization can work if the rotation target is chosen correctly — and that doing so preserves the deployability advantages (uniform dense layout, paged-cache compatibility, fused kernels) that made rotation-based methods attractive at higher bit widths in the first place (Section 4, Table 2 vs. Table 1). The field therefore gains a deployable 2-bit method that does not require sacrificing the serving infrastructure that modern LLM platforms depend on.

The paper's finding that INT2 robustness increases dramatically with model scale (Section 5.2: BF16-to-OSCAR gap shrinks from −3.78 at 4B to −1.42 at 8B to −0.02 at 32B to +0.27 at 358B) shifts the application landscape. For frontier-scale models (100B+), INT2 KV-cache quantization is essentially a solved engineering problem — even naive INT2 on GLM-4.7 achieves 60.49 mean accuracy (Table 2), and OSCAR pushes this to 78.16, slightly above BF16. For small models (4B–8B), a 1–4 point accuracy gap remains, concentrated on the hardest reasoning tasks (AIME25 drops 10.67 points on the 4B model). This means that for large-model deployments, practitioners can adopt INT2 quantization with high confidence using any reasonable method; for small-model deployments, OSCAR's attention-aware calibration is the difference between usable and collapsed accuracy, but it does not fully close the gap on hard reasoning problems. Research attention should therefore shift from "can we do INT2 at all" (which OSCAR answers affirmatively) to "how do we close the remaining gap on small models and hard reasoning tasks" and "how do we adapt the rotation target dynamically to avoid the worst-case degradation on specific problem types."

A less visible but important methodological contribution is the demonstration that one-pass offline calibration generalizes across tasks. The paper reuses the same calibration parameters (estimated from 8k tokens of GPQA-Diamond MMLU-style prompts) across five reasoning and coding benchmarks plus a long-context retrieval evaluation without per-task adaptation (Section 5.1, Table 7). This robustness suggests that attention-covariance structure — specifically, the eigenbases of Q^⊤ Q and V^⊤ S^⊤ S V — is a stable property of the pretrained model, not a surface statistic of the calibration text. Prior work on activation-aware weight quantization (AWQ, GPTQ) had demonstrated similar calibration robustness for weight sensitivity, but extending this property to attention-specific covariance targets for KV caches is new. It means that practitioners do not need to re-calibrate for each deployment domain, which substantially reduces the operational complexity of adopting INT2 KV-cache compression in production serving environments.

Finally, the paper's three-factor rotation decomposition (U · H_Had · P_br) provides a template for designing compound rotations with separable, verifiable properties. Lemma 1 gives an exact guarantee for the Hadamard factor (uniform importance-metric diagonal), the bit-reversal permutation gives a recursive balancing guarantee across quantization groups, and Theorem 1 provides optimality of the eigenbasis under the frozen-error surrogate. This modular approach — where each factor addresses one geometric failure mode with a specific mathematical property — is more interpretable and more easily adapted to new hardware or new quantization schemes than a single end-to-end learned rotation. As the field moves toward mixed-precision formats (NVFP4, MX formats, microscaling), this decomposition may serve as a design pattern for constructing quantization-aware transforms with provable intermediate properties.

Follow-Up Research This Work Enables

Closing the theory gap: optimal rotation under the true (non-frozen, non-diagonal) residual structure. Theorem 1 assumes frozen, diagonal residual covariances E_K and E_V. The paper acknowledges that this assumption is violated in practice by the Hadamard transform and the per-group min-max quantizer. A natural theoretical extension would characterize the optimal composed rotation when E_K is not diagonal and depends on R_K through the quantization scale computation. Specifically: for the per-group asymmetric INT2 quantizer described in Appendix A.5, the residual for channel i in group g depends on the group's dynamic range, which depends on the distribution of K R_K within that group. Can one derive the optimal R_K under this quantizer-specific residual model, perhaps as a fixed point of the rotation and the resulting residual covariance? A strong follow-up would compare the theoretically optimal rotation (under the quantizer-aware residual model) against OSCAR's composed rotation empirically, measuring (a) whether the improvement over U_Q H_Had P_br is large enough to justify the more complex computation, and (b) whether the theoretically optimal rotation preserves the deployability advantages (fixed offline computation, dense layout, fusibility) that OSCAR's heuristic composition provides.

Layer-adaptive rotation targets and mixed-precision allocation. The paper observes that OSCAR's accuracy degradation is concentrated on reasoning-intensive benchmarks (AIME25 drops 10.67 points on Qwen3-4B) and at long context lengths (RULER-NIAH at 128k shows a 41.5-point gap on the 4B model, Table 3). A natural hypothesis is that different transformer layers contribute differently to this accumulated error. For instance, early layers (which process token-level features) may have covariance structure that is more sensitive to attention-target alignment, while late layers (which process abstract representations) may be more robust. A follow-up study could measure the per-layer attention-error contribution (using the error decomposition framework from Figure 2 and Appendix Figure 8) and ask: do some layers benefit more from attention-aware rotation than others? If so, one could allocate the INT2 budget adaptively: use OSCAR's expensive eigenbasis rotation (U_Q H_Had P_br) only on the top-k most sensitive layers, and use cheap Hadamard-only rotation (H_Had P_br) on the remaining layers. This would reduce offline eigendecomposition cost and online rotation cost while preserving most of the accuracy gain. The per-layer error curves in Figure 2 and Appendix Figure 8 (which show layer-wise QK⊤ MSE and SV MSE for different methods) already contain the raw data needed to estimate per-layer sensitivity; a follow-up would need to implement layer-heterogeneous rotation configurations in the SGLang kernel and measure the accuracy-efficiency Pareto frontier.

Dynamic difficulty-adaptive precision and window sizing. The paper's mixed-precision layout uses fixed sink (S_0 = 64) and recent (W = 256) windows determined once offline (Table 5). However, the RULER-NIAH results (Table 3) show that the attention-distribution divergence from FP16 grows with context length (Figure 3), and the AIME25 results show that hard reasoning problems are disproportionately affected. These observations suggest that the optimal window sizes and precision allocation may depend on the estimated difficulty of the current request. Building on the compute-optimal test-time scaling framework (Snell et al., 2024), a follow-up could develop a difficulty estimator for KV-cache quantization: given the calibration-derived rotation and a few initial decode steps, use the KL divergence between quantized and BF16 attention distributions (as in Figure 3) or the PRM-like verifier score on intermediate outputs to estimate whether the current request is "hard" (high accumulation of attention error) or "easy." For hard requests, dynamically expand the BF16 recent window or temporarily promote certain history tokens to higher precision; for easy requests, stay with the default aggressive INT2 configuration. The question is whether difficulty can be detected early enough to be useful — the KL curves in Figure 3 suggest that attention-error divergence starts accumulating within the first few thousand tokens, which may provide sufficient signal. A strong follow-up would implement this adaptive system in SGLang and measure the accuracy-throughput tradeoff against the fixed-configuration baseline on a held-out set of reasoning problems spanning multiple difficulty levels (e.g., MATH500 difficulty quintiles, as in Snell et al.).

Cross-architecture and cross-hardware calibration transfer. The paper demonstrates that OSCAR's calibration parameters transfer across benchmarks — the same rotation and clip thresholds serve for GPQA, HumanEval, LiveCodeBench, AIME25, MATH500, and RULER (Section 5.1). A more ambitious question is whether they transfer across model architectures and hardware platforms. Specifically: (a) If one calibrates OSCAR on Qwen3-8B, how well do those rotations transfer to Qwen3-4B (same architecture family, different scale) or to GLM-4.7 (different architecture family, similar scale)? The eigenbases of Q^⊤ Q encode model-specific attention patterns, which may or may not be conserved across model scales within a family. A positive result (reasonable transfer) would mean that a provider could calibrate once on a small model and reuse parameters across a model family, dramatically reducing calibration cost. A negative result (poor transfer) would indicate that attention-covariance structure is model-specific and must be estimated per-deployment. (b) If one calibrates on H100 and deploys on B200 (different memory bandwidth, different tensor core throughput, potentially different numerical behavior in the fused kernels), do the throughput advantages scale as predicted by the memory-bandwidth reduction alone? The paper provides H100 kernel profiling (Table 8) and states that "additional tuning is needed to understand the best kernel design on other hardware such as B200" (Appendix D). A follow-up study on B200 would provide the data needed for practitioners to project cost savings across hardware generations.

Adversarial robustness of fixed rotations to prompt-engineering attacks. The paper's calibration is performed once offline on a provider-held pool of general-domain text (Section 5.1, Appendix D). The rotation matrices are fixed for all subsequent queries, regardless of their content. This raises a security question: can an adversary craft a prompt whose attention statistics produce Q^⊤ Q and V^⊤ S^⊤ S V eigenbases that are maximally misaligned with the calibration-derived rotations, causing the INT2 quantization error to land in high-query-energy directions and degrading the model's output? This is analogous to adversarial attacks on quantized neural networks where inputs are perturbed to maximize quantization error. In the KV-cache setting, the adversary controls the prompt text (not the model internals), so the attack surface is indirect. A follow-up study would: (a) formalize the threat model (adversary sends prompt, observes model output, cannot access KV-cache tensors), (b) use gradient-based optimization through the quantized attention computation to find prompts that maximize attention-distribution divergence from the FP16 reference, and (c) measure whether such prompts cause measurable degradation on downstream task accuracy compared to benign prompts of similar length and domain. A negative result (adversarially crafted prompts do not degrade accuracy beyond natural variance) would strengthen confidence in fixed offline calibration. A positive result (measurable degradation on specific prompt patterns) would motivate online-adaptive rotation or per-request calibration in security-sensitive deployments.

Extending covariance-aware rotations to mixed-precision formats (NVFP4, MXFP4, microscaling). OSCAR targets uniform INT2 quantization with per-group dynamic scaling. The hardware ecosystem is moving toward standardized low-precision floating-point formats (NVFP4 on Blackwell, OCP microscaling formats) that use shared exponents across small blocks of elements, analogous to OSCAR's per-group scaling. These formats have different error characteristics from INT2: floating-point formats concentrate precision near zero (where most activation values lie after normalization) and have higher dynamic range at the cost of fewer uniformly-spaced levels. The question is whether OSCAR's covariance-target rotation principle transfers to these formats. Specifically: for a block浮点 format with a shared exponent and 2-bit mantissas, the per-channel quantization error is not uniform — channels with values near zero get higher effective precision, while channels with large values get coarser precision. The optimal rotation target under this error model may differ from the uniform-error model that motivates Theorem 1. A follow-up study would: (a) derive the optimal rotation target for a given floating-point format under a frozen-error surrogate analogous to Theorem 1 but with format-specific error characteristics, (b) implement the resulting rotation in a simulated NVFP4 serving path, and (c) compare accuracy against OSCAR's INT2 rotation and against data-oblivious Hadamard rotation on several model scales. This would establish whether attention-aware covariance estimation is a universal principle for low-bit KV-cache compression or specific to uniform integer quantization.

Practical Applications and Downstream Use Cases

On-device or edge deployment of small reasoning models with long-context support. The paper demonstrates that Qwen3-4B-Thinking-2507 with OSCAR INT2 quantization achieves 71.86 mean accuracy across five reasoning and coding benchmarks — within 3.78 points of BF16 — while using approximately 8× less KV-cache memory (Table 2). For edge deployment scenarios where GPU memory is severely constrained (e.g., a single consumer GPU with 8–12 GB, a mobile NPU, or an automotive inference chip), this memory reduction is the difference between supporting 32k-token reasoning traces and being limited to 4k–8k tokens. A 4B-parameter reasoning model with BF16 KV cache at 32k context would require roughly 4 GB of KV-cache memory alone; with OSCAR, this drops to approximately 0.5 GB, fitting comfortably within a 12 GB GPU alongside model weights and activations. The throughput benefits compound in interactive settings: the batch-size-1 decode speedup (3.08× at 100k context on Qwen3-4B, Figure 4 left) directly reduces per-token latency for a single user, improving the responsiveness of on-device assistants. The key practical caveat is the 10.67-point AIME25 gap on the 4B model (Table 2): edge deployments handling hard competition-math problems would see degraded accuracy, but deployments handling routine code completion, document QA, or general reasoning (where gaps are 1–2 points on HumanEval, GPQA, MATH500) would see near-BF16 quality.

Cost-efficient batch inference for reasoning-model evaluation and data generation. Organizations that run large-scale batch inference over reasoning models — for automated evaluation, training-data generation, or self-improvement pipelines (e.g., STaR, ReST^EM-style rejection sampling) — face a cost structure dominated by GPU-hours. The paper's throughput results (Figure 4 right, Table 9) translate directly to cost savings: at 100k context with batch size 32, OSCAR achieves 6.17× higher job-level throughput than BF16 on Qwen3-4B-Thinking and 7.83× on GLM-4.7-FP8. For a batch inference job processing 10,000 queries at 100k context on 8×H100 with GLM-4.7, this means the job completes in approximately 1/8 the wall-clock time (or requires 1/8 the GPU count), directly reducing cloud-compute costs by the same factor. The accuracy impact is negligible at this model scale (OSCAR: 78.16 vs. BF16: 77.89 on GLM-4.7, Table 2), meaning the cost savings come with essentially no quality tradeoff. The calibration cost (a single 8k-token forward pass) is amortized over millions of inference tokens and is effectively zero. A practical deployment would: (1) calibrate OSCAR once offline on the smallest model in the family, (2) verify that the rotation transfers to the target model (or re-calibrate if not), (3) deploy with the fused SGLang kernel, and (4) realize 6–8× throughput improvement on long-context batch workloads relative to BF16 serving.

Prefix-cache-heavy serving workloads in multi-turn agentic applications. Many emerging LLM applications — coding agents (Devin-style), multi-turn conversational assistants with long system prompts, document-grounded QA with large shared context — exhibit high prefix-cache hit ratios where multiple requests share a common long prefix. The paper's Figure 6 demonstrates that OSCAR maintains throughput advantages across all prefix-cache hit ratios (0% to ~100%), and that high cache-hit ratios amplify the decode-speed benefit because prefill recomputation is minimized. In a coding-agent deployment where 100 users share a common repository context (e.g., a 100k-token codebase loaded as a shared prefix), the prefix is precomputed and cached once; subsequent user-specific queries generate relatively short outputs. The dominant cost is decode attention over the shared 100k-token prefix, which is fully INT2-quantized under OSCAR. The 3.08× decode speedup at batch size 1 (Figure 4 left) directly translates to 3× lower per-query latency and 3× higher throughput, while the 8× KV-cache memory reduction enables serving more concurrent users per GPU under the same memory budget. The accuracy of agentic tool-use or code-generation outputs is benchmarked in the paper's LiveCodeBench v6 results: on Qwen3-8B, OSCAR achieves 46.32 vs. BF16 49.01 (a 2.69-point gap), and on GLM-4.7-FP8, 52.63 vs. 49.12 (OSCAR slightly exceeds BF16). The cost-accuracy tradeoff favors OSCAR heavily at large model scales, and is acceptable at small model scales for non-critical coding tasks.

Streaming long-context applications with dynamic cache management. The paper's system design includes a fused kernel for window management (Section 4): as decoding advances, the oldest recent-window token is demoted from BF16 to INT2 in a single kernel launch. This supports streaming applications where the context window slides continuously — for example, real-time transcription with long-term context retention, live document editing with undo-history, or continuous monitoring of sensor feeds with historical reference. In these settings, the KV cache grows unboundedly as new tokens arrive, and the 8× memory compression is essential for preventing GPU out-of-memory errors. The paper's batch scalability stress test (Figure 5) shows that OSCAR's uniform INT2 pages enable scaling to 28 concurrent requests with 100k-token inputs on a single H100, while BF16 runs out of memory at much smaller batch sizes. For a streaming deployment with unbounded context growth, the memory savings enable the system to retain longer history before triggering eviction or summarization, improving downstream task accuracy on queries that depend on distant context. The RULER-NIAH results (Table 3) characterize how retrieval accuracy degrades with context length under OSCAR: on Qwen3-8B, retrieval accuracy drops from 99.5% at 4k to 45.0% at 128k. A practitioner deploying streaming retrieval would use these numbers to set context-length limits where accuracy remains acceptable for their use case, and potentially trigger context summarization or eviction when the expected retrieval accuracy drops below a threshold.

When to Prefer This Method

The paper positions OSCAR as deployable INT2 KV-cache quantization that preserves paged-attention and fused-kernel compatibility, in contrast to channel-wise methods (KIVI, Kitty) that break these properties, and in contrast to rotation-only methods (QuaRot) that collapse at 2 bits. The decision rule for practitioners is:

• Prefer OSCAR over channel-wise methods (KIVI, KVQuant, Kitty) when you need INT2 KV-cache compression and are serving through standard frameworks (SGLang, vLLM) that depend on paged-attention, prefix caching, and fused decode kernels for throughput. OSCAR achieves comparable accuracy (Table 1: 66.67 on Qwen3-8B AIME25 vs. Kitty's 59.67 and KIVI-KV2*'s 57.67, at similar BPE) while maintaining the uniform dense paged layout that enables efficient serving. If you are deploying in a research setting where serving throughput is secondary and accuracy is the sole metric, channel-wise methods may provide marginally better accuracy at the cost of deployability, but the paper does not provide a direct controlled comparison in the same serving setup.

• Prefer OSCAR over QuaRot/Hadamard-only INT2 rotation when you are quantizing to 2 bits and need usable accuracy. The accuracy gap is not marginal — QuaRot-INT2 achieves 1.40 mean accuracy on Qwen3-4B-Thinking and 10.14 on Qwen3-8B, while OSCAR achieves 71.86 and 69.42 respectively (Table 2). The only scenario where QuaRot-INT2 might be acceptable is on very large models (GLM-4.7-FP8) where even Hadamard-only INT2 achieves 75.14 mean accuracy (Table 2) — here, the complexity of OSCAR's offline covariance estimation may not justify the small accuracy improvement (+2.81 points to 77.95). The calibration cost (one 8k-token forward pass plus eigendecomposition) is the primary additional engineering burden; if this cost is acceptable, OSCAR should be preferred at all model scales below ~100B parameters.

• Prefer Saw-INT4 (4-bit) over OSCAR when your accuracy requirements are stringent enough that a 1–2 point gap at small model scales is unacceptable. On Qwen3-8B, Saw-INT4 achieves 69.97 mean accuracy vs. OSCAR's 69.42 (Table 2), a 0.55-point advantage, but at 4.25 BPE vs. OSCAR's 2.28 BPE — roughly half the compression ratio. On Qwen3-32B, Saw-INT4 achieves 74.43 vs. OSCAR's 74.17 (a 0.26-point difference), again at nearly 2× the bit budget. If your serving infrastructure has sufficient GPU memory that 4× KV-cache compression is sufficient (i.e., you are not running out of memory at your target context length and batch size), the marginal accuracy advantage of 4-bit quantization may justify the higher memory cost. If memory is the binding constraint, OSCAR's 8× compression provides substantially more headroom for longer contexts or larger batches. On GLM-4.7-FP8, all INT2/INT4 methods are essentially tied with BF16, and the choice reduces to throughput optimization.

• Prefer OSCAR with the default calibration setup (8k GPQA-Diamond tokens, S_0 = 64, W = 256, ρ_K = 0.96, ρ_V = 0.92, group size 128) as a drop-in starting point. The paper demonstrates that this configuration works robustly across four model scales and five benchmarks without per-task tuning (Table 2), and the calibration procedure is robust to calibration volume and domain (Table 7). Practitioners deploying on new model architectures should: (1) run the default calibration, (2) evaluate on a small held-out benchmark sample at 32k generation, (3) if accuracy is acceptable, deploy directly; if not, run the hyperparameter sweep (Tables 4–6) on the new model to find the optimal configuration. The open-source code release provides the SGLang kernel and calibration scripts, reducing the implementation burden to a single offline pass plus kernel integration.