Inference Optimization2024~117 minarXiv:2410.05265

PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization

score 9Mengzhao Chen, Yi Liu, Jiahao Wang +3↑ 33★ 172Oct 7, 2024cs.LGhttps://github.com/chenmnz/prefixquant

🎯 Pitch

Just two outlier tokens cause 95% of 4-bit quantization error in LLMs—PrefixQuant eliminates this by simply prefilling the KV cache with high-frequency outlier tokens, dropping the error from 7.88 to 0.04 without any training. This static-per-tensor trick surpasses existing dynamic quantization methods, unlocking over 2× faster inference.

1. Executive Summary

This paper proposes PrefixQuant, a quantization method for large language models that isolates token-wise outliers—activation values of extreme magnitude occurring at specific token positions that dominate quantization error—by prefixing high-frequency outlier tokens in the KV cache before inference, a training-free process that constrains outliers to predictable positions and enables their offline precomputation. Evaluated on Llama-2, Llama-3, Mistral-7B-v0.3, and Qwen-2-7B models across W4A4KV4 and W4A8KV4 precision, PrefixQuant reduces 4-bit quantization error from 7.88 (with Hadamard rotation alone) to 0.04 on representative activations and achieves an average accuracy improvement of +3.08 points over SpinQuant on five zero-shot reasoning tasks with Llama-3-8B while enabling up to 2.74× prefilling and 2.16× decoding speedups relative to FP16. Critically, PrefixQuant identifies and targets both upper outlier tokens in down-projection inputs and transformer block outputs and lower outlier tokens in self-attention queries and keys, establishing that isolating token-wise outliers through strategically prefixed KV cache entries enables static per-tensor quantization to surpass prior per-token dynamic quantization methods—but only when the prefixed tokens are selected based on their frequency of occurrence as outliers rather than random or purely positional heuristics.

2. Context and Motivation

The Core Problem: Token-Wise Outliers Dominate Quantization Error

The fundamental problem PrefixQuant addresses is deceptively specific but practically devastating: a tiny handful of token positions in an LLM's activations can cause the vast majority of quantization error, and existing methods fail to adequately control them. The paper opens with a striking example (Figure 1): in a 2048-token input sequence processed through the second transformer block of Llama-2-7B, just 2 outlier tokens account for 94.7% of the total 4-bit per-token dynamic quantization error (measured via mean squared error on the block's output). The remaining 2046 tokens collectively contribute only 5.3% of the error. This is not a marginal effect—it is a dominant bottleneck where two positions determine almost the entire accuracy cost of quantization.

To appreciate why this matters, we need to understand what token-wise outliers are and how they differ from the better-studied channel-wise outliers. When we say "channel-wise outliers," we mean that certain feature dimensions (channels) consistently exhibit large activation magnitudes across many or all tokens. These are structural properties of individual neurons or feature maps—think of a neuron that persistently fires at high magnitude regardless of the input text. Channel-wise outliers have received extensive attention because they are predictable: if channel 512 is an outlier, it will be an outlier for every token in every sequence. This predictability enables a variety of mitigation strategies, including keeping outlier channels in higher precision (Dettmers et al., 2022; Zhao et al., 2023), scaling activations channel-wise to transfer magnitude from activations to weights (Xiao et al., 2023a; Shao et al., 2023), or applying orthogonal transformations like Hadamard rotations to redistribute outlier energy uniformly across all channels (Ashkboos et al., 2024b; Liu et al., 2024b).

Token-wise outliers are fundamentally different and far more insidious. Rather than being tied to specific feature dimensions, they are tied to specific token positions in the input sequence. A token-wise outlier is a position where the maximum activation value across all channels is dramatically larger (or smaller) than at other positions. The paper defines this precisely in Equation 3: for an activation tensor $X \in \mathbb{R}^{T \times C}$ (T tokens, C channels), compute per-token maximum values $M \in \mathbb{R}^T$ , then define the outlier ratio $R_i = M_i / \text{median}(M)$ . A token is classified as an upper outlier if $R_i > \eta_1$ (the paper uses $\eta_1 = 64$ ) and as a lower outlier if $R_i^{-1} > \eta_2$ ( $\eta_2 = 8$ ). In other words, an upper outlier token's maximum activation is at least 64 times larger than the median token's maximum, and a lower outlier token's maximum is at least 8 times smaller.

The critical distinction from channel-wise outliers is their unpredictability across inputs: as Figure 4c shows, while outlier tokens consistently include the very first token (position 0), additional outlier tokens appear at variable positions—sometimes at position 1-20, sometimes at 21-40, sometimes elsewhere entirely. This position depends on the input sequence content. Unlike outlier channels, which occur at fixed channel indices and can therefore be isolated offline regardless of the input, token-wise outliers cannot be predetermined by position alone. You cannot simply reserve certain token positions for higher precision because you don't know which positions will be outliers until you see the input.

Why This Problem Is Important

The practical stakes are high for several interconnected reasons that the paper both implicitly and explicitly addresses.

Quantization is the primary deployment enabler for LLMs. Large language models are prohibitively expensive to run without compression. An FP16 Llama-2-70B requires approximately 140GB of memory for weights alone—well beyond the capacity of consumer GPUs and expensive even for datacenter deployments. Quantization to 4-bit precision (W4A4) reduces this by roughly 4×, making deployment feasible on hardware like the RTX 3090 used in the paper's inference benchmarks. But this compression is only valuable if accuracy doesn't collapse. The token-wise outliers documented in Figures 1 and 2 create a direct accuracy-compression tradeoff: you can compress, but two tokens out of 2048 will destroy your model's output quality.

Token-wise outliers disable efficient static quantization for activations. This is arguably the paper's deepest technical concern. Activation quantization comes in two flavors with a crucial efficiency difference. Dynamic quantization (also called per-token dynamic) computes quantization parameters (scaling factor $s$ and zero-point $z$ from Equation 1) online for each token during inference. It adapts to each token's value range individually, which provides robustness to outlier tokens—an outlier's extreme values only affect its own quantization parameters. The cost is that $s$ and $z$ must be computed on-the-fly, adding per-token overhead and preventing certain operator fusions that offline precomputation would enable.

Static quantization (also called per-tensor static) precomputes a single $s$ and $z$ for an entire activation tensor offline using a calibration dataset. During inference, these fixed parameters are applied uniformly. This is significantly more efficient—no online computation of quantization parameters, easier operator fusion—but it is catastrophically vulnerable to token-wise outliers. A single outlier token with values 1000× the norm will force the shared scaling factor to accommodate those extremes, causing all normal tokens to be crushed into a tiny fraction of the quantization grid and lose essentially all precision. The paper demonstrates this brutally in Table 5: moving from dynamic to static quantization with RTN (round-to-nearest) alone drives WikiText2 perplexity from 12.66 to 29.07 for W4A8KV4 and from 1282.34 to 141.02 for W4A4KV4.

This matters because static quantization would unlock significant additional speedups and deployment efficiencies. The paper shows 2.74× prefilling and 2.16× decoding speedups (Table 6) using its static quantization approach (PrefixQuant-O2) over FP16. Prior methods that achieve comparable accuracy require dynamic quantization (Table 1), sacrificing these efficiency gains. PrefixQuant's core claim—backed by the experimental results—is that eliminating token-wise outliers makes static per-tensor quantization not just viable but superior to prior dynamic methods.

Token-wise outliers cause instability during quantization fine-tuning. Beyond the direct quantization error, Section 4.1 notes that outlier tokens "cause instability during block-wise fine-tuning." This is an understated but important point: when you do block-wise reconstruction (training quantized blocks to match the full-precision block's output), extreme activation values create large gradients that destabilize the optimization. This limits how much fine-tuning can compensate for quantization error—the process itself becomes unreliable in the presence of outliers.

Where Prior Approaches Fall Short

The paper's literature review and experimental setup identify specific limitations in existing work.

Channel-wise methods are necessary but insufficient. Hadamard rotation, as popularized by QuaRot (Ashkboos et al., 2024b) and refined by SpinQuant (Liu et al., 2024b), applies orthogonal transformations that redistribute outlier energy across channels. Figure 2(b) demonstrates the effect: rotation reduces the maximum activation value from over 1,000 to approximately 15—a dramatic improvement. However, the paper quantifies the residual problem precisely: even after rotation, the quantization error remains 7.88 (MSE on block output), which is still nearly 200× larger than PrefixQuant's 0.04. The reason is visible in the $\text{max}(\text{top-1}/\text{median})$ ratios in Figure 3: after rotation, the down_proj input activations still show a ratio of 461 (Figure 3(I.b)), down from 4,161 without rotation but still indicating that the largest token-wise maximum is over 400 times the median. Similarly, the output projection activations (Figure 3(II.b)) show fundamentally unchanged lower outlier patterns—Hadamard rotation redistributes magnitude across channels but does not compress per-token maximum values toward uniformity. The token-wise structure persists because rotation transforms dimensions, not positions.

Existing token-wise methods are computationally prohibitive or limited in scope. Several prior works recognized the token-wise outlier problem but proposed solutions the paper considers impractical:

Gated attention mechanisms (Bondarenko et al., 2024; Sun et al., 2024) modify the model architecture to suppress attention to outlier-causing tokens, requiring model retraining. The paper explicitly contrasts its training-free outlier isolation with these approaches (Section 2, "Token-Wise Outliers" paragraph).
Grid search methods like QFeP (Yang et al., 2024) and CushionCache (Son et al., 2024) identify which tokens to prefix through expensive grid searches over token combinations. The paper provides specific timing comparisons: QFeP takes roughly 12 hours for Llama-3-8B, while PrefixQuant's frequency-based token selection completes in 12 seconds—a 3,600× speedup.
Prior prefixing approaches are W8A8-specific and underperform. Table 13 shows that both QFeP and CushionCache already suffer significant perplexity degradation even at W8A8 precision (e.g., CushionCache achieves 7.37 WikiText2 perplexity on Llama-3-8B vs. PrefixQuant-O2's 6.17). At the lower precisions targeted by PrefixQuant (W4A4, W4A8), these methods were either not demonstrated or would presumably degrade further.

The lower outlier problem is almost entirely unaddressed. Section 4.1 introduces a crucial observation that distinguishes PrefixQuant from prior token-wise outlier work: while previous methods focused exclusively on large-magnitude outliers ("upper outliers"), the paper identifies lower outliers—tokens with abnormally small activation magnitudes—specifically in the query (Q) and key (K) projections of self-attention. Figure 3(III) shows that K values exhibit $\text{max}(\text{median}/\text{min-1}) > 9$ across multiple layers, and even Hadamard rotation (Figure 3(III.b)) does not mitigate these lower outliers—the ratio remains essentially unchanged. Section 3 in the H visualizations (Figures 10, 12, 14, 16, 18, 20) documents this pattern consistently across all tested models: the Q and K tensors show modest upper outlier ratios ( $\text{max}(\text{top-1}/\text{median}) \approx 1.4$ – $2.3$ ) but substantial lower outlier ratios ( $\text{max}(\text{median}/\text{min-1})$ ranging from 7.6 to 23.5 depending on the model). This is significant for quantization because extremely small values also cause precision loss: the quantization step size $s$ must accommodate the full range from minimum to maximum (Equation 2), so a few tokens with near-zero values inflate the quantized range just as extreme large values do.

No existing method combines token-wise outlier elimination with static quantization at low precision. The paper's Table 1 makes explicit what prior methods actually quantize: SmoothQuant, Atom, QoQ, QuaRot, and SpinQuant all use per-token dynamic activation quantization. The state of the art before PrefixQuant simply could not achieve competitive accuracy with per-tensor static quantization—the token-wise outlier problem made it impossible. PrefixQuant's positioning is therefore twofold: it introduces a new, efficient mechanism for eliminating token-wise outliers, and it shows that this mechanism finally makes static quantization outperform dynamic quantization at W4A4 and W4A8 precision.

How This Paper Positions Itself

The paper positions PrefixQuant as a complementary, not competing, approach to channel-wise methods. The authors explicitly build on the Hadamard rotation framework from QuaRot/SpinQuant (Section 4, opening paragraph: "We build our method upon the Hadamard rotation technique"), adopting the same rotation scheme (R1, R2, R3, R4 as detailed in Appendix B and Figure 7). The innovation is additive: Hadamard rotation handles channel-wise outliers, PrefixQuant handles the token-wise outliers that rotation leaves behind. The ablation in Table 5 makes this architectural layering explicit, showing that rotation alone reduces W4A4KV4 perplexity from 1282.34 to 24.98, and PrefixQuant's prefixed tokens further reduce it to 7.53.

The paper also positions itself as making a conceptual contribution to outlier taxonomy. Section 4.1 is unusual for a quantization paper in its depth of outlier characterization: it separately analyzes the number, position, content, and statistical properties (ratio to median) of outlier tokens across multiple model families, distinguishing upper from lower outliers and documenting where each type occurs (down_proj inputs, transformer block outputs, Q/K tensors). This analysis reveals a finding with practical consequences: upper outlier tokens correspond to the same sequence positions as lower outlier tokens but exhibit different patterns in different modules. This means that a single set of prefixed tokens can simultaneously address both types of outliers—you don't need separate handling for Q/K lower outliers vs. down_proj upper outliers, because both are linked to the same token positions.

Finally, the paper frames the training overhead as a key differentiator from prior token-wise methods. The abstract emphasizes that outlier token detection is "training-free and highly efficient (e.g., 1 minute for Llama-3-70B)." Section 4.2 reinforces this by contrasting with "training-based methods that modify SoftMax behavior or add attention bias." The implicit argument is: there are known architectural causes of token-wise outliers (the SoftMax mechanism in self-attention forcing non-zero attention on "attention sink" tokens, as described by Xiao et al., 2023b and Sun et al., 2024), but rather than changing the architecture to prevent outliers from forming, it is simpler and more efficient to accept that they form and then quarantine them to prefixed, precomputed positions where they can do no harm to the rest of the sequence's quantization. This is an elegant engineering tradeoff: architectural retraining would be the principled solution, but prefixing achieves the same practical outcome with negligible cost.

3. Technical Approach

3.1 Reader Orientation

PrefixQuant is a quantization pipeline for large language models that eliminates token-wise outliers by strategically prefixing specific high-frequency tokens to every input sequence and storing their precomputed activations in the KV cache, coupled with block-wise fine-tuning that trains both weight and activation quantization parameters to recover accuracy. It solves the problem that a tiny handful of token positions (as few as 2 out of 2048) account for over 94% of quantization error, making static per-tensor quantization catastrophically inaccurate, by quarantining these outliers to predictable, precomputed positions rather than trying to suppress or redistribute them—a fundamentally different strategy from prior work.

3.2 Big-Picture Architecture (Diagram in Words)

PrefixQuant consists of five major components arranged in a pipeline that transforms a raw pretrained LLM into an efficiently quantized model:

Outlier Token Detector — scans a small calibration dataset through the full-precision model to identify which token types (specific strings like "\n", ".", "the") consistently produce activation outliers, determining both the count o and content of prefixed tokens needed per model. This runs once offline and takes 12 seconds for Llama-2-7B, 1 minute for Llama-3-70B.
Prefixed Token KV Cache Computer — passes the selected outlier tokens as a dummy prefix through the full-precision model once to compute their key and value activations at every layer, storing these as k' and v' (Equation 4). These precomputed KV entries are frozen and reused for all subsequent inference.
Hadamard Rotation Engine — applies orthogonal transformations (R1–R4 in Figure 7) to redistribue channel-wise outliers across all channels, following the QuaRot/SpinQuant framework. Absorbable rotations (R1, R2) are folded into adjacent weight matrices offline; online rotations (R3, R4) execute during inference via fast Walsh-Hadamard transforms.
Quantized Model with Prefixed Injection — takes the base LLM, applies the Hadamard rotations, quantizes weights and activations to 4 or 8 bits, and modifies every transformer block's self-attention to prepend the precomputed k', v' to the KV cache before computing attention. The prefixed tokens' KV entries remain in full precision; all other tokens are quantized.
Block-wise Fine-Tuning Module — trains each transformer block sequentially to minimize MSE between its quantized output and the full-precision block's output, jointly optimizing full-precision weights, weight quantization step sizes, and activation quantization parameters (clipping factors for dynamic quantization, step sizes and zero-points for static quantization).

Information flows through the system in three phases:

Phase 1 (Offline, per-model): Calibration data → Outlier Token Detector → Prefixed Token KV Cache Computer (stores k', v' for all layers). Calibration data also → Grid Search Initialization (finds optimal clipping factors or step sizes per layer/block). Total time: ~0.2–12 minutes for detection + 0.7–12 minutes for grid search.

Phase 2 (Offline training): Training data passes through each transformer block sequentially. For each block, the full-precision block output is the target; the quantized block (with prefixed tokens injected into self-attention) is the student. MSE loss backpropagates through quantization parameters and full-precision weights. Each block trains independently before moving to the next. Total time: ~2.2 hours for Llama-3-8B, ~17 hours for Llama-3-70B.

Phase 3 (Online inference): Input tokens arrive → prefixed tokens prepended to sequence → Hadamard online rotations applied at R3/R4 positions → quantized forward pass through all blocks, with precomputed k', v' injected into self-attention at each layer → output logits.

3.3 Roadmap for the Deep Dive

First, the formal definition of outlier tokens (Equation 3) and the systematic characterization of their properties (number, position, content, statistical ratios) across model families, because choosing which tokens to prefix depends on understanding exactly what makes a token an outlier and how outliers manifest differently in different activation tensors.
Second, the prefixed outlier mechanism itself—how detection works, how tokens are selected for prefixing, how the KV cache computation and injection modify the self-attention forward pass (Equation 4), and crucially, why prefixing works (it constrains outliers to known positions where they can be handled offline without affecting the quantized forward pass).
Third, the integration with Hadamard rotation (Figure 7), because PrefixQuant is not a standalone replacement but a layer built on top of the QuaRot/SpinQuant channel-wise outlier handling framework, and understanding the interaction—rotation handles channel dimensions, prefixing handles token positions—is essential.
Fourth, the grid search initialization for quantization parameters, which establishes the starting point for fine-tuning and which interacts critically with the prefixed tokens: without prefixed tokens, the grid search itself would be corrupted by extreme outlier values.
Fifth, the block-wise fine-tuning procedure, including what parameters are trainable under PrefixQuant-O1 (dynamic) and PrefixQuant-O2 (static), how trainable clipping factors work for dynamic quantization, and what the training objective and hyperparameters are.
Sixth, the two quantization configurations (O1 and O2), because the paper makes a strong claim that O2 (static per-tensor) surpasses prior dynamic methods, and understanding what enables this requires knowing exactly what is quantized and at what granularity.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a practical quantization method paper whose core idea is that token-wise outliers can be eliminated not by suppressing them or retraining the model, but by moving them to predictable positions where their activations can be precomputed and isolated from the quantized forward pass.

Formal Definition and Characterization of Outlier Tokens

The paper defines outlier tokens through a statistical criterion applied to per-token maximum values. Given an activation tensor $X \in \mathbb{R}^{T \times C}$ where $T$ is the sequence length (number of tokens) and $C$ is the channel dimension, the analysis proceeds in two steps. First, compute the per-token maximum magnitudes:

$M_i = \max_{j \in [1, C]} |X_{i,j}|$

where $M_i$ is the maximum absolute activation value for the $i$ -th token across all its $C$ channels. This produces a vector $M \in \mathbb{R}^T$ capturing the extremal behavior of each token position. Second, define the outlier ratio relative to the median of these maxima:

$R_i = \frac{M_i}{\text{median}(M)}$

where $\text{median}(M)$ is the median of the $T$ per-token maximum values.

What it computes: for each token position $i$ , the ratio $R_i$ measures how extreme its maximum activation is compared to the "typical" token's maximum. A ratio of 1 means the token's peak activation equals the median token's peak—it is perfectly normal. A ratio of 64 (the paper's threshold $\eta_1$ ) means the largest magnitude in that token is 64 times the median token's largest magnitude—the token contains values far outside the distribution of normal tokens. For lower outliers, the inverse ratio $R_i^{-1}$ is compared against $\eta_2 = 8$ : if a token's maximum is 8 times smaller than the median token's maximum, it has abnormally small activations.

Why this form: using per-token maximums rather than, say, per-token means or variances directly targets what matters for quantization. Quantization error in the uniform quantization scheme of Equation 1 is dominated by the range of values—the step size $s$ must span from $\beta \min(X)$ to $\gamma \max(X)$ . A single extreme value at one channel in one token forces $s$ to accommodate that range, crushing all other values into fewer effective bits. The ratio-to-median formulation is scale-invariant: multiplying all activations in a layer by a constant factor (which could happen with weight scaling) does not change $R_i$ , so the outlier classification is robust to the overall magnitude of the layer. Using the median rather than the mean as the reference is important because the mean would itself be pulled by outliers, creating a moving target.

The paper uses thresholds $\eta_1 = 64$ and $\eta_2 = 8$ based on empirical observation but provides a principled justification through Figure 3. The visualizations show that with these thresholds, the method cleanly separates the 1–4 outlier positions from the remaining 2044+ normal positions across all tested models, as verified by the post-prefixing ratios dropping from hundreds/thousands to single digits (Figure 3, right column showing $\max(\text{top-1}/\text{median})$ ratios of 2.4–5.3 after prefixing vs. 461–4,359 before).

Systematic characterization across modules. The paper goes beyond a single definition to identify three distinct types of outlier manifestations, each occurring in different activation tensors within the transformer:

(I) Upper outlier tokens in down_proj inputs and transformer block outputs. These are the classic "massive activations" described in prior work (Sun et al., 2024). Figure 3(I) shows that in the raw Llama-2-7B, the down_proj input activations exhibit $\max(\text{top-1}/\text{median}) = 4161$ —the most extreme token has a maximum value over 4000 times the median token's maximum. Even after Hadamard rotation (Figure 3(I.b)), this ratio remains at 461, meaning the outlier persists at a level that would dominate 4-bit quantization (where only 16 discrete levels can represent the entire range). The output of each transformer block (Figure 3(II)) shows a similar but less extreme pattern, with $\max(\text{top-1}/\text{median}) = 11.7$ originally, reduced to 4.6–5.1 after rotation but still indicating non-uniformity. These outliers are the primary target of the prefixing mechanism because they directly cause the 94.7% quantization error dominance shown in Figure 1.

(II) Lower outlier tokens in Q and K projections. Figure 3(III) reveals a phenomenon that the paper claims has been overlooked in prior work on token-wise outliers. In the key projections of self-attention, some tokens have abnormally small maximum values: $\max(\text{median}/\text{min-1}) > 9$ for Llama-2-7B. This means the ratio between the median token's maximum and the smallest token's maximum exceeds 9—these tokens have activations concentrated near zero. Importantly, Hadamard rotation (Figure 3(III.b)) does not mitigate these lower outliers: the ratio remains essentially unchanged. This is because rotation redistributes magnitude across channels but preserves the overall L2 norm per token—if a token has small values in all channels before rotation, it will have small values in all channels after rotation, just in different linear combinations. The same pattern appears in Q projections (Figure 10 in the appendix, showing $\max(\text{median}/\text{min-1}) \approx 9.0$ for Llama-2-7B Q). Lower outliers matter for the same reason upper outliers do: the quantization step size $s = (\gamma\max(X) - \beta\min(X)) / (2^N - 1)$ is determined by both the maximum and minimum of the tensor. Tokens with near-zero minimums expand the quantization range just as tokens with extreme maximums do.

What happens after prefixing: Figure 3(c) shows that injecting prefixed tokens into the KV cache addresses both types simultaneously. The $\max(\text{top-1}/\text{median})$ ratio for down_proj inputs drops from 461 to 2.4. The $\max(\text{median}/\text{min-1})$ ratio for K drops from >9 to <3.3 (exact values: 2.2 for Q, 3.3 for K in Figure 10(c)). The ratio for V drops from 4.5 to 1.6. These are dramatic improvements—the token-wise maximums become nearly uniform, making the activation distribution genuinely quantization-friendly.

Token-level correlation between upper and lower outliers. The paper notes in Section 4.1 that "both upper outlier tokens and lower outlier tokens correspond to tokens at the same position in the sequence, but they exhibit different patterns in different modules." This is a crucial observation that simplifies the method: you only need to detect and prefix one set of token types, because those same positions will benefit all modules where outliers occur, even though the outlier manifests as large values in some layers and small values in others.

Characterizing Outlier Token Properties Across Models

The paper systematically analyzes three properties of outlier tokens across model families: their cardinality, positional distribution, and semantic content.

Number of outlier tokens. The detection procedure counts how many token positions in a calibration dataset exceed the $R_i > \eta_1$ threshold (for upper outliers, which are easier to detect robustly) within each transformer block. Let $O \in \mathbb{R}^b$ be the vector of average outlier counts per block, where $b$ is the number of transformer blocks. Then the number of prefixed tokens is set as:

$o = \lceil \max_{l \in [1,b]} O_l \rceil$

where $\lceil \cdot \rceil$ denotes the ceiling function.

What it computes: the maximum, across all transformer blocks, of the average number of outlier token positions observed in the calibration data. The ceiling ensures $o$ is an integer count of tokens to prefix.

Why this form: outlier tokens are "nearly consistent across layers that contain them" (Section 4.1), meaning if block 5 has 2 outlier positions and block 17 has 3, all outlier positions are covered by prefixing 3 tokens. Using the max across blocks ensures the prefix count is sufficient for the worst-case block. The ceiling handles cases where the average count is fractional (e.g., if some calibration samples have 2 outliers and some have 3, the average might be 2.3, and prefixing 3 tokens covers all cases). Figure 4a shows the empirical distribution: for Llama-2-7B, 96.8% of inputs have exactly 2 outlier token positions, 1.6% have 1, and 1.6% have 3. So $o = 3$ after adding the mandatory [BOS] token.

Position of outlier tokens. Figure 4c visualizes the empirical distribution of outlier token positions in Llama-2-7B: 50.0% occur at position 0 (the initial token), 30.5% at positions 1–20, 10.2% at positions 21–40, and 9.3% at other positions. The concentration near the start of the sequence motivates the prefixing strategy—since outliers naturally cluster at early positions, prepending outlier-prone tokens to the sequence is consistent with the model's existing attention patterns and unlikely to cause unexpected behavior.

Content of outlier tokens. Figure 4b analyzes the semantic content of outlier tokens in Llama-2-7B, excluding the initial position 0 (which is always an outlier regardless of content). The two dominant categories are delimiter tokens: "." (period, 71.4%) and "\n" (newline, 28.6%). More broadly, Figure 8 in the appendix shows that across models, outlier tokens fall into a consistent set of low-semantic, high-frequency delimiters and function words: for Llama-2-13B the top content is "the" (47.6%), "." (24.7%), "," (16.3%), "\n" (7.3%); for Llama-2-70B it is "\n" (57.6%), """ (9.9%), " " (6.3%), "1" (2.9%), "2" (2.6%); for Llama-3-70B it is "," (39.8%), " " (16.3%), " the" (11.3%), " a" (10.9%), "." (8.1%); for Mistral-7B-v0.3 it is "\n" (64.3%), "," (7.0%), "." (5.6%), "to" (4.2%), "of" (3.5%). A critical observation is that "tokens corresponding to the same text may exhibit different patterns depending on their position in the sequence" — a "\n" at position 5 might be an outlier, while a "\n" at position 500 might be normal, because the outlier property is driven by the attention mechanism's early-token bias, not by the token's identity alone.

Absentee outlier tokens in Llama-3-8B and Qwen-2-7B. The paper notes that some models (Llama-3-8B, Llama-3-8B-Instruct, Qwen-2-7B) exhibit outlier tokens only at the initial position—they have no non-initial outlier tokens. For these models, the number of prefixed tokens $o$ is set to 1, with the prefix consisting solely of the [BOS] token. This is a robustness check: PrefixQuant works even when the only outlier is the universally present first token, confirming that the method correctly adapts to model-specific outlier behavior rather than imposing a fixed prefix regardless of actual patterns.

Final prefixed token configurations (Table 5 in the main paper, copied as Figure 5). The per-model prefixed tokens, determined by taking the top-o high-frequency outlier tokens (excluding position 0) and appending [BOS], are:

Llama-2-7B: 3 tokens: ".", "\n", [BOS]
Llama-2-13B: 3 tokens: "the", ".", [BOS]
Llama-2-70B: 4 tokens: "\n", """, [BOS] (note: 3 listed in figure but o = \lceil \max(O) \rceil yields 4 with BOS)
Llama-3-8B / Llama-3-8B-Instruct: 1 token: [BOS]
Llama-3-70B / Llama-3-70B-Instruct: 3 tokens: ",", " ", [BOS]
Mistral-v0.3-7B: 4 tokens: "\n", ",", ".", [BOS]
Qwen-2-7B: 1 token: [BOS]

The Prefixed Token Mechanism: Detection, Injection, and Why It Works

The prefixed token mechanism operates in three stages: detection (identifying which tokens to prefix), KV cache precomputation (computing the frozen key-value entries), and injection (modifying the quantized forward pass to include the prefixed tokens).

Stage 1: High-frequency outlier token detection. The detection is based on a small calibration dataset (8 samples from the Pile with 1024-token sequences, the same data used for grid search initialization). For each sample, the method passes the input through the full-precision model and, for each transformer block, identifies which token positions satisfy $R_i > \eta_1 = 64$ (the upper outlier threshold). Across all samples and blocks, a frequency table of outlier token types is constructed, excluding position-0 tokens (since the first token is always an outlier and is handled by the mandatory [BOS] prefix). The top- $o$ most frequently occurring token types are selected as the prefixed tokens.

The computational cost is dominated by a single forward pass of the calibration data through the full-precision model—no gradient computation, no backpropagation, no weight updates. The paper reports 12 seconds for Llama-2-7B and 1 minute for Llama-3-70B (Table 6). This contrasts with the 12-hour grid search required by QFeP (Yang et al., 2024) and CushionCache (Son et al., 2024), which test many candidate token combinations to find optimal prefixes. The frequency-based approach works because, as the paper demonstrates in Figure 4d, prefixing the top- $o$ high-frequency tokens constrains all outlier tokens to positions 0 through $o-1$ (the prefixed positions), even though the detection was based on a heuristic (frequency count) rather than an exhaustive search.

Stage 2: KV cache precomputation. Once the prefixed token types are determined, they are assembled into a fixed prefix string (for Llama-2-7B: [BOS] followed by "\n" and "." in the order shown in Figure 5). This prefix string is passed through the full-precision model once, and at each transformer block, the key and value activations for these $o$ prefix tokens are stored. Let these stored tensors be $k'_\ell, v'_\ell \in \mathbb{R}^{o \times d_h}$ for each layer $\ell$ , where $d_h$ is the per-head dimension. These stored KV entries remain in full precision (FP16) even when the rest of the model is quantized—they are never quantized because they are precomputed offline and reused identically for every inference.

Stage 3: Forward pass injection (Equation 4). During inference with the quantized model, the self-attention computation at each layer is modified to prepend the prefixed KV entries before the input-dependent KV entries. Concretely, for a given attention head with query, key, and value projections $Q, K, V \in \mathbb{R}^{T \times d_h}$ computed from the current input sequence:

$\text{Attention}(Q, K, V; k', v') = \text{Softmax}\left(\frac{Q \begin{bmatrix} K^T & k' \end{bmatrix}}{\sqrt{d_h}}\right) \begin{bmatrix} V \\ v'^T \end{bmatrix}$

Decomposing this equation: the matrix $Q \in \mathbb{R}^{T \times d_h}$ contains the query vectors for the $T$ input tokens. The concatenation $\begin{bmatrix} K^T & k' \end{bmatrix} \in \mathbb{R}^{d_h \times (T + o)}$ stacks the transposed input keys $K^T \in \mathbb{R}^{d_h \times T}$ and the transposed prefixed keys $k' \in \mathbb{R}^{o \times d_h}$ (transposed to $k'^T \in \mathbb{R}^{d_h \times o}$ ). The product $Q[K^T \; k'] \in \mathbb{R}^{T \times (T+o)}$ computes attention scores from every input query to every input key (columns 0 through $T-1$ ) and to every prefixed key (columns $T$ through $T+o-1$ ). The Softmax normalizes these scores row-wise. The result is then multiplied by $\begin{bmatrix} V \\ v'^T \end{bmatrix} \in \mathbb{R}^{(T+o) \times d_h}$ , producing the attention output as a weighted sum of input values and prefixed values.

What it computes: the standard scaled dot-product attention, but with $o$ additional key-value pairs that are identical for every input sequence. These additional pairs act as constant "background" that every attention head can attend to. For outlier tokens that would normally have extreme Q/K/V values, the model can now direct those extreme attention patterns toward the prefixed positions (which are in full precision and don't affect the quantization of other tokens) rather than toward the input-dependent tokens (which are quantized and would be corrupted by outlier magnitudes).

Why this works—the mechanistic explanation. The paper's key insight is that outlier tokens exist because of the attention mechanism's structural properties, specifically the SoftMax function's requirement that $\sum_j \text{softmax}(s)_j = 1$ for each query—every query must distribute 100% of its attention weight somewhere. When certain tokens (like initial tokens or delimiters) serve as "attention sinks" (Xiao et al., 2023b), they absorb large attention weights, which in turn produces large activations in the value-weighted sum. By providing additional, consistently available attention sink tokens in the KV cache, PrefixQuant gives the model a controlled release valve: outlier attention patterns get directed toward the prefixed tokens, whose KV entries are never quantized, so the extreme values they generate don't propagate into the quantized activation tensors. The input-dependent tokens that are quantized only need to represent moderate, quantization-friendly values.

The empirical evidence for this mechanism is Figure 4d: after prefixing, outlier tokens only appear at positions 0 and 1 (the prefixed positions), with all subsequent positions (2 through 2047) showing normal, non-outlier activation distributions. The $\max(\text{top-1}/\text{median})$ ratios in Figure 3(c) confirm that the activation distribution of the non-prefixed tokens becomes genuinely uniform—ratios of 2.4–5.3 compared to 461–4,359 before prefixing.

A subtle point about the [BOS] token requirement. The paper always includes [BOS] (beginning-of-sequence token, e.g., "<s>" for Llama-2 or "|begin of text|" for Llama-3) as part of the prefixed tokens, even when the adaptive detection would select a different number. This is because position 0 is universally an outlier token across all models, and the [BOS] is the content at position 0 for normal sequences. By including it in the prefix, the method ensures that the model's existing attention sink at position 0 is serviced by a precomputed, full-precision KV entry, which is already present in the prefix. Table 9 shows an ablation where using only 2 prefixed tokens (without [BOS]) achieves excellent perplexity for Llama-2-7B (6.22 vs. 6.21 with 3 tokens including [BOS]), but the authors include [BOS] "for consistency and simplicity" across all models.

Integration with Hadamard Rotation

PrefixQuant is built on top of the Hadamard rotation framework from QuaRot (Ashkboos et al., 2024b) as refined by SpinQuant (Liu et al., 2024b). The interaction between rotation and prefixing is architecturally critical: they handle orthogonal sources of quantization error, and the paper's ablation (Table 5) shows that both are necessary for strong performance.

What Hadamard rotation does. A Hadamard matrix $H$ is an orthogonal matrix (so $HH^T = I$ ) whose entries are $+1/\sqrt{d}$ or $-1/\sqrt{d}$ (after appropriate scaling). Orthogonal transformations preserve L2 norms and dot products: for any vectors $x, y$ , we have $(Hx)^T(Hy) = x^T H^T H y = x^T y$ . This property enables the key trick of "absorbable rotations": if you multiply a weight matrix $W$ on the left by $H$ and the corresponding activation on the right by $H^T$ , the output is $HW H^T x$ , which equals $W x$ if $W$ is appropriately transformed. By choosing $H$ to be a Hadamard matrix (or a randomized variant), the transformation redistributes outlier energy from a few channels uniformly across all channels.

The four rotation positions (Figure 7, Appendix B). The paper follows SpinQuant's rotation scheme with four rotation points per transformer block:

R1 (absorbable): Input activations to the query/key/value/gate/up projection layers are rotated. Since the preceding operation is a linear layer (or input embedding), the rotation can be "absorbed" by modifying the weight matrix: $W_{\text{new}} = H W_{\text{old}}$ . This adds zero computation during inference—the absorption happens offline.
R2 (absorbable): Head-wise rotation of the output projection's input activations. Each attention head's output is rotated independently, then the output projection weight absorbs the inverse rotation: $W_o^{\text{new}} = W_o^{\text{old}} \cdot \text{blockdiag}(H_1^{-1}, ..., H_{n_h}^{-1})$ where $n_h$ is the number of heads.
R3 (online): Output activations of Q and K projections after applying Rotary Position Embedding (RoPE). RoPE applies a position-dependent rotation to each pair of channels, which breaks the linearity needed for absorption—if you apply rotation after RoPE, you can't absorb it into the previous weight. Therefore R3 must execute online during inference, using the fast Walsh-Hadamard transform (FWHT) which computes $Hx$ in $O(d \log d)$ time with no multiplications.
R4 (online): Input activation of the down projection layer. Since there's a Swish (SiLU) nonlinearity between the up/gate projections and the down projection, the rotation can't be absorbed backward past the nonlinearity. It executes online.

Why rotation is insufficient alone. Hadamard rotation operates on the channel dimension: it takes the vector of $C$ channel values at a given token position and linearly combines them to produce a new $C$ -vector with approximately uniform magnitude across channels. However, it does nothing to the token dimension—it cannot transfer magnitude from one token position to another. If position 5 has activations that are 500× larger than position 100 across all channels, rotation will make each individual channel at position 5 have roughly uniform magnitude (instead of a few outlier channels dominating), but position 5 will still have 500× larger total magnitude than position 100. This is confirmed by Figures 3 and 9–20: after rotation, the $\max(\text{top-1}/\text{median})$ ratio decreases substantially (e.g., from 4,161 to 461 for down_proj inputs) but remains far above 1, indicating persistent token-position-inhomogeneity.

The complementary division of labor. Hadamard rotation handles channel-wise outliers: before rotation, a few channels at every token position have extreme values; after rotation, each token's channel distribution is roughly uniform. PrefixQuant handles token-wise outliers: before prefixing, a few token positions have much larger activations than others; after prefixing, all non-prefixed token positions have roughly uniform per-token maximums. The two mechanisms together create activation tensors where both the channel dimension and the token dimension are nearly uniform—the ideal condition for uniform quantization. The ablation in Table 5 quantifies this: starting from the "Base" RTN quantization at W4A4KV4 with 1282.34 perplexity, adding rotation alone improves to 24.98, adding PrefixQuant's prefixed outliers on top of rotation improves to 7.53, and adding block-wise fine-tuning on top of both improves to 7.23.

Grid Search Initialization for Quantization Parameters

Before block-wise fine-tuning, PrefixQuant initializes quantization parameters through a grid search that minimizes MSE on a small calibration set. The details differ between dynamic and static quantization settings.

Standard quantization equations. The paper uses the standard uniform affine quantization formulation (Equations 1 and 2 in Section 3):

$X_{\text{INT}} = \text{clamp}\left(\left\lfloor \frac{X}{s} \right\rceil + z, 0, 2^N - 1\right)$

$s = \frac{\gamma \max(X) - \beta \min(X)}{2^N - 1}, \quad z = -\left\lfloor \frac{\beta \min(X)}{s} \right\rfloor$

where $X \in \mathbb{R}^{T \times C}$ is the full-precision activation tensor, $s \in \mathbb{R}$ is the quantization step size (the distance between adjacent integer levels in the full-precision space), $z \in \mathbb{Z}$ is the zero-point (the integer value that maps to 0 in the full-precision space), $\lfloor \cdot \rceil$ is rounding to nearest integer, and $\text{clamp}(\cdot, 0, 2^N-1)$ clips values to the representable integer range. $\gamma \in [0, 1]$ and $\beta \in [0, 1]$ are clipping factors that control what fraction of the activation range is covered: $\gamma = \beta = 1$ corresponds to standard min-max quantization (covering the full range), while $\gamma < 1$ or $\beta < 1$ clips extreme values to reduce step size at the cost of saturating the extremes.

What this computes: a mapping from full-precision values to $N$ -bit integers where each integer represents a bin of width $s$ in the full-precision space. The dequantized value is approximately $s \cdot (X_{\text{INT}} - z)$ .

Why clipping factors matter: in the presence of outliers, min-max quantization ( $\gamma = \beta = 1$ ) makes $s$ very large because $\max(X)$ or $-\min(X)$ is extreme. Most values in the tensor occupy a small fraction of $[0, 2^N-1]$ , effectively using fewer bits. By clipping (setting $\gamma < 1$ for positive outliers or $\beta < 1$ for negative outliers), we reduce $s$ at the cost of saturating the outlier values to the maximum representable integer. This trades clipping error (outlier values being misrepresented) for reduced rounding error (normal values getting finer quantization). Grid search finds the $\gamma, \beta$ that minimize total MSE.

Grid search procedure for dynamic quantization (PrefixQuant-O1). For per-token dynamic quantization, $\gamma$ and $\beta$ are shared across all tokens within a layer—they are tensor-wise parameters, not per-token. The grid search evaluates candidate $(\gamma, \beta)$ pairs on 8 calibration samples from the Pile dataset with 1024-token sequences. For each candidate, the layer's activations are quantized and dequantized using per-token dynamic quantization (computing $s, z$ online for each token using the shared $\gamma, \beta$ ), and the MSE between the original and dequantized layer outputs is computed. The $(\gamma, \beta)$ pair minimizing this MSE is selected.

Grid search procedure for static quantization (PrefixQuant-O2). For per-tensor static quantization, $s$ and $z$ are directly grid-searched rather than searching $\gamma, \beta$ (since there are no clipping factors in the standard static quantization formulation). The calibration process searches over candidate $(s, z)$ values and selects those minimizing the MSE of the block output (not just the layer output). This is because static quantization at the tensor level must account for interactions across layers within a transformer block—a good step size for one layer might interact poorly with the next layer's quantization.

The critical interaction with prefixed tokens. Table 5 reveals why the grid search benefits so dramatically from prefixed outliers. Without prefixed tokens, the grid search operates on activation tensors where a few token positions have values 1000× larger than normal. The search procedure will optimize $\gamma, \beta$ to handle these extremes, but the MSE landscape is dominated by a few outlier positions—the grid search effectively ignores the quantization quality of the 99% normal tokens because any reasonable $(\gamma, \beta)$ produces huge error at the outlier positions. After prefixing isolates outliers to the prefixed tokens (which are in full precision and excluded from the quantized tensors), the grid search operates on a genuinely uniform distribution, allowing it to find $\gamma, \beta$ that accurately serve all tokens. The improvement is dramatic: at W4A4KV4, grid search initialization with rotation alone achieves perplexity 11.70; adding prefixed outliers before grid search improves this to 7.53—a reduction of 4.17 perplexity points from the initialization alone, before any fine-tuning.

Block-wise Fine-Tuning

Block-wise fine-tuning is the final stage that recovers the remaining accuracy gap between the quantized model (after prefixing and Hadamard rotation) and the full-precision model. The procedure follows the block reconstruction paradigm established by Brecq (Li et al., 2021) and adapted to LLMs by OmniQuant (Shao et al., 2023) and EfficientQAT (Chen et al., 2024a).

Training objective. Each transformer block is trained independently to minimize the MSE between its quantized output and the full-precision block's output:

$\mathcal{L}_{\text{block}} = \text{MSE}(Y_{\text{quant}}, Y_{\text{fp}}) = \frac{1}{T \cdot C} \sum_{i=1}^{T} \sum_{j=1}^{C} (Y_{\text{quant}, ij} - Y_{\text{fp}, ij})^2$

where $Y_{\text{fp}} \in \mathbb{R}^{T \times C}$ is the output of the full-precision block (either the hidden states or the block's final output, depending on what is being matched), $Y_{\text{quant}}$ is the output of the same block with quantized weights and activations but otherwise identical inputs, and the loss is the average squared error over all tokens and channels.

What it computes: a scalar measure of how faithfully the quantized block reproduces the full-precision block's output, averaged over all positions and dimensions.

Why MSE: the goal is to preserve the block's output distribution, not to optimize for any specific downstream task loss. MSE is the standard choice for block reconstruction because it treats all dimensions and positions equally and encourages the quantized block to be a drop-in replacement for the full-precision block with minimal perturbation.

Trainable parameters for PrefixQuant-O1 (dynamic activation quantization). In dynamic quantization, the quantization parameters $s$ and $z$ are computed online from the activation values during the forward pass; they are functions of the data, not learnable parameters. To make dynamic quantization trainable, PrefixQuant-O1 introduces tensor-wise trainable clipping factors $\gamma, \beta$ for activation quantization. Specifically:

For each activation quantizer in each block, $\gamma$ and $\beta$ are initialized from the grid search and then treated as trainable parameters.
These clipping factors are tensor-wise (shared across all tokens in the tensor), not token-wise. The paper explicitly states: "the clipping factors cannot be token-wise, as long-context scenarios introduce excessive storage overhead with token-wise clipping factors."
During training, a forward pass through the quantized block uses the current $\gamma, \beta$ to compute $s = (\gamma \max(X) - \beta \min(X)) / (2^N - 1)$ for each token, then quantizes, then dequantizes. The loss backpropagates through the quantization operation (using straight-through estimator for the rounding function) to update $\gamma$ and $\beta$ .

Additionally, all full-precision weights and weight quantization parameters (step sizes for weight quantizers) are trainable, following the EfficientQAT approach.

Trainable parameters for PrefixQuant-O2 (static activation quantization). In static quantization, $s$ and $z$ are precomputed offline and do not change during inference. This makes them naturally trainable: they are initialized from the grid search and then updated via gradient descent during block-wise fine-tuning. The forward pass uses these fixed $s, z$ values (one per activation tensor) to quantize and dequantize all tokens uniformly. All full-precision weights and weight quantization parameters are also trainable.

Training configuration. The paper specifies the following hyperparameters for block-wise fine-tuning (Section 5.1):

Training data: 512 samples from the Pile (Gao et al., 2020) with 1024-token sequence length, giving 512 × 1024 = 524,288 total training tokens per block. Table 7c shows that this token count achieves satisfactory performance, with further increases (1024 × 1024 tokens) providing only marginal improvement (WikiText2 perplexity 7.41 vs. 7.42).
Sequence length: 1024 tokens per sample. Table 7b shows this is optimal: 256 samples × 2048 tokens yields worse perplexity (7.65), as does 1024 samples × 512 tokens (7.65).
Dataset choice: Pile outperforms C4 (7.42 vs. 7.60 perplexity) and RedPajama (7.42 vs. 7.49), as shown in Table 7a.
Learning rates: Quantization parameters (step sizes for weights, clipping factors or static $s, z$ for activations): $5 \times 10^{-5}$ . Full-precision weights: $5 \times 10^{-6}$ . For Llama-3-70B and Llama-3-70B-Instruct (larger models), reduced to $2 \times 10^{-5}$ and $2 \times 10^{-6}$ respectively.
Batch size: 4 samples.
Epochs: 10 for W4A8KV4, 20 for W4A4KV4. Table 8 shows convergence: W4A8KV4 perplexity plateaus at 6.63 after 10 epochs (vs. 6.90 without fine-tuning); W4A4KV4 plateaus at 7.42 after 20 epochs (vs. 7.93 without fine-tuning). Training for 30 epochs provides negligible additional improvement (6.63 for W4A8, 7.41 for W4A4).
Optimizer: Not explicitly specified in the main text, but the practice of block-wise MSE minimization typically uses Adam or AdamW.

Sequential block training. Blocks are trained one at a time, from first to last. For block $\ell$ , the input comes from the output of the previously trained quantized block $\ell-1$ (or from the full-precision embedding for block 1). The target output is the full-precision block's output given the same input. This sequential approach accounts for error accumulation: the quantized block $\ell$ is trained to be robust to the (slightly perturbed) inputs it will actually receive from the quantized block $\ell-1$ during inference, rather than being trained on clean full-precision inputs that it would never see in deployment.

Why block-wise rather than end-to-end training. Block-wise training is orders of magnitude more memory-efficient: you only need to keep one block's parameters, activations, and gradients in GPU memory at a time. For a 70B model, end-to-end training of all blocks simultaneously would be infeasible on a single GPU. Additionally, block-wise training converges quickly because each block's optimization problem is relatively simple (match a single block's output given fixed inputs) compared to the highly non-convex end-to-end loss landscape.

The stabilization effect of prefixed tokens on training. Section 4.1 notes that outlier tokens "cause instability during block-wise fine-tuning." With outlier magnitudes of 1000+ in the activation tensors (Figure 3), the gradients flowing through the quantizer's rounding operation (via straight-through estimator) can be enormous, causing unstable updates to both $\gamma, \beta$ and the full-precision weights. By isolating outliers to the prefixed KV cache entries (which are never quantized and whose gradients don't flow through quantizers), PrefixQuant ensures that the training operates on activation tensors with values in a well-behaved range (ratios of ~2–5× as shown in Figure 3c), enabling stable convergence.

The Two Quantization Configurations: O1 and O2

The paper defines two operating modes for PrefixQuant (Table 1) that represent different points on the accuracy-efficiency tradeoff spectrum.

PrefixQuant-O1 (dynamic quantization). This configuration matches the quantization granularity of prior methods for fair comparison:

Weight quantization: Per-channel (each output channel has its own quantization step size).
Activation quantization: Per-token dynamic (each token in the sequence computes its own $s, z$ online based on its value range, using the trained clipping factors $\gamma, \beta$ ).
KV cache quantization: Per-head dynamic for keys, same as activation quantization.

O1 is what the paper compares against QuaRot, SpinQuant, DuQuant, and other dynamic quantization methods. The key claim is that PrefixQuant-O1 outperforms these methods even though it uses the same dynamic quantization granularity, because the prefixed tokens create a more uniform activation distribution that makes dynamic quantization more accurate.

PrefixQuant-O2 (static quantization). This is the more efficient configuration:

Weight quantization: Per-channel (unchanged from O1—weight quantization granularity doesn't affect inference latency as significantly because weights are loaded from memory and the quantization parameters are applied during the matrix multiply).
Activation quantization: Per-tensor static (a single $s, z$ for the entire activation tensor, precomputed offline and fixed during inference).
KV cache quantization: Per-head static (a single $s, z$ per attention head, precomputed offline).

The benefit of O2 over O1 is inference efficiency. For per-token dynamic quantization, the inference engine must:

Compute $\max(X)$ and $\min(X)$ for each token (reductions along the channel dimension).
Compute $s = (\gamma \max - \beta \min) / (2^N - 1)$ and $z$ via division and rounding.
Apply these per-token parameters to the quantization/dequantization operations.

For per-tensor static quantization, only a single $s, z$ per tensor is needed, precomputed offline. This eliminates the online reduction and division operations, enables better operator fusion (the quantization parameters can be baked into the matrix multiply kernels), and reduces the number of distinct quantization parameters that must be managed in memory. The paper demonstrates the practical impact in Table 6: PrefixQuant-O2 achieves 2.74× prefilling speedup over FP16 vs. 2.67× for O1, and 2.16× decoding speedup vs. 2.11× for O1—modest but consistent gains from the simplified quantization scheme.

The enabling mechanism. Static quantization is only viable because PrefixQuant eliminates token-wise outliers entirely from the quantized tensors. Without this, a single outlying token with 1000× the normal magnitude would force the static $s$ to be 1000× larger than necessary, destroying precision for all other tokens. With all tokens having roughly uniform per-token maximums (Figure 3c, right column), a single $s, z$ can serve the entire tensor with minimal accuracy loss. This is the paper's core practical contribution: showing that if you can eliminate token-wise outliers, static quantization becomes not just viable but competitive with or superior to dynamic quantization.

Summary of Key Design Choices and Their Justifications

Why prefix rather than prune or modify attention? Pruning outlier tokens would lose information they carry (they are typically delimiters, not meaningless noise). Modifying the attention mechanism (as in Bondarenko et al., 2024) requires retraining and may affect model capabilities. Prefixing preserves the model's ability to attend to these tokens (they still exist and are used by attention) but moves their quantization burden to a one-time offline cost. This is a classic systems tradeoff: pay a small fixed cost (precomputing KV entries for $o$ tokens, storing $o \times d_h \times \text{num_heads} \times \text{num_layers}$ FP16 values) to avoid a large per-inference cost (quantization error or dynamic parameter computation).

Why frequency-based rather than search-based token selection? The 12-second vs. 12-hour comparison (Table 6 vs. QFeP) makes this a pragmatic choice. But there's a deeper justification: if outlier tokens are caused by a structural property of the model (attention sinks) that is triggered by token position relative to the sequence start and by low-semantic content, then any token that occupies those early positions with low-semantic content will serve as an attention sink. The most frequently occurring such tokens (delimiters, common function words) are exactly what the model would naturally encounter in those positions anyway. The frequency heuristic therefore selects tokens that are already "compatible" with the model's expectations, without needing an expensive search.

Why [BOS] as a mandatory prefix token? Position 0 is universally an outlier token. The [BOS] is the token at position 0 for normal sequences. Including it in the prefix ensures the model's learned attention pattern at position 0 is preserved exactly—the precomputed KV for [BOS] at position 0 is what the model expects to see. Without this, the real input's first token would occupy position $o$ (after the prefix), and the model would need to redirect its position-0 attention sink to a different token, which could cause unexpected behavior.

Why separate trainable parameters for activation quantization under dynamic settings? In standard dynamic quantization, there are no trainable parameters— $s$ and $z$ are deterministic functions of the data. The introduction of trainable tensor-wise clipping factors $\gamma, \beta$ is a pragmatic innovation: it enables gradient-based optimization of the quantization range without introducing per-token parameters that would be storage-prohibitive for long sequences. The constraint that $\gamma, \beta$ are tensor-wise (not token-wise) ensures the same parameter count regardless of sequence length, which is essential for deployment.

Why 20 epochs for W4A4KV4 vs. 10 for W4A8KV4? Lower precision introduces more quantization error, which requires more optimization steps to compensate. The block-wise training is doing "error compensation"—adjusting weights and quantization parameters so that the quantized block's output approximates the full-precision output despite the coarser representation. With 4-bit weights and activations (W4A4), there are only 16 levels per value, so the optimization landscape is more constrained and requires more iterations to find a good local minimum.

Why Pile rather than WikiText2 for calibration and training? Table 2 notes that "grayed results use WikiText2 as calibration dataset" (referring to SpinQuant, which trains its rotation matrices on WikiText2). The paper explicitly avoids this for fair comparison: using WikiText2 for training and then evaluating WikiText2 perplexity would constitute training on the test set. By using Pile (a separate dataset with diverse web text), PrefixQuant's improvements reflect genuine generalization rather than overfitting to the evaluation distribution.

4. Key Insights and Innovations

Innovation 1: Token-Wise Outliers Are a Distinct Failure Mode That Cannot Be Solved by Channel-Wise Methods

The paper's most fundamental conceptual contribution is the clean diagnostic separation of token-wise outliers from channel-wise outliers as an independent failure mode for LLM quantization, with distinct causes, properties, and required solutions. This is not merely an incremental observation—it is a reframing of the problem space that the prior literature had implicitly conflated.

What the field assumed before. The dominant paradigm for activation quantization, exemplified by SmoothQuant (Xiao et al., 2023a), OmniQuant (Shao et al., 2023), and QuaRot (Ashkboos et al., 2024b), treated activation outliers as a channel-dimension phenomenon: certain feature dimensions consistently produce large magnitudes across all tokens. The solution space reflected this assumption—channel-wise scaling redistributes magnitude from activations to weights along the channel axis; Hadamard rotation uniformly mixes channel values so that no single channel dominates; mixed-precision approaches keep outlier channels in FP16 while quantizing normal channels. The implicit assumption was that handling channel-wise outliers was sufficient for accurate quantization, because once each token's channel distribution is uniform, per-token dynamic quantization (which computes quantization parameters independently per token) can handle any remaining token-to-token variation.

What this paper shows is different. Section 4.1 demonstrates that token-wise outliers constitute a separate axis of non-uniformity that channel-wise methods do not and cannot address. The evidence is Figure 3, which visualizes per-token maximum values rather than per-channel statistics: even after optimal Hadamard rotation (Figure 3(I.b)), the largest token-wise maximum in Llama-2-7B's down_proj inputs is 461 times the median token's maximum. This ratio—which measures variation across the token dimension—is fundamentally unaffected by rotation because rotation operates on channels within each token independently. A Hadamard matrix can make the 4,096 channels of token 5 each have roughly equal magnitude, but it cannot reduce the fact that token 5's channels collectively have 461× the magnitude of token 100's channels.

The diagnostic metric itself—the ratio of per-token maximums to their median (Equation 3)—is an innovation in framing. Prior work analyzed activation distributions by looking at per-channel statistics (mean, max, variance across tokens) or at the overall tensor histogram. The per-token-maximum-to-median ratio directly measures what matters for quantization: whether a single scaling factor can serve all tokens. A ratio near 1 means yes; a ratio of 461 means no.

Why this matters beyond the specific method. This diagnostic separation implies a design principle for future quantization research: token-wise and channel-wise outliers require different solution mechanisms, and any complete quantization method must address both. The failure of channel-wise methods on token-wise outliers is not a limitation that can be overcome by better rotation matrices or more sophisticated scaling—it is a fundamental mismatch between the operation (which acts on the channel axis) and the problem (which lives on the token axis). This explains why SpinQuant (Liu et al., 2024b), which trains optimal rotation matrices, still leaves token-wise outliers intact and underperforms PrefixQuant despite using identical quantization granularity (Table 2).

The lower outlier discovery as a specific diagnostic advance. The identification of lower outlier tokens in Q and K projections (Figure 3(III)) is a genuinely novel empirical finding with significant implications. Prior work on "massive activations" (Sun et al., 2024) and attention sinks (Xiao et al., 2023b) focused exclusively on large-magnitude outliers. The paper shows that Q and K exhibit the opposite problem—tokens with abnormally small values that are equally destructive to quantization because the step size must span the full range from minimum to maximum. The fact that Hadamard rotation does not mitigate lower outliers (Figure 3(III.b) shows the ratio unchanged) is a particularly crisp demonstration that channel-wise methods are structurally incapable of addressing token-dimension problems: rotation preserves per-token L2 norm, so if a token has near-zero values in all channels, rotation produces near-zero values in different linear combinations but still near-zero overall.

The evidence anchoring this innovation is the systematic visualization across all model families in Appendix H (Figures 9–20), showing that token-wise outliers and their resistance to rotation are universal across Llama-2, Llama-3, Mistral, and Qwen architectures. This is not a quirk of one model—it is a structural property of transformer attention mechanisms.

Innovation 2: Outlier Quarantine Through KV Cache Prefixing Is a Training-Free Structural Solution

The paper's central methodological insight is that token-wise outliers can be eliminated not by suppressing them, modifying the model architecture, or redistributing their energy, but by strategically relocating them to predictable positions where their activations can be precomputed offline and isolated from the quantized forward pass. This is a fundamentally different strategy from all prior approaches to activation outliers.

What the field did before. Prior approaches to token-wise outliers fell into three categories, all of which have significant drawbacks:

Architectural modification (Bondarenko et al., 2024; Sun et al., 2024): Change the model's attention mechanism—add gating, modify SoftMax, introduce attention bias—to prevent outliers from forming in the first place. This requires model retraining and changes the model's behavior, potentially affecting capabilities on tasks that depend on the outlier-producing attention patterns.
Search-based prefixing (QFeP by Yang et al., 2024; CushionCache by Son et al., 2024): Try many candidate token combinations as KV cache prefixes, evaluate quantization error for each, and select the best. This is computationally expensive (12 hours for Llama-3-8B per the paper's timing) and conceptually unsatisfying—it treats prefix selection as a black-box hyperparameter optimization rather than understanding why certain tokens work as prefixes.
Per-token dynamic quantization (all prior weight-activation quantization methods in Table 1): Accept that token-wise variation exists and handle it by computing quantization parameters online per token. This works but prevents the efficiency gains of static quantization.

What makes the quarantine strategy distinctive. The paper's insight is that token-wise outliers are not random or unpredictable—they are caused by a structural property of the attention mechanism (the SoftMax forcing non-zero attention on certain "sink" tokens, typically initial tokens and low-semantic delimiters near the sequence start) and therefore have predictable properties: they occur at a small, consistent number of positions; those positions cluster near the sequence start; and the token content at those positions is consistently low-semantic delimiters and function words. Rather than fighting these properties (trying to prevent the attention mechanism from creating outliers), PrefixQuant exploits them: if outliers consistently occur at early positions on delimiter tokens, then prepending delimiter tokens to every sequence will cause the model to direct its outlier-producing attention patterns toward those prepended positions instead of the input-dependent tokens.

This is conceptually analogous to quarantine in epidemiology: you don't try to prevent the disease from existing; you identify carriers and isolate them so they can't infect the rest of the population. The prefixed tokens are the quarantine zone—their activations can be as extreme as the model wants, but those extremes are precomputed once offline and never enter the quantized computation graph.

Why the frequency-based selection is a key insight, not just an engineering shortcut. The paper could have framed the 12-second detection time merely as an efficiency advantage over grid search. But there is a deeper logic: if outlier tokens are caused by the model's structural bias toward treating certain positions as attention sinks, then any token that frequently occupies those positions will serve as an attention sink. The tokens that most frequently appear as outliers (delimiters like "\n", ".", ","; function words like "the", "a"; spaces) are exactly the tokens that the model most frequently encounters in early sequence positions during training. The frequency heuristic therefore selects tokens that are already compatible with the model's learned attention patterns—they are the tokens the model naturally expects to see as sinks. An expensive grid search over token combinations would likely converge to the same set, because those are the tokens the model is trained to treat as sinks.

This is partially validated by the ablation in Table 10: using only the single highest-frequency token (".") without the additional high-frequency tokens significantly underperforms the adaptive selection (WikiText2 perplexity 12.07 vs. 6.22 for Llama-2-7B without fine-tuning), and random token selection performs terribly (66.51). The frequency ordering captures a genuine signal about which tokens the model treats as attention sinks.

The significance of training-free operation. The paper emphasizes that outlier detection is "training-free and highly efficient" multiple times (abstract, Section 2, Section 4.1). This is not just about computational cost—it reflects a philosophical stance about what should require training. Prior work that modified SoftMax or added attention bias implicitly assumed that eliminating outliers requires changing the model's behavior. PrefixQuant demonstrates that you can leave the model's behavior completely unchanged—the model still produces exactly the same attention patterns and activation values it would have produced—and achieve outlier elimination purely through a structural intervention (changing which tokens are in the KV cache, not how attention operates). This is a stronger result: it shows that the model's outlier-producing behavior is not a bug to be trained away but a predictable consequence of the attention mechanism that can be redirected through input manipulation alone.

The anchoring evidence is Figure 4d, which shows that after prefixing the top-o high-frequency outlier tokens, outlier token positions are constrained to positions 0 and 1 (the prefixed positions) in 100% of cases, down from a distribution spanning positions 0–40+ without prefixing (Figure 4c). This is a binary outcome—the quarantine works perfectly—not a probabilistic improvement.

Innovation 3: Static Quantization Can Surpass Dynamic Quantization When Token-Wise Outliers Are Eliminated

The paper makes a counterintuitive empirical claim with significant practical implications: that per-tensor static quantization—long considered too coarse for LLM activations—can outperform per-token dynamic quantization, but only when token-wise outliers are first eliminated. This inverts the conventional wisdom that dynamic quantization is strictly more accurate and that static quantization represents an accuracy-efficiency tradeoff.

What the field assumed before. The prevailing view, encoded in the design choices of every prior weight-activation quantization method in Table 1 (SmoothQuant, Atom, QoQ, QuaRot, SpinQuant), was that per-token dynamic quantization was necessary for acceptable accuracy. The reasoning was straightforward: activation distributions vary significantly across tokens due to the attention mechanism, and a single static scaling factor cannot adapt to this variation. Static quantization was used only in weight-only quantization scenarios (where the weights are fixed across inputs and can be calibrated offline) or in much higher precision regimes (W8A8).

What this paper proves. Table 2, Table 3, and Table 4 consistently show PrefixQuant-O2 (static per-tensor activation quantization) matching or exceeding PrefixQuant-O1 (dynamic per-token) and significantly outperforming prior dynamic methods. For example, on W4A4KV4 Llama-3-8B, PrefixQuant-O2 achieves 66.37% average zero-shot accuracy vs. 66.74% for PrefixQuant-O1 and 65.35% for SpinQuant (dynamic). On W4A8KV4, PrefixQuant-O2 achieves 72.46% vs. 71.68% for QuaRot (dynamic). On MMLU, O2 achieves 54.65% vs. 51.93% for SpinQuant (dynamic) at W4A4KV4.

Why this finding is non-obvious. Static quantization uses a single step size and zero-point for the entire activation tensor, while dynamic quantization computes these per token. On any distribution with token-to-token variation, dynamic quantization should be at least as accurate—it has strictly more degrees of freedom to adapt to the data. The fact that static quantization outperforms dynamic in several head-to-head comparisons (e.g., Llama-2-13B W4A4KV4: O2 at 70.36% vs. O1 at 70.05%; Llama-3-70B W4A8KV4: O2 at 78.70% vs. O1 at 78.50%) suggests that something is going wrong with dynamic quantization that static avoids.

The paper does not fully explain the mechanism, but the likely explanation is that dynamic quantization parameters are computed from noisy, instantaneously observed activation ranges and are therefore subject to estimation error. When a token's activation range must be computed on-the-fly from a single sample of that token's values, outliers within the token (channel-wise outliers) can still inflate the range estimate even if token-wise outliers have been eliminated. Static quantization parameters, by contrast, are computed offline from many calibration samples, producing a more robust estimate of the typical activation range. The prefixing step eliminates the worst token-wise variation, making the "typical range" genuinely representative of all tokens, which in turn makes the robust static estimate better than the noisy per-token dynamic estimate.

The practical significance. This finding unlocks efficiency gains that were previously considered unavailable at competitive accuracy. Table 6 shows that static quantization (O2) enables modest but real speedups over dynamic (O1): 2.74× vs. 2.67× prefilling, 2.16× vs. 2.11× decoding. More importantly, static quantization enables operator fusion optimizations (not benchmarked in the paper but noted in Section 3 and the quantization literature) that can compound these gains in production deployments. The implication is that the entire field's default choice of dynamic quantization for weight-activation quantization may have been driven primarily by the unaddressed token-wise outlier problem, and solving that problem opens a previously closed efficiency frontier.

The key ablation anchoring this insight is Table 5, row "PrefixQuant-O2 + static quantization": without prefixed outliers, static quantization yields catastrophically bad perplexity (141.02 for W4A4KV4). With prefixed outliers alone (no fine-tuning), it drops to 7.93—already competitive with dynamic quantization baselines. This demonstrates that token-wise outliers are the bottleneck that prevented static quantization from working, not some inherent limitation of static quantization itself.

Innovation 4: The Attention Sink Mechanism Is Exploitable for Quantization Benefit Without Architectural Modification

The paper implicitly makes a conceptual connection that has broader implications for LLM efficiency research: the attention sink phenomenon, previously studied as a property to preserve for long-context generation (StreamingLLM, Xiao et al., 2023b) or as a quirk to suppress (Bondarenko et al., 2024), can be actively exploited as a mechanism for improving quantization by providing the model with strategically placed sink tokens whose activations are handled outside the quantized computation graph.

What prior work established about attention sinks. Xiao et al. (2023b) discovered that LLMs consistently allocate large attention weights to initial tokens, and that preserving these initial tokens in the KV cache is essential for stable long-context generation. Sun et al. (2024) connected this to "massive activations"—the initial tokens produce extreme activation values. The interpretation in both works was that attention sinks are a property to manage: StreamingLLM retains initial tokens to prevent collapse; Bondarenko et al. train models to avoid forming attention sinks. Neither framed attention sinks as something that could be weaponized for efficiency.

What PrefixQuant reveals. The paper demonstrates that attention sinks are not just a property of initial tokens—they can be induced at arbitrary token positions by prefixing tokens that the model naturally treats as sinks (delimiters, function words). Figure 4d provides the direct evidence: after prefixing "." and "\n" before the input, the outlier tokens (attention sinks) move from variable positions (Figure 4c) to fixed positions 0 and 1 (the prefixed positions). The model's attention mechanism, which evolved during pretraining to treat certain token types as sinks based on their content and early-sequence position, can be redirected to treat prefixed tokens of those same types as sinks instead.

This is a conceptual advance because it reframes the relationship between model architecture and efficiency. The dominant paradigm in efficient ML is that improving efficiency requires either (a) changing the model (pruning, distillation, architecture modification) or (b) changing the computation (quantization, sparsity). PrefixQuant demonstrates a third category: rearranging the input to the model so that the model's existing behavior becomes efficiency-friendly. The model is not changed; the computations are the same; but by controlling where the model places its extreme activations (on precomputed, full-precision KV entries vs. on quantized input tokens), we can achieve quantization accuracy that would otherwise require model modification.

Why this matters beyond this paper. This "exploit rather than suppress" approach to attention sinks could generalize to other efficiency challenges. If LLMs have predictable structural biases in where they allocate computation (attention sinks, sparse activation patterns, predictable token importance), then efficiency methods could be designed to work with these biases—providing high-precision pathways for the high-importance computations and aggressive compression for the rest—rather than fighting against them. The paper's frequency-based token selection is a first example: identify what the model wants to treat as important (delimiter tokens as attention sinks) and give it those tokens in a controlled way.

The connection to StreamingLLM is particularly instructive. StreamingLLM showed that retaining initial tokens in the KV cache is sufficient for long-context stability. PrefixQuant shows that you can replace the input's initial tokens with different initial tokens (the prefixed tokens) and the model still functions correctly—the attention sink mechanism cares about positional primacy and token type, not about the specific content of initial tokens. This suggests a degree of flexibility in attention sink formation that could be exploited for other purposes, such as controlled context manipulation or efficient multi-turn conversation by reusing attention sink KV entries across turns.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses WikiText2 (Merity et al., 2016) for perplexity measurement with a 2048-token context length, and five zero-shot common-sense reasoning tasks—PIQA (Bisk et al., 2020), ARC Easy and Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), and WinoGrande (Sakaguchi et al., 2021)—for accuracy evaluation, all measured through lm-eval v0.4.2 (Gao et al., 2024). The calibration and fine-tuning dataset is the Pile (Gao et al., 2020), specifically 8 samples of 1024-token sequences for grid search initialization and 512 samples of 1024-token sequences for block-wise fine-tuning. Additional evaluation on zero-shot MMLU (Hendrycks et al., 2020) is reported for Llama-3-8B as a more challenging benchmark.

Base model(s). The paper evaluates on seven model families spanning multiple scales: Llama-2 (7B, 13B, 70B), Llama-3 (8B, 70B), Llama-3-Instruct (8B, 70B), Mistral-7B-v0.3, and Qwen-2-7B. This range is deliberately broad—it covers both base and instruction-tuned variants, parameter counts from 7B to 70B, and architectures from multiple organizations, testing whether token-wise outliers and PrefixQuant's solution generalize across model families. The selection of Llama-3-8B as the primary ablation model (Table 5, Tables 7–8) is appropriate because it represents the most challenging case: Figure 13 (Appendix H) shows it has extreme down_proj outliers (max(top-1/median) = 2019 without rotation) despite having only initial-token outliers, making it a test of whether prefixing works even when only the [BOS] token needs to be prefixed.

Metrics. WikiText2 perplexity (PPL) is the primary quantitative metric, measured with 2048-token context length—lower is better, with the full-precision FP16 model providing the ceiling. Average zero-shot accuracy on the five reasoning tasks is reported as "Acc." in Tables 2–3: WinoGrande uses acc, while HellaSwag, Arc Challenge, Arc Easy, and PIQA use acc_norm (length-normalized accuracy) following QServe (Lin et al., 2024b). The full breakdown per-task is provided in Table 14 (Appendix G.2). MMLU accuracy is reported separately in Table 4.

Baselines. The paper compares against five prior weight-activation quantization methods:

QuaRot (Ashkboos et al., 2024b): the Hadamard rotation-based method that PrefixQuant builds upon, using per-channel weight, per-token dynamic activation, and per-token dynamic KV cache quantization.
SpinQuant (Liu et al., 2024b): extends QuaRot with learned rotation matrices, using the same quantization granularity. Notably, SpinQuant uses WikiText2 as its calibration dataset (marked in gray in Table 2), which the paper flags as potential overfitting since evaluation includes WikiText2 perplexity.
DuQuant (Lin et al., 2024a): uses dual transformations (rotation + permutation) to handle outliers, with per-channel weight and per-token dynamic activation quantization.
Atom (Zhao et al., 2023): mixed-precision approach with group-wise quantization for both weights and activations.
QoQ (Lin et al., 2024b): W4A8KV4-specific method with per-channel weight and per-token dynamic activation quantization, also using the Pile calibration dataset.

Baselines are evaluated under consistent settings: all reproduced methods (except SpinQuant) use the Pile calibration dataset "to avoid over-fitting for fair comparisons" (Section 5.1). Results sources are detailed in Appendix A: QuaRot W4A4KV4 Llama-2 results come from QoQ's paper; all other QuaRot and DuQuant results are reproduced from official code; Atom results come from QoQ; QoQ Llama-2 results come from the QoQ paper with Llama-3 results reproduced; SmoothQuant results are reproduced from QoQ's codebase; SpinQuant uses pre-trained rotation matrices with official code; EfficientQAT uses official code and pre-quantized models.

Generation budget / compute accounting. Unlike the reference paper which measures test-time compute in "generations" (number of sampled solutions), PrefixQuant operates in a post-training quantization setting where compute accounting is about offline processing time. The paper reports three timing metrics in Table 6 (Appendix C): time to find prefixed outliers (12 seconds for Llama-2-7B, 1 minute for Llama-3-70B), time for grid-search initialization (0.7 minutes for Llama-3-8B, 12 minutes for Llama-3-70B), and time for fine-tuning (2.2 hours for Llama-3-8B with 20 W4A4KV4 epochs, 17 hours for Llama-3-70B). Inference speedup is measured as end-to-end latency on a single RTX 3090 GPU: prefilling speedup for processing 2048 input tokens, and decoding speedup measured as tokens per second when generating 256 new tokens with 2048-token context.

Cross-validation / statistical protocol. There is no cross-validation in this paper—the evaluation is standard train/test split where calibration/fine-tuning data (Pile) is entirely separate from evaluation data (WikiText2, zero-shot tasks, MMLU). The paper explicitly addresses the risk of training-evaluation overlap by noting that SpinQuant's use of WikiText2 for calibration (marked in gray in Table 2) creates an unfair advantage on perplexity metrics, and consistently uses Pile instead for their own method. Table 7a ablates the fine-tuning dataset choice (C4 vs. RedPajama vs. Pile), confirming that Pile generalizes best to WikiText2 evaluation, which partially addresses concerns about distribution matching between training and evaluation corpora.

Main Quantitative Results

The paper organizes results across two precision configurations (W4A4KV4 and W4A8KV4), two PrefixQuant configurations (O1 dynamic and O2 static), and multiple model families. The central narrative is: PrefixQuant-O1 matches or exceeds prior dynamic quantization methods while using identical quantization granularity; PrefixQuant-O2 achieves comparable or better performance using coarser static quantization; and these gains are consistent across model families and scales.

W4A4KV4: PrefixQuant Outperforms All Dynamic Baselines Under Identical Settings

Table 2 presents the headline W4A4KV4 results across Llama-2 (7B, 13B, 70B) and Llama-3 (8B, 70B), measuring WikiText2 perplexity and average zero-shot accuracy.

On Llama-2-7B, PrefixQuant-O1 achieves 5.93 PPL and 66.74% accuracy, compared to SpinQuant's 5.95 PPL and 65.35% accuracy—a 0.02 PPL reduction and +1.39 accuracy improvement. The more striking comparison is against DuQuant (6.20 PPL, 66.25% accuracy): PrefixQuant-O1 improves PPL by 0.27 and accuracy by +0.49 points. PrefixQuant-O2 (static quantization) achieves 6.01 PPL and 66.37% accuracy, still outperforming DuQuant despite using coarser quantization granularity. Against QuaRot (the base rotation method), PrefixQuant-O1 improves accuracy by +2.05 points (66.74% vs. 64.69%).

On Llama-2-13B, PrefixQuant-O1 achieves 5.24 PPL and 70.05% accuracy, matching SpinQuant's 5.24 PPL but exceeding its accuracy by +0.81 points (69.24% vs. 70.05%). PrefixQuant-O2 achieves 5.32 PPL and 70.36% accuracy—interestingly, O2 exceeds O1 in accuracy by +0.31 points despite using static quantization. This is the first indication of the paper's claim that static quantization can outperform dynamic when token-wise outliers are eliminated.

On Llama-2-70B, PrefixQuant-O1 achieves 3.62 PPL and 76.23% accuracy—the best PPL among all methods (SpinQuant: 3.70, DuQuant: 3.77, QuaRot: 3.83) and the best accuracy (SpinQuant: 75.19%, DuQuant: 74.75%, QuaRot: 75.43%). The improvement is +1.04 accuracy points over SpinQuant. PrefixQuant-O2 at 3.81 PPL and 75.48% accuracy is competitive with SpinQuant (3.70 PPL, 75.19% accuracy) despite static quantization.

On Llama-3-8B, the gains are most pronounced. PrefixQuant-O1 achieves 7.26 PPL and 71.31% accuracy, compared to SpinQuant's 7.36 PPL and 68.23% accuracy—a +3.08 accuracy point improvement. Against DuQuant (8.14 PPL, 67.13%), the improvement is +4.18 accuracy points. PrefixQuant-O2 achieves 7.43 PPL and 71.08% accuracy, improving +2.85 points over SpinQuant despite static quantization. The abstract's headline claim of "+3.08 and +2.85 points over SpinQuant on five zero-shot reasoning tasks under dynamic and static quantization settings" maps directly to these numbers. QuaRot performs poorly on this model (8.41 PPL, 65.15%), making PrefixQuant's improvement +6.16 accuracy points.

On Llama-3-70B, PrefixQuant-O1 achieves 4.16 PPL and 77.08% accuracy, dramatically outperforming QuaRot (6.82 PPL, 68.39%) and DuQuant (5.67 PPL, 74.89%). SpinQuant W4A4KV4 results are not available for this model in the main table, though Table 15 mentions a 74.28% accuracy result that is not directly comparable due to different evaluation configurations. The PPL improvement over DuQuant is 1.51 points—the largest absolute gain in the table.

A pattern across model scales and families. Across all five models where SpinQuant results are available (Llama-2-7B/13B/70B, Llama-3-8B), PrefixQuant-O1 strictly dominates SpinQuant in zero-shot accuracy, with margins ranging from +0.81 (Llama-2-13B) to +3.08 (Llama-3-8B) points. The pattern extends to PPL: PrefixQuant-O1 achieves lower or equal PPL in all comparisons. This consistency across model families (Llama-2 and Llama-3 use different architectures and training data) provides evidence that the token-wise outlier problem and PrefixQuant's solution are not model-specific.

The significance of SpinQuant as the primary comparison target. SpinQuant represents the state of the art in rotation-based quantization—it learns optimal rotation matrices rather than using random Hadamard transforms. The fact that PrefixQuant consistently outperforms SpinQuant despite using exactly the same quantization granularity (both use per-channel weight and per-token dynamic activation quantization per Table 1) isolates the contribution of the prefixed token mechanism: the rotation quality is not the bottleneck; the residual token-wise outliers that rotation cannot eliminate are.

W4A8KV4: PrefixQuant Extends to Higher Precision with Consistent Gains

Table 3 presents W4A8KV4 results (slightly higher precision for activations, reducing quantization error from 4-bit activation quantization). The baselines shift: QoQ replaces Atom and DuQuant as the primary comparison at this precision.

On Llama-2-7B, PrefixQuant-O1 achieves 5.67 PPL and 68.04% accuracy vs. QoQ's 5.75 PPL and 67.22% (+0.82 accuracy improvement) and QuaRot's 5.73 PPL and 67.11% (+0.93 accuracy improvement). The margins are smaller than at W4A4KV4 because W4A8 has inherently less quantization error to recover.

On Llama-2-13B, PrefixQuant-O1 achieves 5.05 PPL and 71.25% accuracy vs. QoQ's 5.12 PPL and 70.56% (+0.69 accuracy improvement). Interestingly, PrefixQuant-O2 matches O1 exactly at 71.25% accuracy with 5.07 PPL, demonstrating that static quantization can rival dynamic even at the higher precision.

On Llama-2-70B, PrefixQuant-O1 achieves 3.44 PPL and 76.82% accuracy, compared to QuaRot's 3.46 PPL and 76.31% (+0.51 accuracy improvement) and QoQ's 3.52 PPL and 75.91% (+0.91 accuracy improvement). The differentce between O1 and O2 is small (O2: 3.50 PPL, 76.50%), consistent with the pattern that static quantization becomes more competitive as the overall quantization error floor drops at higher precision.

On Llama-3-8B, PrefixQuant-O1 achieves 6.59 PPL and 72.57% accuracy vs. QuaRot's 6.80 PPL and 71.68% (+0.89 accuracy) and QoQ's 6.89 PPL and 71.35% (+1.22 accuracy). PrefixQuant-O2 at 6.62 PPL and 72.46% accuracy again approaches O1 performance with static quantization.

On Llama-3-70B, the pattern inverts: PrefixQuant-O1 achieves 3.37 PPL and 78.50% accuracy, while PrefixQuant-O2 achieves 3.43 PPL and 78.70% accuracy—O2 exceeds O1 in accuracy by +0.20 points, and QuaRot achieves 3.73 PPL and 78.92% accuracy, slightly exceeding both O1 and O2. This is the only case where a baseline outperforms PrefixQuant in the main tables, and only at W4A8KV4 (the easier quantization regime where differences between methods compress). The O2-over-O1 inversion is qualitatively important: it suggests that at very high precision, the noise in per-token dynamic quantization's online parameter estimation may outweigh its adaptivity benefit.

MMLU Results: The Gains Extend to Knowledge-Intensive Benchmarks

Table 4 provides MMLU evaluation on Llama-3-8B, testing whether the quantization accuracy improvements measured on common-sense reasoning transfer to a more challenging benchmark requiring factual knowledge across 57 subjects.

At W4A4KV4, PrefixQuant-O1 achieves 56.00% MMLU accuracy vs. SpinQuant's 51.93% (+4.07 points) and DuQuant's 50.77% (+5.23 points). QuaRot collapses to 34.25%. PrefixQuant-O2 achieves 54.65%, a +2.72 point improvement over SpinQuant despite static quantization. The FP16 baseline is 62.07%, meaning PrefixQuant-O1 retains 90.2% of full-precision MMLU accuracy at 4-bit precision.

At W4A8KV4, PrefixQuant-O1 achieves 60.49% vs. SpinQuant's 58.25% (+2.24 points) and DuQuant's 58.01% (+2.48 points). PrefixQuant-O2 achieves 59.20%. The FP16 ceiling is 62.07%, so O1 retains 97.5% of full-precision accuracy—approaching lossless compression at this precision for this benchmark.

The MMLU results are important because they address a potential criticism of the zero-shot reasoning benchmarks: PIQA, HellaSwag, ARC, and WinoGrande primarily test surface-level linguistic patterns and commonsense associations, which may be robust to quantization noise. MMLU requires factual recall and multi-domain reasoning, making it a sterner test of whether quantization preserves the model's knowledge. The large margins over SpinQuant (+4.07 at W4A4, +2.24 at W4A8) suggest that token-wise outliers corrupt factual knowledge more severely than simple reasoning, possibly because factual recall depends on precise value propagation through attention, which outlier-dominated quantization disrupts.

Additional Model Families and Instruction-Tuned Models

Table 15 (Appendix G.3) extends results to Mistral-7B-v0.3, Qwen-2-7B, and instruction-tuned Llama-3 variants.

On Mistral-v0.3-7B, W4A4KV4 PrefixQuant-O1 achieves 5.76 PPL and 71.48% accuracy; O2 achieves 5.79 PPL and 71.45%. No baselines are provided for this model configuration, making these results useful for breadth but not for comparative strength assessment.

On Qwen-2-7B, W4A4KV4 PrefixQuant-O1 achieves 7.76 PPL and 70.38% accuracy; O2 achieves 8.37 PPL and 69.22%. The O2 degradation (0.61 PPL, -1.16 accuracy vs. O1) is larger than for other model families, suggesting that static quantization may be more challenging for Qwen-2's activation distribution. Without baseline comparisons, it's unclear whether this is a PrefixQuant limitation or an inherent property of Qwen-2's architecture.

On Llama-3-8B-Instruct, W4A4KV4 O2 achieves 8.96 PPL and 69.97% accuracy; O1 results are missing from the table (possibly omitted due to error). W4A8KV4 O1 achieves 8.73 PPL and 71.00% accuracy; O2 achieves 8.74 PPL and 70.97%. The remarkably small gap between O1 and O2 (0.01 PPL, -0.03 accuracy) on instruction-tuned Llama-3-8B reinforces the pattern that static quantization becomes more viable when token-wise outliers are controlled.

On Llama-3-70B-Instruct, W4A4KV4 O1 achieves 6.49 PPL and 77.66% accuracy; O2 achieves 6.80 PPL and 77.29%. At W4A8KV4, O1 achieves 5.82 PPL and 78.49% accuracy; O2 achieves 5.96 PPL and 78.41%. The instruction-tuned 70B model shows a consistent O1-over-O2 gap that is larger than for the base 70B model, suggesting instruction fine-tuning may increase token-to-token activation variation in ways that static quantization handles less well.

Extended Results: Weight-Only Quantization and W8A8 Comparisons

Table 12 (Appendix F) demonstrates that PrefixQuant's prefixed token mechanism also improves weight-only quantization (where only weights are quantized and activations remain in FP16). Against EfficientQAT (Chen et al., 2024a) in block-wise fine-tuning scenarios:

W3A16g128 on Llama-3-8B: PrefixQuant achieves 7.17 PPL and 72.10% accuracy vs. EfficientQAT's 7.34 PPL and 70.80% accuracy (+1.30 accuracy points). The PPL improvement (0.17) is modest but consistent.

W2A16g128 on Llama-3-8B: PrefixQuant achieves 11.97 PPL and 64.00% accuracy vs. EfficientQAT's 13.55 PPL and 58.95% accuracy—a dramatic +5.05 accuracy point improvement and -1.58 PPL reduction. The gain is substantially larger at 2-bit than 3-bit, consistent with the hypothesis that outlier-induced instability during training (Section 4.1) disproportionately affects lower-precision regimes where the optimization landscape is more fragile.

On Llama-3-70B, W2A16g128 PrefixQuant achieves 11.01 PPL and 72.61% accuracy vs. EfficientQAT's 16.79 PPL and 67.88% accuracy—a +4.73 accuracy point improvement. The pattern from the 8B model replicates at 70B scale, providing strong evidence that the benefit of outlier isolation in the KV cache generalizes to weight-only quantization and across model scales.

Table 13 (Appendix G.1) compares PrefixQuant-O2 with QFeP (Yang et al., 2024) and CushionCache (Son et al., 2024)—the two closest prior works that also set prefixed tokens in the KV cache—at W8A8 precision. PrefixQuant-O2 achieves 5.48 PPL on Llama-2-7B vs. QFeP's 5.75 and CushionCache's 5.87; 4.89 PPL on Llama-2-13B vs. QFeP's 6.00; 3.39 PPL on Llama-2-70B vs. QFeP's 6.01; and 6.17 PPL on Llama-3-8B vs. CushionCache's 7.37. These comparisons are significant because QFeP and CushionCache already use prefixed tokens—PrefixQuant's advantage must come from better token selection (frequency-based vs. grid search) or from the complementary Hadamard rotation and block-wise fine-tuning, rather than from the prefixing concept itself.

Inference Speedup

Table 6 reports end-to-end inference latency for W4A4 quantized Llama-2-7B on an RTX 3090 GPU, comparing against an FP16 baseline. For prefilling (processing 2048 input tokens): FP16 takes 489ms; PrefixQuant-O1 achieves 2.67× speedup (183ms); PrefixQuant-O2 achieves 2.74× speedup (178ms). For decoding (generating 256 new tokens with 2048-token context): FP16 achieves 43 tokens/second; PrefixQuant-O1 achieves 2.11× speedup (91 tokens/second); PrefixQuant-O2 achieves 2.16× speedup (93 tokens/second).

The speedup gap between O1 and O2 is modest but consistent—O2 is roughly 2.6% faster at prefilling and 2.4% faster at decoding. This is expected because static quantization eliminates the online computation of per-token quantization parameters (max/min reductions, scaling factor computations), but these operations are a small fraction of total inference time compared to the matrix multiplications themselves. The more important implication is that O2 achieves these speedups while matching or exceeding O1's accuracy on most models—you get the efficiency gain essentially for free, rather than paying an accuracy penalty for faster static quantization.

An important limitation of the speedup results. The paper benchmarks on a single GPU (RTX 3090) with a single model (Llama-2-7B) at a single precision (W4A4). Speedup factors are highly hardware-dependent and implementation-dependent—different GPUs have different ratios of compute to memory bandwidth, and optimized quantization kernels can vary significantly. The claim of "up to 2.74× prefilling speedup and 2.16× decoding speedup" (abstract) should be understood as a proof-of-concept on consumer hardware, not a universal throughput guarantee. The paper does not benchmark on datacenter GPUs (A100, H100), does not report batch size scaling, and does not compare against optimized FP16 implementations (which might use FlashAttention or other kernel-level optimizations that change the baseline).

Ablation Studies and Robustness Checks

The paper's ablation strategy is organized as an incremental accumulation of techniques, presented in Table 5 as a single comprehensive ablation on Llama-3-8B across both W4A8KV4 and W4A4KV4. Each row adds one technique to the previous, quantifying its marginal contribution.

Base RTN quantization: Starting from round-to-nearest quantization with per-token dynamic activation quantization, W4A8KV4 achieves 12.66 PPL and W4A4KV4 catastrophically fails at 1282.34 PPL. This establishes the baseline: naive quantization is unusable at 4-bit.

+ Hadamard rotation: Adding rotation (the QuaRot framework) reduces W4A8KV4 to 10.88 PPL and W4A4KV4 to 24.98 PPL. The W4A4KV4 improvement from 1282.34 to 24.98 is enormous—a 51× reduction—confirming that channel-wise outliers are the dominant error source for naive quantization. However, 24.98 PPL is still far from the FP16 baseline of 6.14, indicating that channel-wise handling alone is insufficient.

+ Grid search initialization: Optimizing quantization parameters via MSE grid search further reduces W4A8KV4 to 9.08 PPL and W4A4KV4 to 11.70 PPL. The larger relative improvement at W4A4KV4 (24.98 → 11.70, a 53% reduction) vs. W4A8KV4 (10.88 → 9.08, a 16.5% reduction) is consistent with the hypothesis that grid search matters more at lower precision because the clipping-vs-rounding error tradeoff is more sensitive when there are fewer quantization levels.

PrefixQuant-O1: + prefixed outliers (without fine-tuning): Adding prefixed tokens to the dynamic quantization pipeline (but without block-wise training) reduces W4A8KV4 to 6.81 PPL and W4A4KV4 to 7.53 PPL. This is the critical ablation—it isolates the contribution of the prefixed token mechanism from the block-wise fine-tuning. At W4A4KV4, the improvement from 11.70 to 7.53 (-4.17 PPL) demonstrates that token-wise outliers are the dominant remaining error source after rotation and grid search. At W4A8KV4, the improvement from 9.08 to 6.81 (-2.27 PPL) is smaller but still substantial, reflecting that at higher precision, the dynamic quantization can more easily absorb token-to-token variation, making the outlier problem less acute.

PrefixQuant-O1: + block-wise fine-tuning: Training each block sequentially for 10 (W4A8) or 20 (W4A4) epochs reduces W4A8KV4 to 6.59 PPL and W4A4KV4 to 7.23 PPL. The marginal gain from fine-tuning (-0.22 PPL at W4A8, -0.30 PPL at W4A4) is modest compared to the gain from prefixed tokens, indicating that outlier elimination is the primary driver of PrefixQuant's performance, with fine-tuning providing a smaller refinement. This is consistent with the paper's emphasis on the training-free nature of the core contribution.

PrefixQuant-O2: + static quantization (without prefixed outliers): Moving from dynamic to static quantization without the outlier protection of prefixed tokens results in catastrophic degradation: W4A8KV4 PPL jumps to 29.07 (from 9.08 with dynamic, a 3.2× increase) and W4A4KV4 PPL explodes to 141.02 (from 11.70 with dynamic, a 12× increase). This confirms the paper's central motivation: static quantization is fundamentally incompatible with token-wise outliers because a single static scaling factor cannot simultaneously serve normal tokens and outlier tokens.

PrefixQuant-O2: + prefixed outliers (without fine-tuning): Adding prefixed tokens to the static quantization pipeline reduces W4A8KV4 from 29.07 to 6.90 PPL and W4A4KV4 from 141.02 to 7.93 PPL. The W4A4KV4 improvement in particular—from 141.02 to 7.93, a 17.8× reduction—is the single largest jump in the ablation table and directly demonstrates that token-wise outliers are the bottleneck that makes static quantization impossible, and eliminating them makes static quantization viable. The 7.93 PPL after prefixed tokens alone (no fine-tuning) is already better than the spinQuant W4A4KV4 result (7.36 PPL with fine-tuning and dynamic quantization) shown in Table 2 for Llama-3-8B. This is perhaps the paper's single most impressive number: static quantization with prefixed tokens alone (no training) rivals the previous state of the art with dynamic quantization and full training.

PrefixQuant-O2: + block-wise fine-tuning: The final fine-tuning step brings static quantization to 6.62 PPL (W4A8) and 7.41 PPL (W4A4). The marginal gain from fine-tuning for O2 (-0.28 PPL at W4A8, -0.52 PPL at W4A4) is larger than for O1, suggesting that static quantization benefits more from learned parameter adjustment because it has fewer degrees of freedom per-token and must be more precisely calibrated.

Fine-tuning dataset choice (Table 7a): Ablating over C4, RedPajama, and Pile as the fine-tuning corpus on Llama-3-8B W4A4KV4 O2, Pile achieves 7.42 PPL vs. 7.49 for RedPajama and 7.60 for C4. The differences are modest (0.18 PPL between best and worst), indicating that block-wise fine-tuning is not highly sensitive to the specific corpus, as long as it is diverse enough to cover the activation distribution.

Training sequence length (Table 7b): Comparing 256 samples × 2048 tokens, 512 × 1024, and 1024 × 512 on Llama-3-8B W4A4KV4 O2, the optimal configuration is 512 × 1024 (7.42 PPL), with both longer sequences (256 × 2048: 7.65) and shorter sequences (1024 × 512: 7.65) underperforming. This suggests a Goldilocks effect: sequences need to be long enough to include the prefixed tokens and their interaction with subsequent tokens, but not so long that the block-wise reconstruction loss becomes dominated by less informative distant-token predictions.

Total training tokens (Table 7c): Comparing 256 × 1024, 512 × 1024, and 1024 × 1024 samples on Llama-3-8B W4A4KV4 O2, performance saturates at 512 × 1024 (7.42 PPL), with 1024 × 1024 providing minimal additional benefit (7.41 PPL). The paper notes that "lower precision requires more training data" and cites EfficientQAT's use of 4096 × 2048 for W2A16 as evidence of this trend, positioning 512 × 1024 as appropriate for W4A4.

Training epochs (Table 8): At W4A8KV4 O2, perplexity drops from 6.90 (epoch 0) to 6.66 (epoch 5) to 6.63 (epoch 10) and plateaus (6.63 at epochs 20–30). At W4A4KV4 O2, perplexity drops from 7.93 (epoch 0) to 7.53 (epoch 5) to 7.47 (epoch 10) to 7.42 (epoch 20) to 7.41 (epoch 30). The slower convergence at lower precision (20 epochs vs. 10 to plateau) confirms that W4A4KV4 is a harder optimization problem and validates the separate epoch budgets used in the main experiments.

Number of prefixed tokens (Table 9): This ablation tests Llama-2-7B (adaptive count: 3 tokens) and Mistral-7B-v0.3 (adaptive count: 4 tokens) with varying numbers of prefixed tokens. For Llama-2-7B without fine-tuning: 0 prefixed → 333.52 PPL (catastrophic); 1 prefixed → 74.37; 2 prefixed → 6.21; 3 prefixed (the adaptive count, including [BOS]) → 6.22. With fine-tuning: 0 prefixed → 17.63; 1 prefixed → 10.71; 2 → 6.01; 3 → 6.01. The key finding is that the adaptive count is not strict: 2 prefixed tokens (without the additional [BOS]) achieves essentially identical performance to 3 (6.01 vs. 6.01 with fine-tuning). The paper acknowledges this in Appendix D but chooses 3 for "consistency and simplicity." For Mistral-7B-v0.3 without fine-tuning: 0 → 90.02; 1 → 6.12; 2 → 5.84; 3 → 6.43; 4 (adaptive) → 5.89. The non-monotonic behavior at 3 tokens (6.43, worse than both 2 and 4) is unexplained but suggests that including the wrong additional token can be worse than including fewer tokens—the prefixed token set must collectively match the model's attention sink expectations.

Content of prefixed tokens (Table 10): Comparing the adaptive selection of high-frequency outlier tokens against using only the single highest-frequency token and against random selection (averaged over 10 trials, applied to both Llama-2-7B and Mistral-7B-v0.3 with W4A4KV4 O2 without fine-tuning). For Llama-2-7B: adaptive (".\n[BOS]") → 6.22 PPL; only highest frequency ("...") → 12.07; random → 66.51 (mean of 10 trials). For Mistral-7B-v0.3: adaptive ("\n.to[BOS]") → 5.89; only highest frequency ("\n\n\n\n") → 6.23; random → 80.05. The conclusion is clear: high-frequency selection generalizes well (approaching adaptive performance for Mistral when using only repeated "\n" tokens), but frequency ordering matters—the single highest-frequency token underperforms compared to the top-k set. Random selection is essentially useless, confirming that prefix content is a meaningful signal, not an arbitrary placeholder.

Long-context generalization (Table 11, Appendix E): To test whether PrefixQuant works beyond the 1024–2048 training context, the paper evaluates WikiText2 perplexity at 8192 context length on Llama-3-8B without any retraining or adaptation. PrefixQuant-O1 achieves 5.94 PPL (W4A8KV4) and 6.58 PPL (W4A4KV4), compared to QuaRot's 6.79 and 8.41 respectively. PrefixQuant-O2 achieves 6.04 and 6.82. DuQuant achieves 6.19 and 7.27. The FP16 baseline is 5.54. The critical result is that PrefixQuant-O2 (static) maintains its advantage over DuQuant (dynamic) at long context—6.82 vs. 7.27 at W4A4KV4—demonstrating that the prefixed token mechanism does not break down when sequences lengthen. This addresses the concern raised in Section 3.4 (justifying why clipping factors cannot be token-wise): "long-context scenarios introduce excessive storage overhead with token-wise clipping factors."

Comparison with related prefixed-token methods (Table 13, Appendix G.1): At W8A8 precision, PrefixQuant-O2 outperforms both QFeP and CushionCache by substantial margins: on Llama-2-70B, PrefixQuant (3.39 PPL) vs. QFeP (6.01 PPL); on Llama-3-8B, PrefixQuant (6.17 PPL) vs. CushionCache (7.37 PPL). These comparisons use static per-tensor quantization for PrefixQuant-O2 while QFeP and CushionCache use dynamic per-token or per-tensor configurations (as noted in the table). The large gaps suggest that these prior methods either do not handle token-wise outliers as completely as PrefixQuant, or their block-wise fine-tuning is less effective, or both.

Critical Assessment

The experiments in this paper are in many ways stronger than typical quantization papers—they cover seven model families across multiple scales, provide systematic ablations, and compare against an unusually comprehensive set of baselines. However, there are several respects in which the experimental evidence supports the paper's claims with important caveats, and certain claims would benefit from additional experiments that were not performed.

Claim 1: PrefixQuant Significantly Outperforms Prior Dynamic Quantization Methods

The experiments genuinely demonstrate this, but with an important baseline selection concern. In the W4A4KV4 comparisons (Table 2), PrefixQuant-O1 consistently outperforms SpinQuant, DuQuant, QuaRot, and Atom across all models. The margins are substantial: +3.08 accuracy points over SpinQuant on Llama-3-8B, +2.05 over QuaRot on Llama-2-7B, +1.04 over SpinQuant on Llama-2-70B. The ablation in Table 5 provides a mechanistic explanation: the gains come primarily from the prefixed token mechanism (the jump from 11.70 to 7.53 PPL at W4A4KV4), with fine-tuning providing a smaller additional benefit.

However, there are two concerns with the baseline implementations. First, SpinQuant is noted to use WikiText2 as its calibration dataset (grayed rows in Table 2), which means its perplexity numbers benefit from training-test overlap while its zero-shot accuracy numbers do not. This works in SpinQuant's favor for PPL comparisons (making PrefixQuant's PPL advantages more impressive) but potentially understates SpinQuant's true generalization ability on accuracy tasks (since the rotation matrices were overfit to WikiText2). The fair comparison would be to retrain SpinQuant with Pile calibration—the fact that this was not done (likely because SpinQuant provides pre-trained matrices) means the accuracy comparison is against a somewhat suboptimally calibrated SpinQuant.

Second, DuQuant and QuaRot are reproduced using the Pile calibration dataset, which is commendable for fairness. However, the reproduction may not capture all the optimization and hyperparameter tuning present in the original implementations. The QuaRot W4A4KV4 Llama-3-8B result of 8.41 PPL and 65.15% accuracy is substantially worse than other methods, including PrefixQuant without any fine-tuning (7.53 PPL). This large gap might reflect suboptimal reproduction rather than QuaRot's true capability, though the paper's QuaRot results on Llama-2 models align with those reported in QoQ's paper (as noted in Appendix A), lending credibility to the reproduction.

The comparison against QFeP and CushionCache (Table 13) at W8A8 precision is revealing but incomplete. Both methods are shown only at W8A8, and PrefixQuant's primary advantage over them (beyond the accuracy gap at W8A8) is that it works at much lower precision (W4A4, W4A8). The paper does not show QFeP or CushionCache results at W4A4 or W4A8, so it is possible that their methods also extend to lower precision with appropriate modifications. The 12-second vs. 12-hour detection time comparison is a valid practical advantage, but the accuracy comparison at low precision would be more informative for understanding whether PrefixQuant represents a genuinely more effective outlier elimination method or simply a more complete system pipeline.

Claim 2: PrefixQuant-O2 (Static Quantization) Surpasses Prior Dynamic Quantization Methods

This claim is supported but the evidence is nuanced. On Llama-2-13B W4A4KV4, O2 (70.36%) does indeed exceed O1 (70.05%) and SpinQuant (69.24%). On Llama-3-8B, O2 (71.08%) exceeds SpinQuant (68.23%) by a wide margin but falls slightly below O1 (71.31%). On Llama-2-7B, O2 (66.37%) is behind O1 (66.74%) but still ahead of SpinQuant (65.35%). The pattern is not uniform—static quantization is clearly competitive with dynamic on most models but not universally superior. The phrasing "surpasses prior dynamic quantization methods" is accurate for most comparisons, but the paper understates the cases where O1 outperforms O2 (Llama-2-7B, Llama-2-70B, Qwen-2-7B).

Moreover, the O2 advantage is demonstrated only in the context of PrefixQuant—it is possible, but not demonstrated, that applying block-wise fine-tuning with trainable static quantization parameters to SpinQuant or QuaRot (without PrefixQuant's prefixed tokens) would also improve their static quantization performance, though certainly not to O2 levels given the catastrophic failure of static quantization without prefixed tokens (Table 5, row "+ static quantization": 141.02 PPL). The critical experiment that demonstrates the causal role of prefixed tokens in enabling static quantization is the comparison between "PrefixQuant-O2 + static quantization without prefixed outliers" (141.02 PPL) and "PrefixQuant-O2 + prefixed outliers without fine-tuning" (7.93 PPL). This 17.8× improvement is unambiguous: static quantization simply does not work without eliminating token-wise outliers first, and PrefixQuant's mechanism for doing so is highly effective.

What is missing is a direct comparison of O2 against O1 on end-to-end latency for all models, not just one. The paper provides speedups for Llama-2-7B only (Table 6). If the claim is that O2 is more efficient than O1 while being comparably accurate, the accuracy evidence is in Tables 2–4 but the efficiency evidence is limited to a single model on a single GPU. This is a standard pattern in quantization papers—detailed latency benchmarking is expensive and hardware-specific—but it means the practical advantage of O2 over O1 is more hypothesized than demonstrated across the full model range.

Claim 3: Token-Wise Outliers Are the Dominant Remaining Error Source After Channel-Wise Methods

Strongly supported by the ablation evidence. Table 5 shows that going from "Base RTN" (1282.34 PPL) to "+ rotation" (24.98 PPL) addresses channel-wise outliers, a 51× improvement. Going from "+ Grid search initialization" (11.70 PPL) to "+ prefixed outliers" (7.53 PPL) addresses token-wise outliers, a 1.55× improvement. The absolute PPL reduction from rotation is much larger, but this reflects that channel-wise outliers are the first-order problem—without solving them, you never get to the regime where token-wise outliers matter. The important metric is the residual PPL gap to FP16 (6.14): after rotation, the gap is 24.98 - 6.14 = 18.84 PPL; after prefixed tokens, the gap drops to 7.53 - 6.14 = 1.39 PPL—a 13.6× reduction in the gap. This quantifies that token-wise outliers account for the vast majority of the remaining error after rotation.

The visualization evidence in Figure 3 and Appendix H (Figures 9–20) verifies this mechanism directly by showing that post-rotation max(top-1/median) ratios remain at hundreds, and post-prefixing these drop to single digits. The correlation between the ratio reduction and PPL improvement is not quantified statistically (there is no scatter plot of ratio-vs-PPL across layers), but the qualitative pattern is consistent.

A gap in the evidence: the paper does not show what happens if you apply block-wise fine-tuning without prefixed tokens on top of rotation + grid search. Table 5 only shows "PrefixQuant-O1" (which includes prefixed tokens) with and without fine-tuning. The missing row would be "rotation + grid search + block-wise fine-tuning (no prefixed tokens)"—this would quantify how much of the fine-tuning benefit can be achieved without outlier isolation, and hence how much of the outlier-induced PPL gap is recoverable through training alone. Based on the "without fine-tuning" numbers (11.70 PPL with rotation + grid search), it is reasonable to expect that fine-tuning could improve this somewhat, but the instability of training with outlier tokens (mentioned in Section 4.1) suggests the improvement would be limited. The omission of this row is a small but noticeable gap in the otherwise thorough ablation.

Claim 4: Prefixing High-Frequency Outlier Tokens Constrains Outliers to Prefixed Positions

Directly verified by Figure 4d which shows that after prefixing, outlier tokens only appear at positions 0 and 1. The claim is binary and the evidence is binary—the quarantine works as intended. However, the figure shows this for one model (Llama-2-7B) and the paper does not provide equivalent position-distribution plots for other models. This is understandable given space constraints, but it means the universal claim ("constraints outlier tokens to the prefixed tokens" without model qualification) relies partly on the indirect evidence of PPL improvements across models rather than direct visualization of outlier position distributions for each model family. The consistent PPL and accuracy improvements across all models (Tables 2, 3, 15) provide circumstantial evidence that the mechanism generalizes, but a skeptic could argue that for models like Llama-3-8B (which only have initial-token outliers), the benefit might come from a different mechanism than the quarantine effect observed for Llama-2-7B.

Missing Experiments and Evaluations

Several experiments would have strengthened the paper considerably:

Multi-turn conversation or generation tasks. All evaluation is on perplexity, zero-shot classification, and MMLU—static benchmarks that involve single-pass inference. PrefixQuant modifies the KV cache with prefixed tokens, which interact with the attention mechanism across the entire sequence. In multi-turn dialogue or long-form generation where the model's own outputs become part of the KV cache, the interaction between the persistent prefixed tokens and the growing generated context could produce effects not captured by perplexity on fixed text. The paper's long-context evaluation (Table 11) partially addresses this, but WikiText2 is still a fixed-text corpus, not autoregressive generation.

Ablation of the Hadamard rotation components. The paper treats Hadamard rotation as a black-box framework from QuaRot/SpinQuant and does not ablate which rotation positions (R1, R2, R3, R4) contribute to the observed performance. Given that R4 (the online rotation at down_proj input) is where the most extreme token-wise outliers occur (Figure 3(I)), and that R3 (the online rotation at Q/K after RoPE) is where lower outliers persist (Figure 3(III)), it would be informative to see whether PrefixQuant's prefixed tokens could replace some of the online rotations entirely. If prefixed tokens eliminate the need for R3 or R4, the online rotation overhead could be removed, further improving inference speed.

Comparison against activation-aware weight quantization only. The paper focuses on weight-activation quantization, but Table 12 shows that PrefixQuant also improves weight-only quantization (where activations remain FP16). The mechanism by which prefixed KV tokens improve weight-only quantization is not direct—the prefixed tokens don't directly change weight quantization—so the improvement presumably comes through more stable fine-tuning (the outlier instability mentioned in Section 4.1). This connection is stated but not empirically verified: the paper does not compare training curves with and without prefixed tokens to show reduced gradient variance or faster convergence. This would strengthen the claim that outlier-induced training instability is the mechanism for the weight-only gains.

Robustness to different calibration datasets. Table 7a ablates C4, RedPajama, and Pile for fine-tuning, but the outlier token detection itself (which determines the prefixed tokens) is also performed on Pile. It is possible that the set of high-frequency outlier tokens varies across corpora (e.g., code-heavy datasets might have different delimiter patterns than web text). The paper does not test whether prefixed tokens detected from Pile generalize to other evaluation domains, or whether PrefixQuant's performance degrades when the calibration domain and deployment domain differ.

Comparison with other outlier handling approaches at equalized training budget. PrefixQuant's block-wise fine-tuning uses 512 × 1024 tokens over 20 epochs (W4A4). QuaRot and DuQuant typically use lighter calibration (QuaRot uses 128-token sequences from WikiText2 in its original paper). It is possible that part of PrefixQuant's advantage comes from more extensive fine-tuning rather than the prefixed token mechanism per se. The paper partially addresses this by showing the "without fine-tuning" results in Table 5, where PrefixQuant-O1 without fine-tuning (7.53 PPL) already outperforms QuaRot with fine-tuning (8.33 PPL for the "Base" row without rotation correction—though this is not a clean comparison because QuaRot's fine-tuning differs from PrefixQuant's). A cleaner experiment would be to apply PrefixQuant's block-wise fine-tuning procedure to QuaRot/DuQuant models without the prefixed tokens, to test whether the fine-tuning protocol alone accounts for the gains. This is the missing ablation noted earlier.

Statistical significance and variance. No experiment reports error bars, standard deviations, or confidence intervals. Perplexity and accuracy are reported as single numbers without any measure of variance across random seeds or calibration data subsamples. The zero-shot accuracy metrics on benchmarks with 500–2000 test examples (e.g., ARC Challenge has ~1172 test questions, PIQA has ~1838) have inherent sampling variance, and differences of 0.5–1.0 accuracy points may be within noise. The consistent direction and magnitude of improvements across models and benchmarks partially mitigates this concern—it is unlikely that 10+ independent comparisons all favor PrefixQuant by chance—but individual pairwise comparisons lack statistical grounding.

Visualization quality and completeness. The outlier distribution figures in Appendix H (Figures 9–20) are comprehensive but use a somewhat confusing visualization format: each subfigure shows line plots with multiple series (Top-1, Top-2, Top-3, Median, Min-1, etc.) across layers, making it difficult to compare across the three columns (Original, Rotation, PrefixQuant). A more effective visualization would be a single scatter or histogram comparing ratios before and after each processing step, aggregated across layers. The current format requires the reader to mentally compare three complex plots and extract the max(top-1/median) annotations, which are provided but not visually prominent.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Gains

The assumption or constraint. The prefixed token detection procedure, while training-free, requires passing calibration data through the full-precision model to identify outlier tokens. The paper reports this cost as 12 seconds for Llama-2-7B and 1 minute for Llama-3-70B (Table 6, Appendix C) and emphasizes that this is dramatically faster than prior grid-search methods (12 hours for QFeP on Llama-3-8B). However, the detection cost is measured only in wall-clock time and is presented as a one-time offline expense. The paper does not discuss an important hidden assumption: the set of high-frequency outlier tokens is stationary across deployment domains. The outlier token set is computed from the Pile calibration data (8 samples × 1024 tokens), and the paper provides no evidence that tokens identified as outliers in this general web-text corpus remain the correct prefix for specialized domains.

The consequence. A practitioner deploying PrefixQuant on domain-specific text (e.g., medical records, legal documents, non-English languages, or code) faces uncertainty about whether the detected prefix tokens will constrain outliers in that domain. If domain-specific delimiters or function words produce different outlier patterns, the prefixed tokens would be mismatched, and token-wise outliers could leak into the quantized tensors. Table 10 shows that prefix content matters substantially—random prefix selection yields 66.51 PPL vs. 6.22 for the adaptive selection on Llama-2-7B. But this ablation only tests different prefixes within Pile's distribution. There is no cross-domain evaluation (e.g., prefixes detected from Pile evaluated on code or non-English text) to quantify how much the prefix token set generalizes.

What evidence exists in the paper. The paper provides no explicit cross-domain evaluation. The long-context experiment (Table 11) evaluates WikiText2 at 8192 tokens, which constitutes a mild domain shift from Pile but still within English web-text. Table 7a shows that fine-tuning on C4, RedPajama, or Pile yields comparable results (PPL range: 7.42–7.60), but the outlier detection itself is performed on Pile in all cases—the prefixed tokens are not re-detected from the fine-tuning corpus. The paper does not test whether a model quantized with Pile-detected prefixes maintains accuracy when evaluated on, for example, Github code, PubMed abstracts, or multilingual text.

Mitigation status. Not addressed. The paper does not suggest re-detecting prefixes per deployment domain or analyze the sensitivity of prefix detection to calibration data distribution. A practitioner would need to run the detection procedure on their own domain data, which is feasible (12 seconds–1 minute) but adds a deployment step not reflected in the paper's workflow. The failure mode—severe perplexity degradation under distribution shift—is plausible based on Table 10's random-prefix results but is not characterized.

The Virtually Zero Improvement on the Hardest Problems (Weight-Only Quantization at 2-Bit)

The assumption or constraint. The paper frames PrefixQuant as a broadly effective method applicable to "various precision levels (W4A4KV4 and W4A8KV4) and granularities (dynamic and static quantization)" (abstract). While the main weight-activation quantization results at W4A4 and W4A8 are consistently strong, the extension to weight-only quantization (Table 12, Appendix F) reveals a sharp effectiveness boundary at the lowest precision. At W2A16g128 on Llama-3-8B, PrefixQuant achieves 11.97 PPL and 64.00% average accuracy—a 5.05-point accuracy improvement over EfficientQAT, but still a catastrophic degradation from FP16 (6.14 PPL, 72.71% accuracy). The model loses 8.71 accuracy points and nearly doubles perplexity.

The consequence. At 2-bit weight quantization, PrefixQuant improves over prior methods but does not produce a usable model for most applications. An 8.7 accuracy point drop on standard benchmarks implies the model has lost substantial knowledge and reasoning capability. This is not a criticism of PrefixQuant specifically—2-bit quantization is an extreme compression regime where no current method achieves high fidelity—but the paper's framing as achieving "excellent performance across various models, quantization precisions, and granularities" (Section 6) overstates the practical viability at 2-bit. A practitioner evaluating whether to deploy a 2-bit quantized model would see the 64% accuracy on Llama-3-8B (vs. 72.7% FP16) and likely conclude that the compression-accuracy tradeoff is unacceptable for their use case, despite PrefixQuant's relative improvement.

What evidence exists in the paper. Table 12 directly provides the numbers. The PPL degradation from FP16 to W2A16 is 5.83 points for Llama-3-8B (6.14 → 11.97) and 8.16 points for Llama-3-70B (2.85 → 11.01)—both representing more than a 3× increase in perplexity. These are larger absolute degradations than any of the weight-activation quantization results in Tables 2–3, where PPL increases are typically 0.5–2.0 points. The paper does not discuss these numbers in the main text—the weight-only results are relegated to Appendix F with minimal commentary.

Mitigation status. The paper does not frame this as a limitation; it presents the weight-only results as evidence of PrefixQuant's broad applicability. The acknowledgment is implicit: "the optimal token number for fine-tuning datasets may change with quantization precision. Generally, lower precision requires more training data" (Appendix D). The paper cites EfficientQAT's use of 4096 × 2048 tokens for W2A16 (vs. PrefixQuant's 512 × 1024 for W4A4) as evidence that the 2-bit training protocol is likely suboptimal. But this suggests the 2-bit results represent a lower bound on PrefixQuant's capability at 2-bit, not a fundamental limitation. The question of whether more extensive fine-tuning could close the gap to usable accuracy at 2-bit is unexamined.

Single Evaluation Domain: All Tasks Are English Text with Static Benchmarks

The assumption or constraint. The entire evaluation suite—WikiText2 perplexity, five zero-shot common-sense reasoning tasks (PIQA, ARC, HellaSwag, WinoGrande), and MMLU—consists of English-language text benchmarks with static inputs (pre-written questions, fixed passages for perplexity). The paper does not evaluate on: code generation (HumanEval, MBPP), multilingual tasks (which are particularly important for Llama-3 and Qwen-2 that support multiple languages), multi-turn dialogue or conversation, mathematical reasoning (GSM8K, MATH), or long-form generation tasks where the model's own outputs recursively become part of the KV cache.

The consequence. The prefixed token mechanism modifies the self-attention computation by permanently injecting o additional key-value pairs into every attention operation. In static benchmarks where the input context is fixed and the model only generates a single token or class label, these additional KV entries are an additive bias that (as demonstrated) improves quantization accuracy. In autoregressive generation—particularly long-form generation or multi-turn conversation—the prefixed tokens produce a persistent attention pattern that interacts with the growing generated context. The model may learn to use the prefixed token positions differently when it can attend to its own previous outputs, potentially producing degradation not captured by perplexity on fixed text.

This concern is not hypothetical for deployment. Consider a conversational agent: with each turn, the KV cache grows to include the system prompt, the conversation history, and the prefixed tokens. The prefixed tokens' fixed representation—precomputed offline in full precision—interacts with the quantized representations of the input-dependent tokens. As the sequence length grows, the ratio of prefixed tokens to total sequence length shrinks (3–4 tokens vs. thousands), and the stabilizing effect of the prefix on the attention distribution may diminish. The paper's long-context experiment (Table 11, 8192-token WikiText2) provides partial evidence that PrefixQuant generalizes to longer fixed texts, but WikiText2 at 8192 tokens is still a single forward pass over static text, not autoregressive decoding where error accumulation from earlier tokens compounds.

What evidence exists in the paper. The long-context experiment (Table 11) is the only evaluation beyond the standard 2048-token context. It shows PrefixQuant-O1 maintaining good perplexity at 8192 tokens (5.94 PPL vs. 5.54 FP16 for Llama-3-8B at W4A8KV4), but this is still perplexity on fixed text. There is no evaluation of generated text quality, no multi-turn dialogue benchmark, and no code generation accuracy measure. The paper's evaluation is entirely within the standard quantization evaluation paradigm inherited from prior work (QuaRot, SpinQuant, DuQuant)—but PrefixQuant modifies the model's forward pass in a novel way (persistent KV cache entries) that could have downstream effects those methods do not.

Mitigation status. Not addressed. The paper does not discuss the autoregressive generation setting or acknowledge the lack of generation-based evaluation as a limitation. The consistent pattern across static benchmarks provides evidence that the method works for single-pass inference, but the leap to "PrefixQuant for LLM deployment" requires untested assumptions about autoregressive behavior.

The Latency Trade-off Between O1 and O2 Is Under-Characterized

The assumption or constraint. The paper positions PrefixQuant-O2 (static per-tensor quantization) as "more efficient than O1 (i.e. lower latency)" (Table 1 caption) and provides speedup numbers for a single model (Llama-2-7B) on a single GPU (RTX 3090) at a single precision (W4A4). Table 6 reports that O2 achieves 2.74× prefilling speedup and 2.16× decoding speedup over FP16, compared to 2.67× and 2.11× for O1—a modest 2.4–2.6% improvement from O1 to O2. For larger models, different GPU architectures, or batched inference, the O1-to-O2 speedup could differ substantially. On a memory-bandwidth-bound device, the online reduction operations (computing per-token max/min for dynamic quantization) might be essentially free relative to the matrix multiply cost, making the O1/O2 distinction irrelevant for latency. On a compute-bound device with optimized kernels, the elimination of per-token reductions in O2 could yield larger gains.

The consequence. A practitioner deciding whether to implement PrefixQuant-O2 (which requires additional infrastructure for static quantization parameter storage and kernel integration) versus the simpler O1 (which uses standard per-token dynamic quantization) cannot make an informed cost-benefit analysis based on the paper's data. The accuracy difference between O1 and O2 varies across models: O2 equals or exceeds O1 on Llama-2-13B and Llama-3-70B W4A8KV4, but O1 outperforms O2 on Llama-2-7B W4A4KV4 by 0.37 accuracy points and on Qwen-2-7B W4A4KV4 by 1.16 accuracy points. On models where O2 is less accurate, the practitioner needs to know whether the latency savings justify the accuracy loss—and the paper provides latency data for only one configuration.

What evidence exists in the paper. Table 6 provides the only latency comparison between O1 and O2, for one model–GPU–precision combination. Tables 2, 3, 4, and 15 provide accuracy comparisons across all models but do not include latency columns. The paper's claim that O2 is "more efficient" (Table 1) and achieves "up to 2.74× prefilling speedup and 2.16× decoding speedup" (abstract) collapses two different metrics: the specific 2.74× number is for O2 specifically, while the range "up to" suggests this is the maximum observed across configurations (which it is, since only one configuration was benchmarked). A more precise phrasing would be "O2 achieves 2.74× prefilling speedup on Llama-2-7B W4A4 on RTX 3090, with O1 achieving 2.67× on the same setup."

Mitigation status. Partially acknowledged implicitly—the paper never claims comprehensive latency benchmarking and follows the standard practice in quantization papers of providing a representative speedup measurement. However, the explicit framing of O2 as the "more efficient" configuration that "targets to push the limitation of more efficient static quantization" (Section 5.1) creates an expectation of demonstrated efficiency gains that the data supports only narrowly. A practitioner would need to benchmark O1 vs. O2 on their specific hardware, model, and batch size before choosing.

The assumption or constraint. Section 4.1 notes that outlier tokens "cause instability during block-wise fine-tuning" and that PrefixQuant's prefixed tokens stabilize training by isolating outliers to non-quantized KV cache entries. The block-wise fine-tuning procedure (Section 4.3) trains each transformer block sequentially with MSE loss. The assumption is that the prefixed tokens eliminate the training instability, enabling stable convergence in 10–20 epochs.

The consequence. The paper does not quantify or characterize the training instability it claims to solve. There are no training curves showing loss divergence or gradient norm spikes without prefixed tokens, no comparison of training dynamics with and without prefixes, and no reporting of how many random seeds or hyperparameter configurations were needed to achieve the reported results. If block-wise fine-tuning without prefixed tokens is unstable—as the paper asserts—then the post-training quantization pipeline is fragile in ways that matter for practical adoption. A practitioner implementing this method on a new model family would need to know whether the 20-epoch, 5e-5/5e-6 learning rate configuration reliably converges, or whether hyperparameter sensitivity remains even with prefixed tokens.

What evidence exists in the paper. The only evidence for training instability is the qualitative statement in Section 4.1 and the indirect evidence from the ablation in Table 5. The missing row in Table 5—"rotation + grid search + block-wise fine-tuning without prefixed tokens"—would directly demonstrate the instability by showing whether fine-tuning helps or hurts when token-wise outliers remain. Without this ablation, the causal claim that prefixing stabilizes training is inferred from the final performance rather than demonstrated through training dynamics analysis.

Mitigation status. Not addressed. The paper does not provide training curves, report failed training runs, or ablate the training procedure's robustness to hyperparameters. The fine-tuning hyperparameters (Table 8, Appendix D) show that W4A8KV4 converges at 10 epochs and W4A4KV4 at 20, but this convergence evidence is only shown with prefixed tokens active. The instability claim hinges on the reader accepting that the 1282 → 7.53 PPL improvement from "Base" to "PrefixQuant-O1 w/o FT" at W4A4KV4 (Table 5) would not be recoverable through more extensive fine-tuning of the rotation + grid search configuration—a plausible but unverified claim.

The Method Adds `o` Full-Precision KV Cache Entries That Persist for the Entire Sequence

The assumption or constraint. The prefixed tokens are stored as k', v' in the KV cache in full precision (FP16) for every transformer layer, for every attention head. Equation 4 shows these are prepended to the input-dependent KV entries at every self-attention operation. For a model with L transformer blocks, H attention heads, and per-head dimension d_h, the storage cost is o × L × H × d_h × 2 (for K and V) FP16 values, plus o × d_model × L × 2 if any cross-head operations store the concatenated KV tensors. The paper does not report this memory overhead or compare it to the memory savings from KV cache quantization.

The consequence. For the models tested, o ranges from 1 (Llama-3-8B, Qwen-2-7B) to 4 (Mistral-7B-v0.3, Llama-2-70B). For Llama-2-70B with L=80, H=64, d_h=128, the prefixed KV cache uses 4 × 80 × 64 × 128 × 2 = 5,242,880 FP16 values ≈ 10.5 MB. This is a fixed, per-layer overhead that does not scale with sequence length. In a long-context scenario (e.g., 128K tokens), the input-dependent KV cache for the same model at FP16 would be approximately 128,000 × 80 × 64 × 128 × 2 = ~167 GB, so the prefixed overhead is negligible (~0.006% of total KV cache). However, for short sequences or large batch sizes where KV cache is small, the relative overhead is larger: with 2048-token sequences, the prefixed overhead is ~0.4% of the KV cache. This overhead is likely negligible in practice, but it represents a permanent memory cost that cannot be removed by KV cache quantization—the prefixed tokens must remain in FP16 even if the rest of the KV cache is aggressively quantized, creating an asymmetry in the memory layout that kernel implementations must handle.

More importantly, the computational overhead of attending to o additional tokens in every self-attention operation is not benchmarked. The attention computation at each layer now processes T + o key-value pairs instead of T. For typical T (hundreds to thousands), the overhead is small, but for very short sequences (e.g., single-turn queries with T=32), the o=4 additional tokens represent a 12.5% increase in attention computation per layer. The paper's speedup measurements (Table 6) include this overhead implicitly—they benchmark the full quantized model with prefixed tokens—but the overhead is not isolated or discussed.

What evidence exists in the paper. The paper does not report the memory overhead of the prefixed KV cache entries. The speedup numbers in Table 6 implicitly account for the computational overhead (since they benchmark the actual PrefixQuant implementation), but they show 2.74× speedup over FP16, which is dominated by the 4× reduction in matrix multiplication cost from W4A4 quantization. The marginal cost of the additional attention computation is not visible in these aggregate numbers.

Mitigation status. Not addressed. The paper does not discuss the memory or computation overhead of the prefixed tokens. For practical deployment, this overhead is small enough to be acceptable for most use cases, but the lack of explicit accounting means a practitioner cannot verify that the overhead is negligible without implementing and measuring it themselves.

7. Implications and Future Directions

How This Work Changes the Landscape

PrefixQuant shifts the conversation around LLM activation quantization from a channel-centric to a token-position-centric view of what causes quantization error. The paper does not introduce a new quantization algorithm, a new architecture, or a new training paradigm—it introduces a diagnostic reframing with an elegantly simple corresponding intervention. The core insight is that token-wise outliers are not a residual nuisance to be absorbed by adaptive per-token quantization; they are a structurally distinct failure mode caused by the attention mechanism's SoftMax requirement that every query distributes non-zero attention weight, and they can be eliminated entirely by redirecting the model's attention-sink behavior toward controlled, precomputed positions.

This is not a paradigm shift—it builds explicitly on Hadamard rotation for channel-wise outliers and block-wise fine-tuning for error compensation—but it does something more practically valuable: it closes a gap that the field had implicitly accepted as fundamental. Before PrefixQuant, the default wisdom was that static per-tensor activation quantization was simply too coarse for LLMs, and that per-token dynamic quantization was the necessary cost of acceptable accuracy. The paper's Tables 2–4 demonstrate that this was not a property of LLM activations per se, but of uncontrolled token-wise outliers. Once those outliers are quarantined (Figure 4d), static quantization not only becomes viable but can match or exceed dynamic quantization while enabling additional inference speedups (Table 6). This reframes the accuracy-efficiency tradeoff: what was previously accepted as a hard constraint is revealed as a solvable engineering problem with a ~12-second detection cost.

The paper also reconciles a subtle tension in the prior literature. StreamingLLM (Xiao et al., 2023b) and related work established that initial tokens serve as "attention sinks" that are essential for stable generation—they cannot simply be removed without causing model collapse. Massive activation work (Sun et al., 2024) showed that these same attention sinks produce extreme activation values that destroy quantization accuracy. The apparent contradiction was: attention sinks are necessary for model behavior, but attention sinks destroy quantization. PrefixQuant resolves this by showing that attention sinks are redirectable: you can give the model a different set of initial tokens to use as sinks, precompute their activations in full precision offline, and the model will use those sinks instead of the input-dependent tokens. This preserves the model's attention behavior (it still has sinks to absorb attention mass) while eliminating the quantization problem (the sinks' extreme activations never enter the quantized computation graph). The insight is that the model's need for attention sinks and the quantizer's need for uniform activation distributions are not in conflict—they can be satisfied simultaneously through a structural rearrangement of the input.

Which research directions become more attractive? Verifier-free, structural approaches to quantization that exploit properties of the model's forward pass rather than relying on post-hoc optimization or expensive search. The paper makes a compelling case that the cheapest solution is often the most elegant: rather than training rotation matrices (SpinQuant), searching over prefix combinations (CushionCache), or modifying the attention architecture (Bondarenko et al.), you can simply detect which tokens the model wants to treat as sinks and give them those tokens in a controlled way. This suggests a broader class of methods that work with the model's existing structural biases rather than fighting against them—an "exploit, don't suppress" approach to efficient inference.

Which directions become less attractive? Purely channel-wise outlier mitigation as a standalone solution. PrefixQuant demonstrates unambiguously that channel-wise rotation alone (even with learned matrices, as in SpinQuant) leaves substantial residual quantization error from token-wise outliers (Table 5: 24.98 PPL with rotation only at W4A4KV4, vs. 7.53 after adding prefixed tokens). Future quantization methods must address both axes—channel and token—or they will hit the same ceiling. Methods that propose ever-more-sophisticated rotation schemes without addressing token-position non-uniformity are fighting the last war.

Follow-Up Research This Work Enables

Dynamic prefix selection during autoregressive generation. The paper's prefix tokens are statically precomputed from a calibration dataset and fixed for all inputs. This raises a natural question: in autoregressive generation, as the model produces its own output tokens and those tokens enter the KV cache, do new token-wise outliers emerge from generated content that wasn't present in calibration? A follow-up study would instrument the quantized PrefixQuant model during long-form generation (e.g., 4096+ tokens of open-ended text), measure per-token maximum activations at each layer for the generated tokens, and test whether the max(top-1/median) ratios from Figure 3 remain controlled throughout generation or drift upward as the sequence lengthens. If drift occurs, it would motivate dynamic prefix refreshing—periodically inserting additional prefixed tokens into the KV cache during generation to recapture outlier attention patterns. The experiment would quantify trajectory-level outlier statistics (not just final perplexity) and correlate outlier drift with degradation in generation quality metrics like MAUVE or human preference scores.

Cross-domain prefix transfer and domain-adaptive detection. Table 10 shows that the content of prefixed tokens matters dramatically (6.22 PPL for adaptive vs. 66.51 for random on Llama-2-7B without fine-tuning), but all experiments use prefixes detected from Pile and evaluate on Pile-derived or similar English web-text benchmarks. A rigorous follow-up would: (1) detect prefixes from Pile, then evaluate PrefixQuant models on domain-shifted corpora—PubMed abstracts, GitHub code, legal documents (MultiLegalPile), and multilingual text (FLORES-200)—measuring both perplexity and task-specific metrics; (2) for each target domain, run PrefixQuant's detection procedure on in-domain calibration data and compare the resulting prefix token sets against the Pile-derived set; (3) quantify whether domain-adaptive prefix selection recovers performance. The key finding would be whether outlier token identity is universal across domains (driven purely by model architecture and training) or domain-dependent (driven by token frequency distributions). A result showing that delimiter tokens like "\n" and "." are universal outliers across domains would validate the paper's implicit claim of broad applicability; a result showing substantial variation would motivate lightweight domain-adaptive detection as a necessary deployment step.

Ablation of individual Hadamard rotation positions with and without prefixing. The paper adopts QuaRot/SpinQuant's full rotation scheme (R1–R4, Figure 7) without ablating which rotations remain necessary after prefixed tokens eliminate the worst outliers. A targeted ablation would train PrefixQuant variants that disable one rotation at a time (R1 only, R1+R2, R1+R2+R3, all four) and measure the perplexity and accuracy impact across multiple models at W4A4KV4. The hypothesis is that R4—the online rotation at the down_proj input, which Figure 3(I) shows is the site of the most extreme upper outliers—might become unnecessary once prefixed tokens eliminate the token-wise outliers that rotation was partially mitigating. If removing R4 recovers comparable accuracy, the online rotation overhead could be eliminated entirely, improving inference throughput beyond the 2.74×/2.16× reported in Table 6. A clean negative result (all rotations remain necessary) would demonstrate that channel-wise and token-wise outliers are genuinely independent problems requiring independent solutions.

Training dynamics analysis with and without prefixed tokens. Section 4.1 claims that outlier tokens "cause instability during block-wise fine-tuning," but this is never directly demonstrated. A follow-up study would run the block-wise training procedure (Section 4.3) on a model with rotation but without prefixed tokens, logging per-epoch validation perplexity, gradient norm statistics, and the variance of quantization parameter updates across epochs, then compare to the equivalent training run with prefixed tokens active. The concrete measurements would be: (1) the standard deviation of block-wise loss across training steps (a measure of optimization noise); (2) the frequency of gradient norm spikes exceeding a threshold; (3) the correlation between outlier token presence (per Figure 3 ratios) and local loss curvature. If instability is confirmed, it would demonstrate that prefixing provides a dual benefit—direct quantization error reduction and training stabilization—which would motivate using prefixing even in scenarios where only weight quantization is needed (aligning with the weight-only gains in Table 12). If instability is not confirmed (training converges without prefixes, just to a worse optimum), it would reframe prefixing as purely a quantization-time intervention.

Interaction between prefixed tokens and position encoding. The paper's prefixed tokens are inserted at the beginning of every sequence, shifting all subsequent tokens' position indices by o. For models using Rotary Position Embedding (RoPE), this means the input-dependent tokens now start at position o rather than position 0, and the relative position distances between tokens are preserved but their absolute positions shift. The paper implicitly assumes this shift is harmless—the model was trained on variable-length sequences and should be robust to small absolute position shifts—but this assumption is never tested. A follow-up would compare PrefixQuant against a variant where the prefixed tokens' position encodings are set to negative indices (so the first real token remains at position 0) and measure whether the position shift affects long-context perplexity or task accuracy. This matters for very long sequences where absolute position encoding may interact with the model's context-length generalization behavior. The experiment would reveal whether PrefixQuant's mechanism depends on the prefixed tokens being first in the sequence (positional primacy as an attention-sink trigger) or simply being present in the KV cache (content-based attention sinking, independent of position).

Extension to vision-language models and encoder-decoder architectures. The paper evaluates exclusively on decoder-only autoregressive LMs (Llama, Mistral, Qwen). Token-wise outliers in the attention mechanism are hypothesized to arise from the SoftMax forcing non-zero attention scores—a mechanism present in all transformer attention operations, including cross-attention in encoder-decoder models and self-attention in vision transformers. A natural extension would apply PrefixQuant's detection procedure to: (1) the encoder self-attention and cross-attention in T5 or FLAN-T5; (2) the vision encoder and text decoder of LLaVA or similar VLMs; (3) the self-attention in ViT-based vision encoders. For each architecture, the specific research question is whether outlier token types differ (e.g., do image patch tokens produce outliers analogous to text delimiter tokens?) and whether the prefixing mechanism can be adapted to encoder-decoder settings where the input length is fixed but cross-attention introduces a second sequence dimension. The practical payoff would be extending PrefixQuant's static quantization benefits to the broader model zoo beyond autoregressive LMs.

Practical Applications and Downstream Use Cases

On-device deployment of mid-sized LLMs on consumer GPUs. The paper's primary speedup demonstration—2.74× prefilling and 2.16× decoding on an RTX 3090 (Table 6)—directly enables running models that would otherwise exceed consumer GPU memory at usable throughput. An FP16 Llama-2-13B requires ~26GB for weights alone, exceeding the RTX 3090's 24GB; with W4A4 PrefixQuant-O2, the weight memory drops to ~6.5GB, fitting comfortably with room for KV cache. More broadly, for any deployment where GPU memory is the binding constraint (edge devices, consumer laptops with 8–16GB VRAM), PrefixQuant's ~4× memory reduction for weights and activations, combined with static quantization's operator fusion benefits, enables model scales that would otherwise require cloud offloading. The practical recipe implied by the paper: detect prefixes once per model (1 minute for Llama-3-70B), run block-wise fine-tuning once (2–17 hours depending on scale), then deploy the quantized model with precomputed prefix KV cache entries. The one-time offline cost amortizes over all subsequent inference.

Batch inference pipelines for evaluation and data generation. Organizations running large-scale evaluations (e.g., scoring thousands of model outputs against benchmark suites) or generating synthetic training data through LLM inference can benefit from PrefixQuant's accuracy-efficiency profile. The paper shows that at W4A8KV4, PrefixQuant-O1 retains 97.5% of FP16 MMLU accuracy (60.49% vs. 62.07% on Llama-3-8B, Table 4) and 98.8% of average zero-shot accuracy (72.57% vs. 72.71%, Table 3). For batch workloads where throughput dominates and small accuracy losses are acceptable (e.g., filtering candidate outputs, generating distractors, scoring routine classifications), switching from FP16 to PrefixQuant-O2 yields ~2.7× throughput improvement with near-lossless accuracy on knowledge-intensive benchmarks. The static quantization configuration (O2) is particularly attractive for batch inference because the fixed quantization parameters require no per-token online computation, simplifying kernel implementation and reducing CPU-GPU synchronization points.

Low-latency interactive applications where memory bandwidth dominates. The paper's decoding speedup (2.16× tokens/second, Table 6) targets the autoregressive generation phase where each token depends on the previous one, making the process memory-bandwidth-bound (loading weights from memory dominates latency). In interactive applications—chatbots, code completion, real-time translation—where users perceive latency directly, the 2×+ decoding speedup from 4-bit quantization with PrefixQuant directly improves user experience. The additional prefixed token overhead (0.4% more KV cache entries at 2048-token context, as estimated in Section 6) is negligible relative to the weight-memory bandwidth savings. The key practical consideration: PrefixQuant's prefix detection and fine-tuning must be rerun when the base model is updated (fine-tuned on new data, instruction-tuned, etc.), but this is a one-time cost per model version that can be integrated into the model release pipeline.

Quantization-aware training for custom fine-tuned models. Practitioners fine-tuning Llama-family models on domain-specific data can integrate PrefixQuant into their training pipeline. Rather than training in FP16 and quantizing post-hoc, they can: (1) start from a precomputed PrefixQuant checkpoint (weights, quantization parameters, and prefix KV cache entries); (2) continue block-wise fine-tuning on their domain data using the same MSE reconstruction objective (Section 4.3) but with in-domain calibration samples; (3) optionally re-detect prefixed tokens if the domain shift introduces new delimiter patterns. The paper's Table 7a shows that Pile, C4, and RedPajama all yield comparable fine-tuning results (7.42–7.60 PPL), suggesting the procedure is robust to calibration data distribution. This enables deployment of heavily quantized, domain-adapted models without a separate quantization step after fine-tuning.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration2023· inference-optimization← LLM-judged similar
- READ MORE2023· inference-optimization← in-domain predecessor
- "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization2024· inference-optimization← similar topic
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding2015· inference-optimization← similar topic
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration2024· inference-optimization← in-domain predecessor
- Addition is All You Need for Energy-efficient Language Models2024· inference-optimization← in-domain predecessor
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test2025· inference-optimization← similar topic
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training2025· inference-optimization← similar topic
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision2024· inference-optimization← similar topic
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models2024· low-precision← topic thread
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits2024· pretraining← LLM-judged similar
- ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks2023· low-precision← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads2024· inference-optimization→ in-domain predecessor
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs2024· inference-optimization→ in-domain predecessor
- Inference Scaling for Long-Context Retrieval Augmented Generation2024· reasoning→ topic thread
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval2024· inference-optimization→ similar topic
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads2024· inference-optimization→ similar topic
- Speculative Streaming: Fast LLM Inference without Auxiliary Models2024· inference-optimization→ similar topic