QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

🎯 Pitch

LLMs can be made entirely 4-bit—weights, activations, and KV cache—without any high-precision channels, yet preserving 99% of zero-shot accuracy. The trick is to randomly rotate the activations offline, which mathematically leaves the model's output unchanged but completely eliminates the destructive outlier features.

1. Executive Summary

This paper introduces QuaRot, a quantization scheme that applies randomized Hadamard rotations to eliminate outlier features from LLM hidden states without altering model output, enabling end-to-end 4-bit quantization of all weights, activations, and KV caches. The method is evaluated on the LLAMA-2 family using WikiText-2 perplexity and six zero-shot tasks. QuaRot operates through two complementary mechanisms: computational invariance—fusing orthogonal transformations into weight matrices offline to remove outliers without additional runtime cost—and online Hadamard rotations inserted at specific points (the feed-forward down-projection, the attention value projection, and the key/value heads) to maintain outlier-free activations throughout the forward pass. Quantizing LLAMA2-70B to 4 bits with QuaRot preserves 99% of the zero-shot performance of the FP16 baseline while achieving up to 3.33× prefill speedup and 3.89× memory saving during decoding, establishing that 4-bit end-to-end inference is attainable without reserving any high-precision channels only when both the residual stream and the attention mechanism undergo incoherence processing.

2. Context and Motivation

The Core Problem: Activation Outliers Make Uniform Low-Bit Quantization Impossible

The fundamental obstacle this paper tackles is deceptively simple: LLM activations contain massive outlier values that make low-bit quantization extremely destructive. Figure 1 (left) illustrates this vividly: in the tenth layer of LLAMA2-7B, the distribution of activations at the input to the feed-forward network spans several orders of magnitude, with a small number of elements having values 10–100× larger than the typical activation. This phenomenon is well-documented across model families and layers.

Why does this matter for quantization? In any uniform quantization scheme, you must choose a single scale factor that maps the full range of values to the integer grid. If 99.9% of values fall within a small range but 0.1% are massive outliers, you face an impossible choice: either (a) clip the outliers, losing their information and causing large reconstruction errors, or (b) scale to accommodate them, which means the vast majority of normal values get squeezed into just 1–2 quantization bins, losing essentially all precision. Either way, the signal-to-noise ratio of the quantized representation collapses.

This is not a minor implementation detail — it is the central reason why activation quantization has lagged behind weight quantization. Weights are relatively well-behaved: they can be made approximately Gaussian with per-channel scaling, and methods like GPTQ (Frantar et al., 2022) can achieve good 4-bit weight-only quantization with minimal perplexity loss. But the moment you try to quantize the activations during inference so that the actual matrix multiplications can be done in low precision (what the field calls joint quantization or end-to-end quantization), the outliers destroy the computation. Prior to this work, no method had achieved end-to-end 4-bit inference (weights, activations, and KV cache all in 4 bits) on LLMs at scale while maintaining competitive accuracy, and the outlier problem was the primary barrier.

Why End-to-End Quantization Matters Practically

The motivation for joint quantization goes beyond academic interest. The paper identifies two distinct deployment bottlenecks (Section 1):

Prefill stage is compute-bound. When processing a long prompt (the "prefill" phase), the model must perform large matrix multiplications between the input activations and every weight matrix across all layers. These operations are arithmetic-intensive, and the GPU's tensor cores are the limiting resource. Quantizing both the weights and the activations to 4 bits means the core matrix-multiply operation (GEMM) can be performed using 4-bit integer arithmetic on tensor cores — which can execute 4-bit operations at substantially higher throughput than 16-bit floating point operations on the same hardware. But this speedup is only realizable if the activations entering each layer are quantized online during inference. If activations must remain in FP16 because of outliers, the theoretical throughput advantages of 4-bit weights cannot be fully realized — you still need mixed-precision operations.

Decoding stage is memory-bound. After the prompt is processed, generating tokens one-by-one (the "decoding" phase) has a fundamentally different bottleneck: the KV cache. For each new token, the model must attend over all previous tokens' key and value vectors. For long sequences or large batch sizes, this KV cache can be gigabytes in size. Every attention computation requires reading keys and values from memory, and memory bandwidth — not compute — is the bottleneck. Quantizing the KV cache to 4 bits reduces the memory footprint by up to 4×, directly reducing the I/O cost of attention. However, like activations, the key and value vectors also exhibit outliers (as documented by Hooper et al., 2024 and Liu et al., 2024), making uniform quantization of the KV cache just as problematic.

These two bottlenecks are complementary and demand a unified solution: you need both activation quantization (to speed up the prefill GEMM) and KV cache quantization (to reduce memory pressure during decoding). And both are blocked by the same underlying phenomenon: outliers.

Where Prior Approaches Fall Short

The paper provides a detailed taxonomy of existing work and identifiesspecific shortcomings in each category (Section 2):

Weight-only quantization methods (GPTQ, AWQ, SPQR, QuIP, QuIP#). These approaches (Frantar et al., 2022; Lin et al., 2023; Dettmers et al., 2023; Chee et al., 2024; Tseng et al., 2024) quantize only the model parameters, keeping activations in FP16 during inference. The main computation remains in high precision: you dequantize each weight on-the-fly before the matrix multiply, or you compute in high precision with quantized weights. This provides memory savings for storing the model, but does not accelerate the actual computation and does not address KV cache memory pressure. The paper acknowledges that several of these methods—particularly QuIP and QuIP# (Chee et al., 2024; Tseng et al., 2024)—introduced the idea of using rotations (orthogonal transforms) to improve weight quantization by reducing incoherence (making weight matrices more uniform). QuaRot builds directly on this insight but extends it to activations and the KV cache.

8-bit activation quantization (LLM.int8(), SmoothQuant). The first generation of activation quantization methods targeted 8-bit precision, where the outlier problem is less severe. LLM.int8() (Dettmers et al., 2022) identifies outlier features during inference and keeps them in 16 bits while quantizing the rest — a mixed-precision approach that results in poor performance because the sparse outlier-handling code breaks the regular computation pattern that GPUs are optimized for. SmoothQuant (Xiao et al., 2023) takes a different approach: it uses a calibration set to compute per-channel scaling factors that smooth the activation magnitudes, effectively transferring the quantization difficulty from activations to weights (where it's easier to handle). This works for 8-bit but requires careful hyperparameter tuning (the smoothing factor α) and the authors show it completely collapses at 4 bits — on LLAMA2-7B, SmoothQuant in 4 bits achieves 83.12 WikiText-2 perplexity (Table 1), which is essentially useless compared to the 5.47 baseline.

4-bit methods that keep some features in high precision (Atom, QUIK). Recognizing that 8-bit methods don't scale to 4-bit, recent work has retreated to partial high-precision strategies. Atom (Zhao et al., 2023) identifies outlier features offline (using a calibration set) and develops complex custom kernels for mixed-precision matrix multiplication where some channels are kept in higher precision. QUIK (Ashkboos et al., 2023) keeps the down-projection layer in 8 bits while quantizing everything else to 4 bits. These approaches admit defeat on the fundamental problem: they cannot achieve uniform 4-bit quantization. They require identifying and treating specific channels as "special," which leads to (a) complex, irregular computation patterns that are hard to optimize in hardware, (b) hyperparameters that need tuning (which channels to keep in high precision? how many?), and (c) higher memory usage than a truly uniform 4-bit scheme because the high-precision channels must be stored separately.

KV cache quantization (FlexGen, KVQuant, KIVI). There is a parallel line of work focused specifically on compressing the KV cache. Sheng et al. (2023) use 4-bit group-wise quantization. KVQuant (Hooper et al., 2024) pushes to 3 bits using feature-wise quantization and non-uniform representations. KIVI (Liu et al., 2024) achieves 2-bit KV cache quantization. These methods document that outliers also exist in the keys (not just the residual stream activations) and employ increasingly elaborate techniques to handle them—per-channel quantization, non-uniform codebooks, keeping high-precision outliers, and pre-RoPE caching (storing keys before the rotary positional embedding is applied, which changes the distribution). While effective for the KV cache specifically, these methods don't address the activation quantization problem, and their complexity (multiple bit-widths, non-uniform formats, special outlier handling) makes them challenging to implement efficiently in hardware kernels.

Rotations for quantization (QuIP, QuIP#). The closest intellectual precursors to QuaRot are QuIP (Chee et al., 2024) and QuIP# (Tseng et al., 2024), which introduced the idea of incoherence processing: multiplying weight matrices on the left and right by random orthogonal matrices to make the weight values more uniform and thus easier to quantize. The key insight is that a weight matrix W can be replaced by $Q_1 W Q_2^T$ where $Q_1$ and $Q_2$ are orthogonal, and if the rotation is applied correctly, the output of the layer is unchanged (because applying $Q_2$ to the input and $Q_1^T$ to the output of the next layer cancels the effect). However, in QuIP/QuIP# the rotations are undone at runtime — during the forward pass, you multiply the input by $Q_2$ before the linear layer and multiply the output by $Q_1^T$ after it, meaning you still need two additional matrix-vector products per layer. These extra operations add computational overhead. More critically, QuIP/QuIP# only apply rotations to the weights — they do not address activation quantization (beyond how weight rotations indirectly affect activation distributions) and do not touch the KV cache or attention module at all.

How QuaRot Positions Itself

QuaRot's central claim is that all of the above methods are fighting symptoms rather than the root cause. The root cause is that the hidden state of an LLM—the activations flowing between transformer blocks through the residual stream—has a structure that concentrates information in a small number of dimensions, producing outliers. Rather than working around these outliers (by keeping them in high precision, smoothing them, or using non-uniform quantization), QuaRot eliminates them entirely by rotating the representation.

The intellectual foundation is the computational invariance theorem from SliceGPT (Ashkboos et al., 2024), which proved that you can multiply the hidden state of a transformer by an orthogonal matrix Q and fuse $Q^T$ into the subsequent weight matrices without changing the model's output, so long as the normalization layer (RMSNorm) doesn't rescale the data. QuaRot extends this idea in several crucial ways beyond SliceGPT's original application (which was structured pruning, not quantization):

It uses Hadamard matrices specifically, chosen for their fast transform ( $O(d \log d)$ rather than $O(d^2)$ ), their orthogonality, and their proven effectiveness at reducing incoherence (making matrices more uniform). QuaRot needs only 1.5 Hadamard transforms per transformer layer (the down-projection and the attention output projection), compared to two per weight matrix in QuIP#.
It applies rotations beyond the residual stream. QuaRot recognizes that outliers exist within blocks as well as between them — in the feed-forward network's intermediate activations and in the attention mechanism. The paper inserts online Hadamard transforms at three strategic points: before the FFN down-projection, before the attention value projection (head-wise), and before the key/query computation (also head-wise). Each of these is fused into weight matrices where possible, minimizing runtime overhead.
It addresses the KV cache explicitly. Previous rotation-based quantization methods ignored attention, meaning the KV cache — a major memory bottleneck — remained unquantized or required separate complex handling. QuaRot applies head-wise Hadamard rotations to both keys and values before they enter the cache, making their distributions uniform and enabling simple uniform 4-bit quantization. The paper uses post-RoPE caching (rotating after positional encoding is applied) to avoid the overhead of reverse-rotating at every decoding step.
It is a complete system. QuaRot is not just a theoretical demonstration — it includes CUDA kernel implementations using CUTLASS for 4-bit GEMM on tensor cores, FlashInfer integration for quantized attention, and full integration with Hugging Face models. The paper reports actual speedup numbers (3.33× prefill speedup on LLAMA2-70B) rather than theoretical FLOP counts.

The paper frames its contribution in direct contrast to the prior state-of-the-art (Table 1): where previous 4-bit methods either collapsed (SmoothQuant: 83.12 perplexity), required high-precision channels (QUIK: 256 outlier features kept in FP16), or were never shown to work end-to-end with quantized activations (Atom: requires complex mixed-precision kernels), QuaRot achieves 6.10 WikiText-2 perplexity on LLAMA2-7B (vs. 5.47 baseline) with zero high-precision channels and uniform quantization throughout. This is the first demonstration that true end-to-end 4-bit LLM inference is possible without any features identified and retained in higher precision.

A subtle but important aspect of the positioning: QuaRot does not introduce a new quantization algorithm. The actual quantization is done with standard GPTQ (for weights) and round-to-nearest (for activations, with a simple clipping ratio). The entire contribution is in transforming the model so that standard quantization methods work well. This is philosophically different from work like SmoothQuant or Atom that develop increasingly sophisticated quantization techniques to handle difficult distributions. QuaRot makes the distributions easy first, then applies the simplest possible quantization.

3. Technical Approach

3.1 Reader Orientation

QuaRot is a model transformation pipeline — a sequence of offline weight modifications and online computation insertions — that takes a pretrained LLM and produces an arithmetically equivalent model whose activations, weights, and KV cache all have uniform, outlier-free distributions suitable for simple 4-bit quantization. The system solves the problem of activation outliers blocking end-to-end low-bit inference by rotating the representation at every point where information flows between or within transformer blocks, using orthogonal (specifically Hadamard) matrices that spread outlier values across all dimensions without changing what the model computes.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Weight Modifier (Stage 1a) — operates offline. Absorbs RMSNorm scaling parameters into adjacent weight matrices, then fuses randomized Hadamard matrices into every weight matrix in the model using the computational invariance property. This is done once before deployment and has zero runtime cost.
Online Hadamard Transform (FFN down-projection) — inserted into the forward pass at runtime. Applies a fast Walsh-Hadamard transform to the activations between the gating/up-projection and the down-projection in each feed-forward block. This is fused into the down-projection weight matrix (half the transform is done offline in the weights, half online on the activations).
Online Hadamard Transform (Attention Value Projection) — inserted into the forward pass at runtime. Applies head-wise Hadamard rotations to the value vectors before they enter the attention computation, with the inverse fused into the output projection weights. Also applies a cross-head Hadamard transform after attention to fully spread information across heads.
Online Hadamard Transform (Key/Query Rotation) — inserted into the forward pass at runtime. Applies head-wise Hadamard rotations to both query and key vectors after positional encoding (RoPE) is applied. Since the rotations cancel in the attention score computation, this is arithmetically a no-op but makes the KV cache distribution uniform for quantization.
Quantization Operations (Stage 2) — applied after the model is transformed. Weights are quantized offline using GPTQ (or optionally round-to-nearest). Activations are quantized online using symmetric per-token quantization with a constant clipping ratio. The KV cache is quantized online using asymmetric per-group quantization.

Information flows as follows: input tokens enter the first transformer block → RMSNorm is applied (without scaling) → activations are quantized to INT4 → the first weight matrices (already rotated and quantized) perform INT4 matrix multiplication → results are cast back to FP16 → inside the FFN, a Hadamard transform is applied → activations are quantized again → the down-projection performs INT4 matmul → inside the attention module, queries and keys are rotated head-wise after RoPE → values are rotated head-wise → attention scores are computed (rotations cancel) → a cross-head Hadamard transform is applied → the output projection performs INT4 matmul → quantized KV vectors are stored in the cache → the output (rotated) hidden state passes to the next block.

3.3 Roadmap for the Deep Dive

First, the computational invariance theorem and its application to between-block rotations (Stage 1a): how Q is chosen, fused into weights, and why this eliminates outliers in the residual stream without changing model output.
Second, the FFN activation rotation (Stage 1b): why outliers exist within the FFN block, where the Hadamard transform is inserted, and how the inverse is fused into the down-projection weights.
Third, the attention value projection rotation (Stage 1c): the implicit $W_v W_{out}$ product structure, how head-wise rotations are applied and cancelled, and the cross-head Hadamard transform that completes the rotation.
Fourth, the key/query rotation (Stage 1d): why keys have outliers, the interaction with RoPE, the choice of post-RoPE caching, and how the rotation cancels in attention scores.
Fifth, the quantization scheme (Stage 2): weight quantization via GPTQ with per-column symmetric quantization, activation quantization via per-token symmetric quantization with constant clipping, and KV cache quantization via asymmetric per-group quantization.
Sixth, the kernel implementation and the specific CUTLASS/FlashInfer integration that realizes the speedups on GPU hardware.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that orthogonal (Hadamard) rotations can be applied to every component of a transformer — between blocks, within the FFN, and within attention — to make all activations and KV cache entries uniformly distributed, and that the inverse rotations can be fused into weight matrices so that the model's output is arithmetically unchanged while the distributions become quantization-friendly.

Computational Invariance and Between-Block Rotation (Stage 1a)

The foundational mechanism enabling QuaRot is the computational invariance theorem from SliceGPT (Ashkboos et al., 2024). The theorem states that in a pre-norm transformer, you can multiply the hidden state (the activations flowing between blocks through the residual stream) by an orthogonal matrix $Q$ , and cancel out this transformation by multiplying the weight matrices that consume this hidden state by $Q^T$ on the appropriate side. The model's output is preserved exactly (up to floating-point precision), but the representation is rotated.

Why does this work through a normalization layer? The key is how RMSNorm operates. RMSNorm computes:

$x_i \leftarrow \frac{x_i}{\|x\|}$

for each row of the activations, where $\|x\|$ is the $\ell_2$ norm of the row vector. If you apply an orthogonal matrix $Q$ to the activations before RMSNorm, the norm is unchanged because orthogonal matrices preserve $\ell_2$ norms: $\|xQ^T\| = \|x\|$ . This gives the commutation property:

$\text{RMSNorm}(X) = \text{RMSNorm}(XQ^T) Q$

where $X$ is the activation matrix (rows are tokens, columns are features), $\text{RMSNorm}$ divides each row by its norm, and $Q$ is an orthogonal matrix. In operational terms: normalizing first and then rotating gives the same result as rotating first, normalizing, and then rotating back. This means that if you multiply the output of a block by $Q^T$ (so the hidden state becomes $XQ^T$ ), the RMSNorm at the start of the next block will produce $\text{RMSNorm}(XQ^T)$ , and if the first weight matrix of that next block is multiplied by $Q$ on the input side, the linear layer computes $\text{RMSNorm}(XQ^T) \cdot (QW) = \text{RMSNorm}(X) \cdot W$ , which is exactly what the original model would have computed.

The procedure in QuaRot (Stage 1a) — offline, no runtime cost:

First, the scaling factors from RMSNorm (the learnable parameters $\alpha$ that multiply each normalized feature) are absorbed into the adjacent weight matrices. RMSNorm originally computes:

$y_i = \alpha_i \cdot \frac{x_i}{\|x\|}$

By absorbing $\alpha$ into the weights, RMSNorm becomes a pure normalization operation (division by norm only, no learned scaling). Specifically, if a weight matrix $W$ immediately follows RMSNorm in the forward pass, QuaRot replaces $W \leftarrow \text{diag}(\alpha) W$ , and RMSNorm drops the scaling step. This is crucial because the computational invariance argument requires that no rescaling happens in the normalization — the norm must be the only operation so that $Q$ and $Q^T$ can cancel cleanly.

Second, a randomized Hadamard matrix $Q$ of size $d \times d$ is chosen, where $d$ is the hidden dimension of the model. The matrix is constructed as:

$\tilde{H} = H \cdot \text{diag}(s)$

where $H$ is a Walsh-Hadamard matrix (a structured orthogonal matrix with entries in $\{+1, -1\}$ scaled by $1/\sqrt{d}$ ), and $s$ is a vector of random signs drawn from $\{+1, -1\}$ . The randomization via $\text{diag}(s)$ is important: a pure Hadamard matrix has a fixed structure that might not distribute all outlier directions uniformly (some directions could align with Hadamard rows by chance). Randomizing the signs ensures that any systematic structure in the activations is scrambled.

When the hidden dimension $d$ is not a power of 2 (which is required for Walsh-Hadamard matrices), the paper factorizes $d = 2^n m$ where $m$ is the size of a known Hadamard matrix (using the library from Sloane, 2024), and uses a Kronecker construction $H_d = H_{2^n} \otimes H_m$ . The transform $H_d x$ can then be computed in $O(d(m + n))$ operations — still fast, but the paper doesn't specify exact overhead for non-power-of-2 cases.

Third, $Q$ is fused into the weight matrices. For every weight matrix that consumes the hidden state (appears on the left side of a transformer block — the input side), $Q^T$ is pre-multiplied:

$W \leftarrow Q^T W$

For example, the key projection weight matrix $W_k$ becomes $W_k \leftarrow Q^T \text{diag}(\alpha) W_k$ , where $\text{diag}(\alpha)$ is the absorbed RMSNorm scaling.

For every weight matrix that produces the hidden state (appears on the right side of a block — the output side), $Q$ is post-multiplied:

$W \leftarrow W Q$

This means the following weight matrices receive modifications in a typical LLAMA-style transformer:

Input to the block (gate, up, query, key, value projections): $W \leftarrow Q^T \text{diag}(\alpha) W$
Output of the block (down-projection in FFN, output projection in attention): $W \leftarrow W Q$

The net effect: the hidden state flowing from block to block is now $X Q$ instead of $X$ . When this rotated hidden state enters the next block's RMSNorm, its norm is unchanged (orthogonal matrices preserve $\ell_2$ ). The subsequent weight matrices have been pre-multiplied by $Q^T$ , which cancels the rotation. Information is preserved exactly.

Why this eliminates outliers: An outlier in the original hidden state is a dimension where a few tokens have values much larger than the rest of the distribution. When multiplied by $Q^T$ , the value in that outlier dimension is spread across all dimensions of the rotated state. Specifically, if the original activation vector for a token is $x \in \mathbb{R}^d$ with an outlier in dimension $i$ , the rotated vector $xQ$ has:

$(xQ)_j = \sum_{k=1}^d x_k Q_{k,j}$

The outlier's contribution ( $x_i Q_{i,j}$ ) is distributed across all $j$ according to the entries of the $i$ -th row of $Q$ . Since $Q$ is a randomized Hadamard matrix, its entries are $\pm 1/\sqrt{d}$ , so the outlier energy is divided approximately equally across all $d$ output dimensions. The result, visible in Figure 1 (right), is a distribution with no extreme values — a tight, approximately Gaussian distribution centered near zero.

Comparison to QuIP#: QuIP# (Tseng et al., 2024) also applies random orthogonal matrices to weight matrices to reduce incoherence, but it does so on both the left and right of each weight matrix independently, and the rotations must be undone at runtime (the input is multiplied by the right-rotation before each linear layer, and the output by the left-rotation after). QuaRot applies a single global rotation $Q$ to the hidden state, which cascades through the entire model. QuaRot needs only $1.5$ Hadamard transforms per transformer layer at runtime (down-projection + attention output), compared to $2$ per weight matrix in QuIP#. The difference arises because QuaRot relies on computational invariance — the same $Q$ propagates through RMSNorm — while QuIP# treats each weight matrix independently.

FFN Activation Rotation (Stage 1b)

The between-block rotation (Stage 1a) makes the residual stream activations (the inputs and outputs of each transformer block) outlier-free. However, outliers also exist within each block — specifically, in the intermediate activations of the feed-forward network. In a gated FFN (Figure 2), the computation is:

$A = X \cdot W_{\text{gate}}, \quad B = X \cdot W_{\text{up}}$ $C = \sigma(A) \odot B$ $Y = C \cdot W_{\text{down}}$

where $X$ is the input (already rotated by $Q$ from Stage 1a, so it's $XQ$ in the actual forward pass), $\sigma$ is an activation function (SiLU in LLAMA-2), $\odot$ is element-wise multiplication, and $W_{\text{down}}$ is the down-projection that produces the output. The problem: after the gating and up-projection and the element-wise multiplication, the intermediate activations $C$ can develop outlier patterns again, even if $XQ$ is well-behaved. This is because the gating mechanism ( $\sigma(A) \odot B$ ) can selectively amplify certain dimensions.

To fix this, QuaRot inserts a Hadamard transform $H$ between the element-wise multiplication and the down-projection, and fuses the inverse $H^T = H$ (Hadamard matrices are symmetric) into the down-projection weights:

$W_{\text{down}} \leftarrow H W_{\text{down}} Q$

The factorization: $W_{\text{down}} Q$ is the global rotation from Stage 1a (post-multiplied because $W_{\text{down}}$ produces the hidden state). The additional $H$ pre-multiplication further rotates the input side of the down-projection. At runtime, the forward pass becomes:

Compute $A = (XQ) \cdot (Q^T \text{diag}(\alpha) W_{\text{gate}})$ — the $Q$ and $Q^T$ cancel, producing the original $A$ .
Compute $B = (XQ) \cdot (Q^T \text{diag}(\alpha) W_{\text{up}})$ — similarly cancels.
Compute $C = \sigma(A) \odot B$ — same as original.
Apply online Hadamard transform: $C' = C H$ — this is computed at runtime using a fast Walsh-Hadamard kernel in FP32 or FP16.
Compute output: $Y = C' \cdot (H W_{\text{down}} Q) = C H H W_{\text{down}} Q = C W_{\text{down}} Q$ — the $H$ and $H$ cancel because Hadamard matrices are orthogonal and symmetric ( $HH = I$ ).

The result $Y = (C W_{\text{down}}) Q$ is exactly the original output rotated by $Q$ , which is what the next block expects (since its input weights have been multiplied by $Q^T$ ).

Why insert the Hadamard here specifically: The intermediate activations $C$ are the last point in the FFN where outlier patterns can emerge before the down-projection. By rotating $C$ with $H$ , the distribution of values entering the quantized down-projection matmul is uniform. Without this rotation, $C$ would have outliers, and the 4-bit quantization of $C$ (which happens right before the matmul) would be destructive. The $H$ is fused into $W_{\text{down}}$ offline, so only the forward transform $C H$ is computed at runtime — a single fast operation that costs roughly 5–7% overhead on the down-projection matmul (Table 14: the INT4 + Hadamard column is at most 7% slower than INT4 alone).

Why a Hadamard matrix rather than a random rotation: Hadamard matrices have $O(d \log d)$ transform cost via the Fast Walsh-Hadamard Transform (FWHT), compared to $O(d^2)$ for a general dense orthogonal matrix. They are also exactly orthogonal (no numerical drift), and their entries are all $\pm 1/\sqrt{d}$ , which is the ideal distribution for spreading outlier energy uniformly. A random orthogonal matrix (e.g., QR decomposition of a random Gaussian matrix) would cost $O(d^2)$ at runtime and need to be stored explicitly ( $d^2$ parameters). The paper's ablation (Appendix A.5, Table 8) confirms that random orthogonal matrices are worse than Hadamard: LLAMA2-7B perplexity is 7.45 with random orthogonal vs. 6.10 with Hadamard, though the gap shrinks for larger models (only 0.28 perplexity difference on LLAMA2-70B).

Attention Value Projection Rotation (Stage 1c)

The attention mechanism presents a new challenge: unlike the FFN where all dimensions mix freely, attention operates head-wise. Each head has its own key, query, and value projections, and the attention computation mixes information only within each head (via the attention scores) before concatenating across heads for the output projection. An outlier in the value vectors of one head cannot be spread to other heads by simple global rotation — it's trapped within that head. So QuaRot needs to apply rotations within each head and then across heads.

The key observation is that $W_v$ (value projection) and $W_{out}$ (output projection) are implicitly multiplied together within each head in the attention computation. The attention output for all heads is:

$Y = \text{concat}[P_1 V_1, \ldots, P_{n_h} V_{n_h}] W_{out}$

$= \sum_{h=1}^{n_h} P_h X W_v^{(h)} W_{out}^{(h)}$

where:

$P_h \in \mathbb{R}^{s \times s}$ is the attention score matrix for head $h$ (rows sum to 1, computed from queries and keys via softmax), where $s$ is the sequence length,
$V_h = X W_v^{(h)}$ is the value matrix for head $h$ (size $s \times d_h$ , where $d_h$ is the head dimension),
$W_v^{(h)} \in \mathbb{R}^{d \times d_h}$ is the value projection for head $h$ (a slice of the full $W_v$ matrix),
$W_{out}^{(h)} \in \mathbb{R}^{d_h \times d}$ is the output projection for head $h$ (a slice of $W_{out}$ ).

The product $W_v^{(h)} W_{out}^{(h)}$ is a $d \times d$ matrix that maps from the hidden state through the values and back. This product can absorb a head-wise rotation: if we multiply $W_v^{(h)}$ on the right by a $d_h \times d_h$ Hadamard matrix $H_{d_h}$ and multiply $W_{out}^{(h)}$ on the left by the same $H_{d_h}$ , the product $W_v^{(h)} H_{d_h} H_{d_h} W_{out}^{(h)} = W_v^{(h)} W_{out}^{(h)}$ is unchanged because $H_{d_h} H_{d_h} = I$ .

The offline weight modification: Since the weights for all heads are stored concatenated in memory ( $W_v = [W_v^{(1)} | \ldots | W_v^{(n_h)}]$ and similarly for $W_{out}$ ), this head-wise rotation is equivalent to:

$W_v \leftarrow W_v \cdot (I_{n_h} \otimes H_{d_h})$ $W_{out} \leftarrow (I_{n_h} \otimes H_{d_h}) \cdot W_{out}$

where $I_{n_h}$ is the $n_h \times n_h$ identity matrix, $\otimes$ is the Kronecker product, and $(I_{n_h} \otimes H_{d_h})$ is a block-diagonal matrix where each $d_h \times d_h$ block is $H_{d_h}$ and blocks are arranged along the diagonal. In concrete terms: the columns of $W_v$ corresponding to each head are right-multiplied by $H_{d_h}$ , and the rows of $W_{out}$ corresponding to each head are left-multiplied by $H_{d_h}$ . This is done offline with zero runtime cost.

The online Hadamard heads transform: The above modification ensures that the value vectors emitted by the value projection are head-wise rotated: $V_h' = X W_v^{(h)} H_{d_h}$ . But to achieve a full Hadamard rotation across all $n_h d_h = d$ dimensions of the concatenated value vectors — sharing information across heads — QuaRot uses the identity:

$H_{n_h d_h} = (I_{n_h} \otimes H_{d_h}) (H_{n_h} \otimes I_{d_h})$

This identity holds when both $n_h$ and $d_h$ are powers of 2 (which is true for all LLAMA-2 model sizes: e.g., LLAMA2-7B has $n_h = 32$ , $d_h = 128$ ). The proof follows from the mixed-product property of the Kronecker product. It means that a full Hadamard transform on all $d$ dimensions can be factored into: (1) a head-wise Hadamard $I_{n_h} \otimes H_{d_h}$ (already fused into weights), followed by (2) a cross-head Hadamard $H_{n_h} \otimes I_{d_h}$ , which must be applied online.

Since $I_{n_h} \otimes H_{d_h}$ is already applied to $W_v$ and $W_{out}$ offline, QuaRot additionally applies $H_{n_h} \otimes I_{d_h}$ to $W_{out}$ offline (replacing it in the concatenated weight matrix) and inserts an online block after the attention computation that computes:

$Z \leftarrow Z \cdot (H_{n_h} \otimes I_{d_h})$

where $Z$ is the concatenated attention output (before the output projection). This cross-head transform can be implemented efficiently: reshape $Z$ from shape $(s, n_h \cdot d_h)$ to $(s \cdot d_h, n_h)$ , apply a Walsh-Hadamard transform of size $n_h$ to each row (which mixes information across heads while keeping within-position dimension intact), then reshape back. The cost is $O(s \cdot d_h \cdot n_h \log n_h) = O(s \cdot d \log n_h)$ .

What this achieves: Without these rotations, the value vectors within each head could develop outliers, and the concatenated values would have a structured sparsity pattern (some heads dominating). After the head-wise and cross-head rotations, the value vectors are uniformly distributed across all dimensions, and the KV cache (which stores these value vectors) can be quantized uniformly.

Key and Query Rotation (Stage 1d)

Keys in the attention mechanism present a distinct challenge from values because of the role of positional encoding. In LLAMA-2, this is Rotary Position Embedding (RoPE) (Su et al., 2021), which applies a position-dependent rotation to the query and key vectors within each head before the attention score computation.

The attention scores for head $h$ are computed as:

$Q_h = \text{RoPE}(X W_q^{(h)})$ $K_h = \text{RoPE}(X W_k^{(h)})$ $P_h = \text{Softmax}\left(\frac{Q_h K_h^T}{\sqrt{d_h}} \odot M\right)$

where RoPE applies a $d_h \times d_h$ block-diagonal rotation matrix (with $2 \times 2$ rotation blocks) parameterized by the position index, and $M$ is the attention mask (e.g., causal).

The crucial observation is the same as for values: $Q_h$ and $K_h$ appear in a product $Q_h K_h^T$ , so a rotation applied to both will cancel. Specifically, if we rotate $Q_h \leftarrow Q_h H_{d_h}$ and $K_h \leftarrow K_h H_{d_h}$ , then:

$(Q_h H_{d_h}) (K_h H_{d_h})^T = Q_h H_{d_h} H_{d_h}^T K_h^T = Q_h K_h^T$

because $H_{d_h}$ is orthogonal ( $H_{d_h} H_{d_h}^T = I$ ). The attention scores are unchanged.

The challenge with RoPE: Unlike the value projection where the rotation can be fused into the weights offline, RoPE is applied after the linear projection but before the attention score computation. If we tried to fuse $H_{d_h}$ into $W_q$ and $W_k$ , we would have:

$Q_h' = \text{RoPE}(X W_q^{(h)} H_{d_h})$

But RoPE is not linear — $\text{RoPE}(X W_q^{(h)} H_{d_h}) \neq \text{RoPE}(X W_q^{(h)}) H_{d_h}$ in general. The rotation and the positional encoding do not commute. So we cannot simply fuse the Hadamard into $W_q$ and $W_k$ and expect the attention scores to be preserved.

QuaRot's solution: online head-wise rotation after RoPE. QuaRot applies the Hadamard rotation to queries and keys after RoPE has been applied, at runtime:

$Q \leftarrow \text{RoPE}(X W_q) \cdot (I_{n_h} \otimes H_{d_h})$ $K \leftarrow \text{RoPE}(X W_k) \cdot (I_{n_h} \otimes H_{d_h})$

This means both queries and keys are rotated head-wise after positional encoding. Since both are rotated by the same matrix, the attention scores (which compute $Q_h K_h^T$ ) are unchanged — the $(I_{n_h} \otimes H_{d_h})$ and its transpose cancel.

The runtime cost: For each token during prefill and decoding, QuaRot must:

Compute the query and key vectors as usual (FP16 matmul + RoPE).
Apply the head-wise Hadamard transform $I_{n_h} \otimes H_{d_h}$ to the queries and keys.
The rotated keys (and values, from Stage 1c) are then quantized and stored in the KV cache.

For decoding (single token generation), this means applying a Hadamard transform of size $d_h$ to each of the $n_h$ heads for the single query vector — $n_h$ small Hadamard transforms. For the key vector (which is cached), the rotation happens once when the key is generated and then the rotated, quantized key is stored. This cost is amortized over all subsequent decoding steps.

Post-RoPE vs. Pre-RoPE caching: The paper explicitly chooses post-RoPE caching (rotate after RoPE, then cache). The alternative (used by KVQuant, Hooper et al., 2024) is pre-RoPE caching: cache the keys before RoPE is applied, store them in the original space, and apply RoPE + rotation on-the-fly during decoding. Pre-RoPE caching has the advantage that the query doesn't need to be rotated (since the key rotation can be undone when loading from cache). However, pre-RoPE caching has a critical disadvantage: at each decoding step, the model must apply RoPE + inverse rotation to every cached key vector (sequence length of them) for each new query. This adds overhead proportional to sequence length. With post-RoPE caching, only the single new query needs rotation at each step — a constant cost per token. The paper argues this is more efficient for the decoding phase where there is one query and many cached keys.

Why keys need rotation: Keys, like activations and values, exhibit outliers (documented by Hooper et al., 2024; Liu et al., 2024). After RoPE, the positional encoding can amplify certain frequency components, creating large values in specific dimensions. The head-wise Hadamard rotation spreads these across the head dimension, making the key distribution uniform and suitable for uniform 4-bit quantization in the KV cache.

Quantization Scheme (Stage 2)

After the model is transformed (all rotations applied, all weights modified), QuaRot applies three distinct quantization procedures: one for weights, one for activations, and one for the KV cache. Each uses different quantization parameters chosen for their specific distribution characteristics and access patterns.

Weight Quantization (Stage 2a) — offline, per-column symmetric:

Weights are quantized offline using GPTQ (Frantar et al., 2022) as the default method, with an option for simple round-to-nearest (RTN). The quantization is per-column symmetric: each column (output dimension) of the weight matrix gets its own scale factor, computed as:

$s_j = \frac{\max(|W_{:,j}|)}{2^{b-1} - 1}$

where $b = 4$ (4-bit), so the divisor is 7 (largest representable INT4 value). The quantized weight is:

$\hat{W}_{i,j} = \text{round}\left(\frac{\text{clip}(W_{i,j}, -\alpha_j, \alpha_j)}{s_j}\right)$

where the clipping threshold $\alpha_j$ is chosen per-column by a linear search over squared error on a calibration set (128 samples from WikiText-2 training set, 2048 sequence length). For GPTQ, the quantization is applied column-by-column with Hessian-based compensation: after quantizing one column, the error is distributed to the remaining unquantized columns using the inverse Hessian of the calibration data, which minimizes the output error rather than the weight error.

The choice of per-column (rather than per-tensor or per-row) is standard: weight columns correspond to different output features, and their magnitude ranges can vary substantially. A single scale per column gives each output neuron appropriate dynamic range.

Why GPTQ rather than just RTN? RTN quantization (simple rounding with per-column scales and a fixed clipping ratio) is simpler and requires no calibration data, but it's less accurate because it doesn't account for interactions between columns. Table 3 shows the gap: on LLAMA2-7B, RTN gives 8.37 WikiText-2 perplexity vs. 6.10 for GPTQ (a 2.27 point gap). However, the gap shrinks with model size: on LLAMA2-70B, RTN gives 4.14 vs. 3.79 for GPTQ (only 0.35 point gap). For 6-bit and 8-bit, RTN is essentially lossless (Table 9: 8-bit RTN on LLAMA2-70B gives 3.33 perplexity, identical to FP16 baseline). This suggests that larger models are more robust to quantization error, and simple RTN may suffice for sufficiently large models at moderate bit-widths.

Activation Quantization (Stage 2b) — online, per-token symmetric:

Activations are quantized online during inference, immediately before each matrix multiplication. The quantization is per-token symmetric: each row (token) of the activation matrix gets its own scale factor, computed from the maximum absolute value in that row:

$s_i = \frac{\max_j(|X_{i,j}|)}{7}$

with a constant clipping ratio of 0.9 applied before computing the maximum. Specifically, the actual scale is:

$s_i = \frac{0.9 \cdot \max_j(|X_{i,j}|)}{7}$

The constant 0.9 was chosen from an ablation (Appendix A.2, Table 5): on LLAMA2-7B with input quantization only, clipping ratios of 1.0, 0.95, 0.9, and 0.85 gave perplexities of 5.938, 5.910, 5.828, and 5.850 respectively. The value 0.9 is the best, and the results are not highly sensitive — the worst (1.0, no clipping) is only 0.11 perplexity worse than the best (0.9).

Why per-token and not per-tensor or per-channel? Per-tensor quantization (one scale for all tokens in a batch) fails because different tokens can have very different activation magnitudes (e.g., a token with an outlier feature vs. one without). Per-channel quantization (one scale per feature dimension) would require different scales for each output element, which is incompatible with efficient integer matrix multiplication — the GEMM kernel expects a single scale per row of the input. Per-token quantization is the natural trade-off: it handles token-to-token magnitude variation while being efficiently implementable in hardware (the scale is simply multiplied per-row in the dequantization step).

The dequantization step: After the INT4 GEMM computes the matrix product in an INT32 accumulator, QuaRot casts the result to FP16 and multiplies by the appropriate scales. Specifically, if the input activations $X$ are quantized with per-token scales $\text{scale}_x \in \mathbb{R}^{n \times 1}$ and the weights $W$ are quantized with per-column scales $\text{scale}_w \in \mathbb{R}^{1 \times m}$ , the dequantized output is:

$Y_{i,j} \leftarrow \text{cast\_to\_fp16}(\hat{Y}_{i,j}^{\text{INT32}}) \cdot \text{scale}_x[i] \cdot \text{scale}_w[j]$

The outer product of the two scale vectors gives the correct per-element dequantization factor.

KV Cache Quantization (Stage 2c) — online, per-group asymmetric:

The KV cache is quantized using asymmetric per-group quantization with a group size of 128. Unlike the symmetric quantization used for weights and activations, asymmetric quantization uses both a scale and a zero-point:

$\hat{v} = \text{round}\left(\frac{\text{clip}(v, \alpha_{\min}, \alpha_{\max}) - z}{s}\right)$

where $s = (\alpha_{\max} - \alpha_{\min}) / (2^b - 1)$ is the scale, $z = \text{round}(\alpha_{\min} / s)$ is the zero-point offset, and $\alpha_{\min}, \alpha_{\max}$ are determined with a clipping ratio of 0.95 (chosen from the ablation in Appendix A.2: values of 0.95 and 0.9 give nearly identical perplexity). Asymmetric quantization can represent distributions that are not centered at zero, which is important for the KV cache because after RoPE and the Hadamard rotations, the value distributions may not be perfectly zero-mean (activation functions and attention patterns can introduce biases).

The group size of 128 means that keys and values are divided into groups of 128 elements along the feature dimension, and each group gets its own scale and zero-point. Since the head dimension in LLAMA-2 models is exactly 128 (e.g., LLAMA2-7B has $d_h = 128$ ), this means each head's key and value vectors are quantized with a single group — effectively one scale per head position per token. For the KV cache specifically, the feature dimension within each head is exactly 128 in the models evaluated (Table 15 confirms head dimensions of 128 for all LLAMA-2 models), so group size 128 means per-head quantization.

Why asymmetric quantization for KV cache but symmetric for everything else? Symmetric quantization is simpler (no zero-point, easier dequantization) and works well when the distribution is approximately zero-mean, which is true for the rotated activations and weights after incoherence processing. The KV cache values, however, can accumulate biases from the attention mechanism (the weighted sum of values is not necessarily zero-mean) and from RoPE (which rotates vectors but doesn't guarantee zero-mean along each dimension). Asymmetric quantization handles these offsets naturally at the cost of storing an additional zero-point per group (typically 8 bits, negligible compared to the 4-bit values).

The KV cache operation during decoding: At each decoding step, the attention module:

Computes the query vector and rotates it head-wise (online, FP16).
Loads quantized key and value vectors from the cache for all previous positions.
Dequantizes each group on-the-fly: $\tilde{v} = s \cdot \hat{v} + z$ .
Computes the attention dot products in FP16 using the dequantized keys and values.
Appends the new quantized key and value to the cache.

The paper implements this using the FlashInfer library (Ye, 2023), which provides optimized kernels for quantized attention with on-the-fly dequantization.

Ablation on KV cache bit-width (Appendix A.3, Table 6): The paper explores various combinations of key and value bit-widths. With the rest of the model in FP16 (only KV cache quantized), the results show:

4-bit keys + 4-bit values: 5.51 perplexity on LLAMA2-7B (vs. 5.47 baseline — only 0.04 loss).
3-bit keys + 4-bit values: 5.65 perplexity (0.18 loss).
4-bit keys + 3-bit values: 5.54 perplexity (0.07 loss).
3-bit keys + 3-bit values: 5.68 perplexity (0.21 loss).

Keys are more sensitive to quantization than values: going from 4-bit to 3-bit on keys costs more perplexity than going from 4-bit to 3-bit on values. This matches findings from KVQuant and KIVI. The paper also shows 2-bit KV cache is feasible but with larger degradation (5.75 for 4-bit keys + 2-bit values, 8.06 for 2-bit keys + 4-bit values).

Kernel Implementation and Hardware Efficiency

QuaRot's speedups are realized through custom CUDA kernels for three operations: 4-bit matrix multiplication, online Hadamard transforms, and quantized attention.

4-bit matrix multiplication: The paper uses the CUTLASS library (NVIDIA, 2023) for INT4 GEMM on Tensor Cores. Tensor Cores on NVIDIA GPUs (Ampere architecture, RTX 3090) can perform $4 \times 4$ matrix multiplications in INT4 with INT32 accumulation at substantially higher throughput than FP16. The kernel flow:

Quantize the input activations to INT4 (per-token, symmetric, with the 0.9 clipping ratio).
Pack the INT4 values into sub-byte format (two INT4 values per byte).
Call CUTLASS INT4 GEMM with the packed activations and the offline-quantized (also packed) weights.
The accumulator is INT32.
Cast the INT32 result to FP16 and dequantize by multiplying with the per-token activation scales and per-column weight scales.

Figure 7 and Table 14 provide detailed benchmarks. For the down-projection layer in LLAMA2-7B (size $11008 \times 4096$ ), the FP16 matmul takes 2.569 ms, the INT4 matmul without Hadamard takes 0.749 ms (3.43× speedup), and INT4 with online FP32 Hadamard takes 0.798 ms (3.22× speedup including the transform overhead). With FP16 Hadamard, it's 0.801 ms. The Hadamard overhead is approximately 6–7% of the INT4 matmul time, which is small relative to the overall speedup.

Online Hadamard transforms: The Hadamard transforms are implemented using the Fast Walsh-Hadamard Transform (FWHT), which computes $Hx$ in $O(d \log d)$ time using a butterfly algorithm. Two variants are tested: FP32 precision (default, used for all accuracy experiments) and FP16 precision. Appendix A.7 (Table 10) shows that FP16 Hadamard transforms produce nearly identical accuracy: on LLAMA2-7B, WikiText-2 perplexity is 6.08 with FP16 vs. 6.10 with FP32, and zero-shot average accuracy is 66.21% vs. 65.64% (within noise). This suggests FP16 is sufficient and could provide additional speedup, though the paper's performance benchmarks use FP32 by default.

Quantized attention with KV cache: The attention kernel is implemented in three routines using FlashInfer:

Init (prefill): Computes attention for the full prompt using Flash Attention (Dao et al., 2022) with dequantized keys and values. During prefill, all key and value vectors are available at once, so they can be dequantized and processed in the standard Flash Attention kernel.
Append (decoding): Quantizes the new key and value vectors for the current token and appends them to the KV cache.
Decode (decoding): Given the query vector for the current token, loads the quantized cache for all previous tokens, dequantizes on-the-fly, computes attention dot products, and produces the attention output.

Table 15 benchmarks the decode routine. For LLAMA2-7B attention (32 heads, 128 dim), FP16 decoding of one token with 2047 cached tokens takes 0.713 ms at batch size 1. The INT4 version takes 1.033 ms — actually slower than FP16 at batch size 1 because the quantization/dequantization overhead dominates when memory I/O isn't the bottleneck. However, at batch size 64, the FP16 time grows to 2.098 ms while INT4 is 1.168 ms — a 1.80× speedup — because the larger batch size makes memory bandwidth the bottleneck and the 4× smaller KV cache reduces I/O. The paper notes that speedups are only visible for "larger batch sizes or longer sequences" where the I/O savings outweigh quantization overhead.

Why end-to-end INT4 is faster: The speedup comes from two sources:

During prefill: INT4 GEMM on Tensor Cores has higher throughput than FP16 GEMM (roughly 3–4× for the matrix sizes in LLAMA-2, as shown in Figure 7).
During decoding: the 4× smaller KV cache (4-bit vs. 16-bit) reduces memory bandwidth pressure, which is the bottleneck for the attention operation.

The critical enabler is that QuaRot makes quantization possible at all — without the Hadamard transformations, 4-bit activation quantization would cause catastrophic accuracy degradation (as shown by SmoothQuant's 83.12 perplexity at 4 bits in Table 1). The speedup numbers (up to 3.33× prefill on LLAMA2-70B, 3.89× memory saving) are the realized benefit of solving the outlier problem.

Summary of Design Choices and Their Justifications

Randomized Hadamard over pure Hadamard: Pure Hadamard matrices have a fixed structure; randomizing the signs (multiplying by $\text{diag}(s)$ ) ensures that any systematic alignment between the activation outlier directions and the Hadamard rows is broken. Without randomization, an outlier aligned with a Hadamard row could survive the rotation.
Post-RoPE rather than pre-RoPE KV cache: Post-RoPE caching means the rotation can be applied once when the key is generated rather than redone at every decoding step for all cached keys. The cost is that the query must also be rotated, but this is a single vector per step (constant cost) rather than proportional to sequence length.
GPTQ for weights, RTN for activations: GPTQ uses calibration data and Hessian-based error compensation to minimize output error from weight quantization, which is important because weight quantization error compounds across layers. Activations are quantized online and need to be fast, so simple RTN with a constant clipping ratio is chosen. The gap between GPTQ and RTN for weights shrinks with model size, suggesting RTN may be sufficient for very large models.
Per-column symmetric for weights, per-token symmetric for activations: Weights need per-column (output-channel) scaling because different output features have different magnitude ranges. Activations need per-token scaling because different tokens in a sequence can have very different activation magnitudes.
Asymmetric quantization for KV cache, symmetric for weights/activations: The KV cache values can have non-zero mean due to attention biases and RoPE, making symmetric quantization (which assumes the distribution is centered at zero) a poor fit. Asymmetric quantization adds a zero-point per group but handles offset distributions correctly.
Group size 128 for KV cache: Matches the head dimension $d_h = 128$ in LLAMA-2, meaning each head's key/value vector is quantized with exactly one group. This is clean architecturally and provides sufficient granularity (128 elements per group is fine-grained enough that the distribution within each group is approximately uniform).
Constant clipping ratio of 0.9 for activations: Chosen from a small sweep (Appendix A.2) where 0.9 was optimal. The sensitivity is low — perplexity varies by only 0.11 across the range [0.85, 1.0] — so the exact value is not critical. This is important because it means no per-token or per-layer search is needed for the activation clipping threshold; a single constant works across all layers.

4. Key Insights and Innovations

Innovation 1: Outliers Are a Distributional Problem, Not a Numerical One—So Fix the Distribution, Not the Quantizer

The dominant approach to activation quantization before QuaRot was to build increasingly sophisticated quantizers that could tolerate outliers. LLM.int8() (Dettmers et al., 2022) kept outlier channels in FP16 while quantizing the rest—a mixed-precision patch. SmoothQuant (Xiao et al., 2023) transferred quantization difficulty from activations to weights via per-channel scaling factors, effectively smoothing the outlier magnitudes rather than eliminating them. Atom (Zhao et al., 2023) and QUIK (Ashkboos et al., 2023) both identified specific outlier features offline and retained them in higher precision. In every case, the strategy was: outliers exist, so build a quantizer that can handle them. The resulting systems were complex—irregular memory layouts, custom mixed-precision kernels, hyperparameters controlling which channels receive special treatment—and none achieved uniform 4-bit end-to-end inference.

QuaRot makes a fundamentally different move: don't build a smarter quantizer; make the data easier to quantize. The insight is that outliers are not an intrinsic property of the computation the model performs—they are an artifact of the basis in which the hidden state is represented. A large value in one dimension of the hidden state means that dimension carries a lot of the signal's energy. When you rotate the representation with an orthogonal matrix, that energy gets spread uniformly across all dimensions. The information is preserved exactly (orthogonal transforms are invertible), but no single dimension now has an outsized magnitude. This is a distributional fix, not a numerical one.

What makes this intellectually distinctive is the reframing of the quantization problem from quantizer design (a numerical optimization problem) to representation design (a linear algebraic problem). The paper's central result—that 4-bit end-to-end inference works with zero high-precision channels when the representation is rotated, but collapses to 83.12 WikiText-2 perplexity without it (Table 1, SmoothQuant 4-bit)—is a clean demonstration that the bottleneck was never the quantization algorithm. GPTQ and round-to-nearest are both simple, standard methods. The bottleneck was always the distribution of the data being quantized, and prior work was solving the wrong problem.

This reframing has implications beyond the specific Hadamard rotation technique. It suggests a general principle: before engineering a complex quantizer, ask whether a lossless transformation of the representation can make the problem easier. The fact that random orthogonal matrices work—though worse than Hadamard, they still dramatically improve quantizability (Table 8: 7.45 WikiText-2 perplexity with random orthogonal vs. 83.12 with SmoothQuant)—confirms that any rotation that spreads energy uniformly helps, and the choice of Hadamard is an optimization (for speed and uniformity), not a requirement. This opens the door to future work exploring other structured orthogonal transforms tailored to specific architectures or hardware.

Innovation 2: Computational Invariance Is a Design Primitive, Not Just a Theoretical Curiosity

The computational invariance theorem from SliceGPT (Ashkboos et al., 2024) was originally developed for structured pruning: you can remove rows and columns from weight matrices by rotating the representation and deleting dimensions that carry little signal. QuaRot repurposes this theorem for an entirely different goal—quantization—and in doing so, elevates it from a one-off pruning technique to a general design primitive for LLM compression.

The key conceptual move is recognizing that computational invariance gives you a "free" transformation budget. Because an orthogonal rotation of the hidden state can be cancelled by fusing its inverse into subsequent weight matrices, you can alter the representation geometry at one point in the network without paying any runtime cost elsewhere. QuaRot exploits this to apply a single global rotation $Q$ to the residual stream—eliminating outliers between blocks—and then applies additional "paid" rotations (online Hadamard transforms) only where they cannot be fused: inside the FFN (before the down-projection) and inside attention (head-wise key/query/value rotations). The paper characterizes this as needing only ~1.5 Hadamard transforms per transformer layer at runtime, versus ~2 per weight matrix in QuIP# (Tseng et al., 2024), which treats each weight matrix independently and undoes rotations at runtime.

What makes this more than an incremental optimization is that it reveals a design space that prior work hadn't systematically explored. SliceGPT showed you can rotate once (for pruning). QuIP# showed you can rotate every weight matrix independently (for weight quantization). QuaRot shows you can rotate selectively—at exactly the points where outliers emerge, and only there—by reasoning about where in the network the representation drifts away from uniformity. The three rotation points (residual stream, FFN intermediate activations, attention head values/keys) are not chosen arbitrarily; they correspond to the three points where nonlinear operations (RMSNorm, SiLU gating, and softmax attention) can re-concentrate signal energy into specific dimensions after a previous rotation has spread it out. This is a diagnostic insight: nonlinearities re-create outliers downstream of rotations, so rotations must be re-applied after each nonlinearity.

The evidence that this selective approach is sufficient is the full system performance: adding rotations at just these three points—rather than at every linear layer as QuIP# does—enables end-to-end 4-bit quantization with strong accuracy (Table 1: 6.10 WikiText-2 perplexity on LLAMA2-7B). The fact that no additional rotation points are needed suggests the analysis of where outliers emerge is correct and complete for the transformer architecture studied.

Innovation 3: The KV Cache Can Be Quantized Uniformly If—and Only If—the Attention Mechanism Itself Is Rotated

Prior work on KV cache quantization (FlexGen, KVQuant, KIVI) operated under the assumption that keys and values are intrinsically difficult to quantize uniformly. The evidence was clear: keys exhibit outliers (Hooper et al., 2024; Liu et al., 2024), values accumulate biases from attention patterns, and applying uniform 4-bit quantization to the KV cache without special handling caused significant accuracy degradation. The response from the field was to develop increasingly complex quantization schemes specific to the KV cache: per-channel quantization, non-uniform codebooks, keeping outlier channels in high precision, pre-RoPE caching with inverse rotations. These were essentially the same "build a smarter quantizer" strategy applied to a different tensor.

QuaRot makes the case that the KV cache outlier problem is the same problem as the activation outlier problem, and it has the same solution: rotate the representation so the distribution becomes uniform before quantization. The conceptual contribution is recognizing that the attention computation has an internal structure—the implicit $W_v W_{out}$ product and the $QK^T$ product—that can absorb rotations without changing the attention output. This is not obvious from the standard transformer equations, which present keys, queries, values, and attention scores as separate quantities. Only by examining the products that matter ( $V \cdot W_{out}$ and $Q \cdot K^T$ ) does the rotation-invariance become visible.

The head-wise rotation scheme ( $I_{n_h} \otimes H_{d_h}$ applied to values, keys, and queries) is clever engineering, but the intellectual contribution is the diagnosis: the attention mechanism's head structure does not prevent uniform quantization; it just requires head-structured rotations. The cross-head Hadamard transform ( $H_{n_h} \otimes I_{d_h}$ ) that completes the full rotation is the key recognition that rotating within heads is not enough—information must also be spread across heads to achieve full uniformity. Without this cross-head step, outliers within individual heads would remain, and the concatenated KV cache would still have a block-sparse structure (some heads dominating) that uniform quantization would handle poorly.

The evidence for this claim comes from the ablation in Appendix A.3 (Table 6): with only KV cache quantized and everything else in FP16, 4-bit KV quantization with QuaRot's rotations achieves 5.51 WikiText-2 perplexity on LLAMA2-7B—only 0.04 above the FP16 baseline of 5.47. This is a negligible degradation, achieved with simple asymmetric uniform quantization (group size 128, constant clipping ratio 0.95). The contrast with prior KV cache quantization methods—which needed per-channel or non-uniform schemes to approach this level of accuracy—confirms that the rotation, not the quantization algorithm, is doing the heavy lifting.

A subtler contribution is the post-RoPE caching design choice. The paper explicitly argues against the pre-RoPE approach used by KVQuant, noting that applying inverse rotations to the entire cached key set at every decoding step adds cost proportional to sequence length, while rotating only the single new query vector is constant-cost. This is an architectural insight about where to place the complexity: by accepting that both queries and keys must be rotated (rather than only keys, as in pre-RoPE schemes), QuaRot makes the decoding cost independent of cache size—a significant practical advantage for long-sequence generation.

Innovation 4: Verifier-Free, Calibration-Free Quantization Becomes Possible When Outliers Are Eliminated

A striking result that emerges from the ablation studies—but isn't framed as a central contribution—is that QuaRot enables simple round-to-nearest (RTN) quantization to be competitive, and sometimes lossless, without any calibration data or hyperparameter search. Table 3 shows that 8-bit QuaRot-RTN on LLAMA2-70B achieves 3.33 WikiText-2 perplexity—identical to the FP16 baseline—and 77.17% average zero-shot accuracy versus 77.07% for the baseline. Even at 4 bits, RTN on LLAMA2-70B (4.14 perplexity, 73.63% accuracy) is only 0.35 perplexity and 1.35 accuracy points behind GPTQ (3.79, 75.98%), which requires 128 calibration samples and 2 hours of quantization time.

This is not merely a convenience result. It reveals something fundamental about the relationship between representation geometry and quantization difficulty. GPTQ's Hessian-based compensation scheme exists to correct errors introduced when quantizing weight matrices that have non-uniform eigenvalue spectra—some directions in the weight space matter more than others for the output. The fact that RTN (which ignores these interactions, quantizing each weight independently) nearly matches GPTQ after QuaRot's rotations suggests that the rotations are making weight matrices not only more uniform in their individual entries (reducing incoherence) but also more isotropic in their singular value structure—the sensitivity of the output is more evenly distributed across weight entries.

This has practical significance: calibration data is a deployment burden. It must be representative of the inference distribution, it requires additional computation during model preparation, and it introduces a potential source of distributional bias. A method that achieves lossless 6- and 8-bit quantization with no calibration data and no hyperparameters (beyond the fixed clipping ratios 0.9 and 0.95, which are insensitive) dramatically simplifies the deployment pipeline. The paper could have emphasized this more strongly: it's not just that QuaRot works, but that it works with the simplest possible quantizer—suggesting that the representation geometry, not the quantization algorithm, is the dominant factor.

The negative result with the ReST $^{EM}$ revision model variant (Appendix K, Figure 16) further reinforces this point, though it is reported modestly. Attempting to optimize the revision model with reinforcement learning caused sequential revisions to degrade performance, likely because on-policy data collection amplified spurious correlations. This is a cautionary tale: not all model optimization techniques compose. The positive results depend on the specific transformation pipeline (randomized Hadamard rotations at specific network points) working synergistically with the base model's learned representations. This suggests that representation-level interventions (rotations) are more robust to downstream optimization than algorithmic interventions (revision training), which is a design principle worth articulating.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary quantitative evaluation uses WikiText-2 (Merity et al., 2016) for language modeling perplexity with 2048 sequence length, following the convention in prior quantization work. The zero-shot evaluation uses six tasks: PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), HellaSwag (Zellers et al., 2019), LAMBADA (Radford et al., 2019), and ARC Easy and Challenge (Clark et al., 2018), all run through the LM Evaluation Harness (Gao et al., 2021) with default parameters. The calibration set for GPTQ weight quantization uses 128 samples from the WikiText-2 training set with 2048 sequence length; no calibration data is needed for RTN quantization.
Base model(s). All main experiments use the LLAMA-2 family (Touvron et al., 2023) at three scales: 7B, 13B, and 70B parameters. These models are chosen because they represent widely-used pretrained LLMs with publicly available weights, span a meaningful size range, and use architectural components (gated FFN, RoPE, grouped-query attention in the 70B model) that exercise all aspects of QuaRot's rotation scheme. Appendix results additionally cover LLAMA-3 8B and 70B (Tables 11-12) and Phi-3-mini-4k-instruct (Table 13).
Metrics. For language modeling: WikiText-2 perplexity (PPL), lower is better. Perplexity is computed as $\exp(\text{cross-entropy loss})$ on the test set. For zero-shot tasks: per-task accuracy (%) and an unweighted average across all six tasks. All zero-shot evaluations use greedy decoding (no sampling). For performance analysis: speedup over FP16 baseline (ratio of wall-clock times) for prefill and peak memory saving factor during decoding (ratio of peak GPU memory usage).
Baselines. The paper compares against SmoothQuant (Xiao et al., 2023) at 4-bit, OmniQuant (Shao et al., 2023) at 4-bit, QUIK at 4-bit (Ashkboos et al., 2023), and Atom at 4-bit with group-wise quantization (Zhao et al., 2023). SmoothQuant and OmniQuant results are extracted from Shao et al. (2023). The FP16 (unquantized) baseline is always included. For weight-only quantization ablations (Appendix A.4), baselines include standard RTN and GPTQ without rotation.
Generation budget / compute accounting. Compute is not budgeted per-generation as in inference-time scaling papers; instead, the evaluation uses standard language modeling perplexity (loss over a fixed test corpus) and zero-shot task accuracy. The performance analysis benchmarks wall-clock time on specific hardware (NVIDIA RTX 3090) for single transformer blocks, varying batch size and sequence length. All 4-bit linear layer benchmarks report milliseconds per operation averaged over 100–1000 runs. Memory savings are computed as the ratio of peak GPU memory for FP16 vs. QuaRot during decoding of 50 tokens with 16 batch size and varying prefill sequence lengths.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The paper uses fixed test sets (WikiText-2 test, standard zero-shot evaluation splits from LM Evaluation Harness). The weight quantization clipping ratio is selected per-column via linear search over squared error on the calibration set; activation and KV cache clipping ratios (0.9 and 0.95 respectively) are fixed constants chosen from a small sweep on LLAMA2-7B (Appendix A.2, Table 5). No standard deviations or confidence intervals are reported for any accuracy metric.

Main Quantitative Results

Language Modeling: WikiText-2 Perplexity with 4-bit End-to-End Quantization

Table 1 presents the headline language modeling results. The FP16 baselines are 5.47 (7B), 4.88 (13B), and 3.32 (70B) WikiText-2 perplexity.

QuaRot vs. prior 4-bit methods (GPTQ weights, RTN activations, 4-bit KV cache):

QuaRot achieves 6.10 perplexity on 7B, 5.40 on 13B, and 3.79 on 70B — losses of 0.63, 0.52, and 0.47 perplexity points respectively.
SmoothQuant at 4-bit collapses: 83.12 on 7B and 35.88 on 13B — these are effectively useless (perplexity above 10 indicates the model has lost most of its predictive power).
OmniQuant at 4-bit: 14.26 on 7B and 12.30 on 13B — better than SmoothQuant but still 2–3× the QuaRot perplexity.
QUIK-4B, which keeps 256 outlier features in higher precision: 8.87 on 7B, 7.78 on 13B, 6.91 on 70B. QuaRot outperforms QUIK by 2.77, 2.38, and 3.12 points respectively, while using zero high-precision channels vs. QUIK's 256.
Atom-128G with group-wise quantization (group size 128, keeping 128 outlier features): 6.03 on 7B and 5.26 on 13B. QuaRot-128G: 5.93 on 7B and 5.26 on 13B — a 0.1 point improvement on 7B and tied on 13B, again without any outlier feature handling.

The critical takeaway from Table 1 is the "0" in QuaRot's #Outlier Features column. Every prior method that achieved non-catastrophic perplexity at 4-bit did so by either (a) keeping specific features in higher precision, (b) using complex mixed-precision kernels, or (c) both. QuaRot is the first method to achieve usable 4-bit perplexity with uniform quantization and no special channels.

Group-wise quantization trade-off (Table 4):

Using GPTQ for weight quantization with various group sizes for both weights and activations (KV cache group size fixed at 128):

QuaRot (no group-wise): 6.10 / 5.40 / 3.79 (7B / 13B / 70B)
QuaRot-256G: 5.98 / 5.28 / 3.63
QuaRot-128G: 5.93 / 5.26 / 3.61
QuaRot-64G: 5.88 / 5.25 / 3.58

The improvement from no group-wise to 128G is 0.17, 0.14, and 0.18 perplexity points on 7B, 13B, and 70B respectively—modest but consistent. The diminishing returns from 128G to 64G (only 0.05, 0.01, 0.03 improvement) suggest that 128G captures most of the benefit, which is convenient because group size 128 matches the head dimension in LLAMA-2 and enables efficient kernel implementations.

Round-to-nearest weight quantization (Table 3 and Table 9):

The full results across 4, 6, and 8 bits with both RTN and GPTQ weight quantization (Table 9):

8-bit RTN on LLAMA2-70B: 3.33 WikiText-2 perplexity, identical to FP16 baseline. Zero-shot average: 77.17% vs. 77.07% baseline — a 0.10% improvement (within noise). This is lossless quantization with no calibration data and no hyperparameter tuning.
6-bit RTN on LLAMA2-7B: 5.56 perplexity (0.09 loss), 69.42% zero-shot average (0.40% loss). Effectively lossless.
4-bit RTN: 8.37 / 6.09 / 4.14 perplexity on 7B / 13B / 70B. The gap from RTN to GPTQ is 2.27 (7B), 0.69 (13B), and 0.35 (70B). The narrowing gap with model size is a key finding: for the 70B model, RTN is only 0.35 perplexity behind GPTQ, suggesting that very large models are intrinsically more robust to quantization error.

Zero-Shot Task Accuracy

Table 2 presents the zero-shot accuracy of 4-bit QuaRot (GPTQ weights, RTN activations, 4-bit KV cache) on six tasks.

LLAMA2-70B:

FP16 average: 77.07%. QuaRot average: 75.98%. Loss: 1.09 percentage points (1.4% relative).
Best-preserved task: PIQA (82.43% vs 82.70%, -0.27), ARC-Challenge (56.23% vs 57.34%, -1.11).
Worst-preserved task: HellaSwag (81.82% vs 83.84%, -2.02).

The paper's claim that QuaRot "preserves 99% of the zero-shot performance" on LLAMA2-70B is supported: 75.98 / 77.07 = 98.6%.

LLAMA2-13B:

FP16 average: 72.59%. QuaRot average: 69.79%. Loss: 2.80 points (3.9% relative).

LLAMA2-7B:

FP16 average: 69.82%. QuaRot average: 65.64%. Loss: 4.18 points (6.0% relative).

The degradation is monotonic with model size: larger models lose less. This is consistent with the well-known observation that larger models are more compressible.

Table 9 provides the full breakdown including 6-bit and 8-bit results. At 8-bit GPTQ on LLAMA2-70B: 77.28% average vs. 77.07% baseline — actually above baseline (likely noise). At 6-bit GPTQ on LLAMA2-7B: 69.77% vs. 69.82% baseline — essentially identical. These confirm that 6- and 8-bit QuaRot are lossless across all model sizes.

Performance Analysis: Prefill Speedup and Decoding Memory Saving

All performance experiments are on a single NVIDIA RTX 3090 GPU per transformer block (the full model does not fit on the GPU for large batch sizes).

Prefill speedup (Figure 4 Left, Table 16):

For 2048 sequence length prefill on LLAMA2-70B:

Batch size 1: 3.16× speedup
Batch size 16: 3.32× speedup
Batch size 64: 3.33× speedup

On LLAMA2-7B:

Batch size 1: 1.97× speedup
Batch size 16: 2.11× speedup
Batch size 64: 2.16× speedup

The speedup increases with batch size (more compute-bound) and with model size (larger matrices benefit more from INT4 Tensor Core throughput). The near-linear scaling with batch size confirms that the GEMM, not the Hadamard transform or quantization overhead, dominates runtime.

Decoding memory saving (Figure 4 Right, Table 17):

For decoding with batch size 16, varying prefill sequence lengths from 256 to 4096:

LLAMA2-7B: peak memory saving of 3.63× to 3.75× (increases with sequence length because the KV cache grows relative to fixed model parameters).
LLAMA2-70B: 3.89× across all sequence lengths (essentially constant because the 70B model uses grouped-query attention with fewer KV heads, making the KV cache proportionally smaller relative to the model, so the savings ceiling is reached even at short sequences).

The paper notes that the 3.89× figure is expected to increase for the full model (vs. single layer) because constant-size objects (embeddings, final LM head) become negligible across many layers.

Linear layer micro-benchmarks (Figure 7, Table 14):

For the LLAMA2-7B down-projection layer (11008 × 4096), batch size 1:

FP16: 2.569 ms
INT4 (no Hadamard): 0.749 ms (3.43× speedup)
INT4 + FP32 Hadamard: 0.798 ms (3.22× speedup)
INT4 + FP16 Hadamard: 0.801 ms (3.21× speedup)

The Hadamard transform overhead is 6.5% (FP32) to 7.0% (FP16) over the INT4 matmul alone. The paper argues this is small enough to be acceptable, and the FP16 Hadamard being nearly identical to FP32 (while using less register pressure and memory) is a practical insight for deployment optimization.

For LLAMA2-70B down-projection (28672 × 8192), batch size 1:

FP16: 12.450 ms
INT4 + FP32 Hadamard: 2.911 ms (4.28× speedup)

The speedup ratio increases with matrix size, consistent with Tensor Core INT4 throughput scaling better than FP16 for larger GEMMs.

Attention decode micro-benchmarks (Table 15):

For LLAMA2-7B attention (32 heads × 128 dim), decoding a single token with 2047 cached tokens:

FP16, batch size 1: 0.713 ms. INT4, batch size 1: 1.033 ms — INT4 is slower.
FP16, batch size 16: 1.348 ms. INT4, batch size 16: 1.018 ms — INT4 is 1.32× faster.
FP16, batch size 32: 2.098 ms. INT4, batch size 32: 1.168 ms — INT4 is 1.80× faster.

The crossover point where INT4 becomes faster is between batch sizes 8 and 16 (the exact threshold is not reported). This is because at small batch sizes, the quantization/dequantization overhead dominates over the I/O savings from the 4× smaller cache. The larger the batch size (or the longer the sequence), the more memory-I/O-bound the attention becomes, and the more the 4-bit KV cache helps. This crossover behavior is correctly predicted by the paper's framing: the decoding stage is memory-bound, and reducing memory traffic helps only when memory traffic is the bottleneck.

Ablation Studies and Robustness Checks

Clipping ratio for activation quantization (Appendix A.2, Table 5): On LLAMA2-7B with only input quantization (everything else FP16), clipping ratios of 1.0, 0.95, 0.9, and 0.85 give WikiText-2 perplexities of 5.938, 5.910, 5.828, and 5.850 respectively. The optimum is 0.9 with only 0.11 perplexity variation across the full range, indicating insensitivity to this hyperparameter. For KV cache quantization: ratios of 1.0, 0.95, 0.9, 0.85 give 5.513, 5.510, 5.517, 5.532 — even less variation (0.022 range), with 0.95 fractionally best. This insensitivity means the fixed clipping ratios are not a hidden source of tuning advantage over baselines.

KV cache bit-width combinations (Appendix A.3, Table 6): With the rest of the model in FP16 (weight and activation quantization off), only the KV cache quantized with asymmetric per-group (128) quantization:

4-bit keys + 4-bit values: 5.51 (7B), 4.91 (13B), 3.33 (70B) — losses of 0.04, 0.03, and 0.01.
4-bit keys + 3-bit values: 5.54, 4.93, 3.35 — additional 0.03 loss from value bit reduction.
3-bit keys + 4-bit values: 5.65, 5.01, 3.38 — additional 0.14 loss from key bit reduction.
3-bit keys + 3-bit values: 5.68, 5.02, 3.39.
4-bit keys + 2-bit values: 5.75, 5.09, 3.43 — still usable.
2-bit keys + 4-bit values: 8.06, 6.42, 3.89 — substantially worse, especially on 7B.

The key finding: keys are more sensitive to quantization than values. The asymmetry (4K3V: 5.54 vs. 3K4V: 5.65 on 7B) shows that reducing keys from 4 to 3 bits costs ~0.14 perplexity while reducing values costs only ~0.03. This matches previous findings from KVQuant and KIVI. The practical implication is that mixed-precision KV cache (e.g., 4-bit keys + 2-bit values) may be more efficient than uniform reduction. On the 70B model, 2-bit values (5.75) is actually better than 2-bit keys (8.06), reinforcing the asymmetry.

Weight-only quantization with and without QuaRot rotations (Appendix A.4, Table 7): Keeping activations and KV cache in FP16, quantizing only weights:

4-bit GPTQ without QuaRot: 8.25 (7B), 5.65 (13B), 3.87 (70B).
4-bit GPTQ with QuaRot: 5.60 (7B), 5.00 (13B), 3.41 (70B) — gains of 2.65, 0.65, and 0.46 perplexity.
3-bit GPTQ without QuaRot: NaN on 7B (failed), 9.51 on 13B, 5.91 on 70B.
3-bit GPTQ with QuaRot: 6.09 (7B), 5.37 (13B), 3.72 (70B).
2-bit GPTQ with QuaRot: 22.07 (7B), 10.41 (13B), 5.60 (70B). Without QuaRot: Infinity on 7B and 13B, 25.30 on 70B.

The rotation alone—without any activation or KV cache quantization—provides substantial benefits, and the gap between rotated and unrotated weight quantization widens at lower bit-widths. On the 70B model, QuaRot enables usable 2-bit weight quantization (5.60 perplexity), while unrotated 2-bit is essentially random (25.30). This isolates the contribution of incoherence processing from the activation quantization benefits, confirming that the weight-side improvement is real and substantial.

Random orthogonal matrices vs. Hadamard (Appendix A.5, Table 8): Using a random orthogonal matrix (QR decomposition of a uniform random matrix) instead of a randomized Hadamard for the global rotation Q:

LLAMA2-7B: Random 7.45 vs. Hadamard 6.10 (1.35 gap).
LLAMA2-13B: Random 5.84 vs. Hadamard 5.40 (0.44 gap).
LLAMA2-70B: Random 4.07 vs. Hadamard 3.79 (0.28 gap).

Random orthogonal matrices are consistently worse, but the gap narrows with model size. This suggests that the specific structure of Hadamard matrices (entries ±1, maximally incoherent) matters more for smaller models, while larger models are robust to the choice of rotation. The paper does not explore why Hadamard outperforms random; possible explanations include (a) Hadamard's maximal incoherence (all entries have equal magnitude), (b) the Walsh structure spreading energy uniformly in the frequency domain, or (c) better conditioning for subsequent quantization. The narrowing gap with model size is consistent with larger models being more compressible in general.

FP16 vs. FP32 Hadamard transform precision (Appendix A.7, Table 10): Using FP16 precision for the online Hadamard transforms instead of FP32:

LLAMA2-7B: FP32 gives 6.10 perplexity, 65.64% zero-shot average. FP16 gives 6.08 perplexity, 66.21% average. The 0.57% zero-shot improvement with FP16 is within noise, and perplexity is essentially unchanged.
LLAMA2-13B: FP32 gives 5.40 perplexity, 69.79% average. FP16 gives 5.40 perplexity, 69.72% average. Identical perplexity, 0.07% difference in accuracy.

This is practically important: FP16 Hadamard transforms use less memory, less register pressure, and potentially faster execution than FP32. The paper's accuracy experiments use FP32, but these results suggest FP16 is sufficient, which matters for deployment optimization.

LLAMA-3 results (Appendix A.8, Tables 11-12): Extending to the LLAMA-3 family:

LLAMA3-8B 4-bit QuaRot: 8.16 perplexity vs. 6.14 baseline (2.02 loss). With 128G: 7.36 (1.22 loss). Zero-shot: 65.18% average vs. 73.22% baseline (8.04% loss).
LLAMA3-70B 4-bit QuaRot: 6.66 perplexity vs. 2.86 baseline (3.80 loss). With 128G: 5.51 (2.65 loss). Zero-shot: 69.21% average vs. 79.94% baseline (10.73% loss).

These results are substantially worse than the LLAMA-2 results (where 70B lost only 0.47 perplexity). The paper notes that "LLAMA-3 is more sensitive to quantization" but does not investigate why. Possible explanations include differences in training data, architecture, or activation distributions — LLAMA-3 might have different outlier patterns that the Hadamard rotations don't eliminate as effectively. This is a concerning finding: QuaRot's effectiveness may not generalize uniformly across model families, and the paper doesn't provide diagnostic tools to predict when it will work well.

Phi-3-mini results (Appendix A.9, Table 13): On Phi-3-mini-4k-instruct (Abdin et al., 2024):

4-bit GPTQ: 7.85 perplexity vs. 6.35 baseline (1.50 loss). Zero-shot: 66.34% average vs. 73.11% baseline (6.77% loss).
6-bit GPTQ: 6.63 perplexity vs. 6.35 baseline (0.28 loss). Zero-shot: 72.67% vs. 73.11% baseline (0.44% loss) — nearly lossless.
8-bit RTN: 6.58 perplexity, 73.02% average — nearly lossless (6-bit GPTQ and 8-bit RTN tie on zero-shot average).

The 4-bit results are worse than LLAMA-2 but better than LLAMA-3. The 6- and 8-bit results remain strong. Again, no analysis of why Phi-3 is more sensitive than LLAMA-2 is provided.

Critical Assessment

Claim 1: "QuaRot enables end-to-end 4-bit inference without any channels identified for retention in higher precision." This claim is clearly supported by the data. Table 1 shows QuaRot with 0 outlier features achieving 6.10, 5.40, and 3.79 perplexity on LLAMA-2 7B/13B/70B, while all competitive 4-bit baselines require higher-precision channels. However, the claim should be understood as applying to the LLAMA-2 architecture specifically. The LLAMA-3 results (Table 11) with losses of 2.02 (8B) and 3.80 (70B) perplexity are arguably not "enabling end-to-end 4-bit inference" in a practical sense — a 3.80 perplexity loss on a 2.86 baseline is a ~133% increase in perplexity, which would be unacceptable for most applications. The paper acknowledges this implicitly by noting LLAMA-3 is "more sensitive" but does not discuss the implications for the generality of the claim. A more precise statement would be: "QuaRot enables end-to-end 4-bit inference on LLAMA-2 class models with no high-precision channels; effectiveness varies across model families and may require group-wise quantization for acceptable accuracy on more recent architectures."

Claim 2: "Our 4-bit quantized LLAMA2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance." The numbers are accurate: 3.79 vs. 3.32 perplexity (0.47 loss) and 75.98% vs. 77.07% zero-shot accuracy (98.6% retention). Both metrics are clearly reported. However, a weakness is the absence of any measure of variance. Without standard deviations or confidence intervals, we cannot distinguish a 0.47 perplexity loss from a 0.35 or 0.60 loss with any statistical rigor. For the zero-shot tasks, the 1.09 percentage point accuracy loss on the 70B model is small but could be statistically significant or noise — we cannot tell. The paper would be strengthened by reporting results over multiple random seeds or with bootstrap confidence intervals on the zero-shot averages.

Claim 3: "QuaRot achieves up to 3.33× prefill speedups and 3.89× memory saving during the decoding stage." These numbers come from single-transformer-block benchmarks on an RTX 3090 (Tables 16-17) and are accurate as reported. However, there are important caveats that a critical reader should note:

First, the speedups are measured on a single block, not the full model. The paper states (Section 5.2) that "the whole model does not fit on our GPU cluster for large batch sizes." The full-model speedup could differ because (a) operations like embedding lookup, final LM head, and cross-layer operations (which are not quantized) add fixed overhead, reducing the relative speedup, and (b) memory management across multiple layers introduces fragmentation and scheduling overhead absent in single-block benchmarks. The paper argues that speedups "could be improved by optimizing our kernels" but doesn't quantify the gap between single-block and full-model performance.

Second, the decoding memory savings (3.63–3.89×) include the KV cache but not the model weights — the paper is comparing peak memory during decoding of one transformer block, which includes the KV cache for that block. The full model's memory savings would include weight memory reduction as well (weights are 4-bit vs. 16-bit), potentially making the overall savings larger than 3.89×. But this is not reported.

Third, the attention decode benchmarks (Table 15) reveal that INT4 KV cache is actually slower than FP16 at small batch sizes (1–8). The crossover point is not precisely characterized, but it means the claimed speedups are conditional on sufficient batch size or sequence length. For single-query inference (batch size 1, the most common deployment scenario for interactive applications), the decoding stage may be slower with QuaRot's 4-bit KV cache than with FP16. This tradeoff—latency vs. throughput—is not discussed.

Claim 4: "6 and 8-bit quantization is lossless with simple round-to-nearest quantization." This is the strongest and best-supported claim in the paper. Table 9 shows that 8-bit RTN on LLAMA2-70B achieves 3.33 perplexity (identical to FP16) and 77.17% zero-shot average (0.10% above baseline). On LLAMA2-7B, 8-bit RTN achieves 5.50 perplexity (0.03 loss) and 69.65% average (0.17% loss). These are essentially perfect. The fact that this requires no calibration data, no GPTQ, no hyperparameter search (beyond the fixed clipping ratio), and no outlier channel identification makes this result practically significant independent of the 4-bit results. A deployment that needs guaranteed no-loss compression can use 8-bit QuaRot-RTN with high confidence. The paper could have made this a stronger focus.

Missing experiments and analyses:

No full-model performance benchmarks. All speedup numbers are per-block. The actual end-to-end throughput improvement on a real inference serving system (with batching, scheduling, and the full model pipeline) is unknown. This matters because the overhead of quantization/dequantization and Hadamard transforms might interact with memory management, kernel launch overhead, and pipelining in non-obvious ways at full-model scale.
No comparison to 4-bit weight-only + FP16 activations. The paper compares against 4-bit joint quantization methods (SmoothQuant, OmniQuant, QUIK) but doesn't include the obvious baseline: use GPTQ for 4-bit weight-only quantization (keeping activations in FP16), which would give some of the memory savings but none of the compute speedups. The weight-only ablation (Table 7) provides this for perplexity, but no speedup or memory comparison is made. This would help a practitioner decide whether the additional complexity of activation quantization is worth the speedup.
No investigation of why LLAMA-3 degrades more. Table 11 shows LLAMA3-70B losing 3.80 perplexity vs. 0.47 for LLAMA2-70B — an 8× larger degradation. The paper offers no analysis. Possible causes (different training data, different activation distributions, architectural differences) are unexplored. This is a significant gap because it suggests QuaRot's effectiveness may be specific to LLAMA-2's training recipe.
No ablation on the randomization of Hadamard matrices. The paper uses randomized Hadamard ( $\tilde{H} = H \cdot \text{diag}(s)$ with random signs $s$ ) but never ablates this against a pure Hadamard. The QuIP# paper found randomization important, but the effect size in the context of activation quantization is unknown. The comparison to random orthogonal matrices (Table 8) partially addresses this but conflates two differences (Hadamard vs. non-Hadamard structure, and random signs vs. deterministic).
Single calibration set. The GPTQ calibration uses 128 samples from WikiText-2 training. No sensitivity analysis to calibration set size or domain is provided. If the calibration set is in-domain with the test set (both WikiText-2), the results may slightly overestimate real-world performance on out-of-domain text. The RTN results (which don't use calibration) partially mitigate this concern by showing that GPTQ's advantage over RTN is modest, especially for large models.
No latency analysis for the full decoding loop. The attention decode benchmarks (Table 15) measure only the attention component, not the full per-token generation time including FFN layers, RMSNorm, and the online Hadamard transforms. A breakdown of per-token latency across all components would help identify bottlenecks.

Weaknesses in the experimental design:

The evaluation provides strong evidence for the central claim (QuaRot enables 4-bit end-to-end inference on LLAMA-2 without high-precision channels) but weaker evidence for generality. The single-model-family focus (LLAMA-2 for most experiments), the per-block rather than full-model performance benchmarks, and the absence of variance estimates on accuracy metrics are the most notable gaps. The LLAMA-3 results (Appendix A.8) partially address generality but reveal concerning degradation that is left unexplained. A reader considering adopting QuaRot for a model architecture other than LLAMA-2 would need to run their own evaluation—the paper does not provide principles for predicting when the method will work well.

6. Limitations and Trade-offs

6.1 The Effectiveness of QuaRot Varies Dramatically Across Model Families, and the Paper Provides No Diagnostic Framework for Predicting When It Will Work

The assumption or constraint. The paper's central claim — that Hadamard rotations eliminate outliers and enable uniform 4-bit quantization — is validated almost entirely on the LLAMA-2 model family. All main experiments (Tables 1–4, Figures 1–4) use LLAMA-2 7B, 13B, and 70B. The authors extend evaluation to LLAMA-3 (Appendix A.8) and Phi-3-mini (Appendix A.9), but the results reveal a substantial gap that is never analyzed or explained. On LLAMA2-70B, 4-bit QuaRot loses 0.47 WikiText-2 perplexity (3.79 vs. 3.32 baseline). On LLAMA3-70B, the same method loses 3.80 perplexity (6.66 vs. 2.86 baseline) — an 8× larger degradation. On LLAMA3-8B, the loss is 2.02 perplexity (8.16 vs. 6.14). The paper notes only that "LLAMA-3 is more sensitive to quantization" (Appendix A.8) but offers no investigation of why, no ablation to isolate the cause, and no guidance for practitioners evaluating whether their target model is "LLAMA-2-like" or "LLAMA-3-like" in its sensitivity. Phi-3-mini shows intermediate sensitivity (1.50 perplexity loss at 4-bit).

The consequence. A practitioner considering QuaRot for a model not in the LLAMA-2 family cannot predict whether they will get the 0.47 perplexity loss seen on LLAMA2-70B, the 3.80 loss seen on LLAMA3-70B, or something in between. The difference between these regimes is the difference between "deployable with minimal degradation" and "unacceptable quality loss for most applications." The paper provides no diagnostic tools — no measurement to run on a new model to estimate how sensitive it will be, no analysis of which architectural or training properties correlate with QuaRot effectiveness. Possible explanations (different activation distributions, different outlier patterns not eliminated by Hadamard rotations, different training data or objectives) are unexplored. This means the method's applicability is currently an empirical question requiring full implementation and evaluation per model family, with no theoretical or diagnostic shortcut.

What evidence exists. Appendix A.8, Tables 11 and 12 provide the LLAMA-3 numbers that reveal the gap. Table 1 provides the LLAMA-2 numbers for contrast. No ablation or analysis of the gap appears anywhere in the paper. The raw numbers are reported without comment beyond the single sentence acknowledging sensitivity. There is no measurement of whether the Hadamard rotations actually eliminate outliers in LLAMA-3 to the same degree as in LLAMA-2 (no Figure 1 equivalent for LLAMA-3), no comparison of weight incoherence pre- and post-rotation across model families, and no experiment testing whether the degradation comes from weight quantization, activation quantization, or KV cache quantization specifically.

Mitigation status. Not addressed. The paper does not investigate the LLAMA-3 degradation, does not propose modifications to QuaRot that might improve robustness across model families, and does not suggest diagnostic procedures. A practitioner is left to run their own full evaluation. This is the single most consequential limitation for anyone considering deploying QuaRot on a model not in the LLAMA-2 family.

6.2 All Performance Benchmarks Are Per-Transformer-Block, Not Full-Model; True End-to-End Speedups and Memory Savings Are Unknown

The assumption or constraint. Section 5.2 explicitly acknowledges: "We provide all our experiments on a single transformer block as the whole model does not fit on our GPU cluster for large batch sizes." All speedup numbers (Figure 4, Table 16), all memory saving numbers (Figure 4, Table 17), and all micro-benchmarks (Figure 7, Tables 14–15) are measured on individual transformer blocks, not on a complete end-to-end inference pipeline. The paper's headline claims — "up to 3.33× prefill speedups" and "3.89× memory saving" — are therefore per-block measurements, not full-system throughput or latency figures.

The consequence. A full-model deployment includes operations that QuaRot does not quantize or accelerate: token embedding lookup, the final LM head (which produces logits over the full vocabulary and is typically a large matrix multiply), cross-layer operations (residual connections, though these are cheap), and the overhead of Python-level orchestration, kernel launch, and memory management across all layers. These unquantized components add fixed overhead that reduces the relative speedup compared to the per-block measurement. For example, if a transformer block accounts for 90% of total inference time and sees a 3× speedup, the end-to-end speedup is roughly 2.5×, not 3×. Conversely, the weight memory savings (4× from 4-bit quantization) would make the full-model memory savings larger than the per-block KV-cache-only savings, but this is not reported. Without full-model benchmarks, a practitioner cannot estimate the actual throughput improvement or latency reduction they would observe in a production inference serving system. The per-block numbers represent an optimistic upper bound on speedup (since unquantized fixed costs are excluded) but a potentially pessimistic bound on memory savings (since weight compression benefits are not included in the KV-cache-only memory measurement).

What evidence exists. The per-block numbers are well-documented in Tables 14–17 and Figure 4. The acknowledgment that full-model benchmarks were infeasible is explicitly stated. But no analysis quantifies what fraction of total inference time or memory the per-block measurements represent, making it impossible to extrapolate to full-model performance from the paper alone. The authors note that performance "could be improved by optimizing our kernels (e.g., fusing the quantization operations into the MatMul)" but do not quantify the gap between current implementation and an optimized one.

Mitigation status. Not addressed as a limitation beyond the brief acknowledgment. The paper suggests that memory savings would increase for the full model ("since as the number of layers increases the effect of constant size objects in memory becomes much less significant") but provides no numbers. The speedup extrapolation in the opposite direction (full-model being slower than per-block) is not discussed. This is a practical gap: reproducing QuaRot and measuring end-to-end performance on available hardware is left entirely to the practitioner.

6.3 The 4-Bit KV Cache Is Slower Than FP16 at Small Batch Sizes, Making the Decoding Speedup Conditional on Sufficient Throughput Demand

The assumption or constraint. The paper frames the decoding stage as "memory-bound" and argues that reducing KV cache size via 4-bit quantization reduces memory I/O and therefore improves speed. However, this argument only holds when memory bandwidth is the bottleneck. The attention decode micro-benchmarks (Appendix A.10, Table 15) reveal a crossover: for LLAMA2-7B attention (32 heads × 128 dim) decoding with 2047 cached tokens, the INT4 KV cache is slower than FP16 at batch sizes 1–8, only becoming faster at batch size 16 and above. At batch size 1, FP16 takes 0.713 ms vs. INT4 at 1.033 ms — the INT4 version is 45% slower. At batch size 4, FP16 takes 0.781 ms vs. INT4 at 1.033 ms — still 32% slower. The crossover point where INT4 becomes faster lies somewhere between batch sizes 8 and 16, though the exact threshold is not precisely characterized.

The consequence. For the most common deployment scenario — single-query interactive inference with batch size 1 — QuaRot's 4-bit KV cache is a latency regression, not an improvement. The quantization and dequantization overhead (loading packed INT4 values, unpacking, applying scales and zero-points) dominates over the I/O savings when there is only one query attending over the cache. This means the decoding speedup claim applies only to high-throughput batch inference settings (batch size ≥ 16, or equivalently, serving many concurrent requests). For applications like chatbots, code completion, or interactive assistants where latency per token is the primary metric, QuaRot's KV cache quantization may actually degrade user experience. The paper does not discuss this latency-throughput tradeoff, nor does it suggest hybrid strategies (e.g., using INT4 KV cache only when batch size exceeds a threshold).

What evidence exists. Table 15 provides the raw timing numbers that reveal the crossover. The paper does not plot or discuss this crossover explicitly; it only notes generically that "a speed up is only visible if this reduction is more significant than the quantization overhead which happens for either larger batch sizes or longer sequences." The quantitative threshold (between batch sizes 8 and 16) is not stated.

Mitigation status. Not addressed. The paper does not propose strategies for avoiding the small-batch regression (e.g., storing the KV cache in FP16 for the most recent tokens and INT4 for older ones, or only quantizing when memory pressure exceeds a threshold). The focus remains on throughput-oriented metrics without acknowledging the latency cost.

6.4 The Paper Does Not Measure or Discuss the Numerical Precision Requirements for the Hadamard Transforms and Rotation Cancellation

The assumption or constraint. The computational invariance theorem guarantees arithmetic equivalence between the original and rotated models assuming exact real arithmetic. In floating-point hardware, the forward Hadamard transform ( $C' \leftarrow C H$ ) and its fused inverse in the weight matrix ( $W_{\text{down}} \leftarrow H W_{\text{down}}$ ) will not cancel perfectly due to rounding errors. The paper reports that FP16 Hadamard transforms produce nearly identical accuracy to FP32 (Appendix A.7, Table 10), suggesting the cancellation error is small in practice. However, this experiment only varies the precision of the online Hadamard; the weight modifications (fusing $H$ , $Q$ , and $Q^T$ into the weight matrices) are done in the model's native precision (presumably FP16 for LLAMA-2), and the compounded effect of multiple layers of imperfect cancellation is never measured.

The consequence. As models grow deeper and quantization becomes more aggressive, the accumulated floating-point error from imperfect rotation cancellation could compound across dozens of layers. The paper provides no measurement of the output difference between the original model and the rotated (but unquantized) model — what is sometimes called the "transformation error." If this error is non-negligible, then some fraction of the observed perplexity degradation attributed to quantization may actually be due to numerical imprecision in the transformation itself. This matters for two reasons: (a) it sets a floor on how accurate QuaRot can be regardless of quantization quality, and (b) it means that models with lower native precision (e.g., those trained in BF16 rather than FP32) might suffer more from transformation error, potentially explaining part of the LLAMA-3 degradation if those models were trained at different precision.

What evidence exists. The only relevant ablation is Appendix A.7 (Table 10), which varies the online Hadamard precision between FP32 and FP16 and finds negligible difference. However, this does not measure the transformation error of the full rotated-but-unquantized model versus the original. No experiment reports the logit difference, perplexity difference, or any other metric comparing the FP16 baseline to a QuaRot-rotated model with all quantization turned off. A simple diagnostic — run the rotated model in FP16 with no weight or activation quantization and measure the perplexity — would isolate the transformation error from the quantization error, but this experiment is not reported.

Mitigation status. Not addressed. The paper implicitly assumes the transformation error is zero or negligible (consistent with computational invariance holding in exact arithmetic), but never verifies this in finite precision. The insensitivity to FP16 vs. FP32 online Hadamard (Table 10) provides partial reassurance but does not substitute for a direct measurement of transformation error.

6.5 The Difficulty Estimation for Clipping Ratios and Calibration Dependence Introduces Implicit Data Assumptions That Are Not Evaluated for Robustness

The assumption or constraint. While QuaRot-RTN with 6–8 bits achieves lossless quantization without calibration data, the headline 4-bit results depend on two forms of data-dependent tuning: (1) GPTQ weight quantization uses a calibration set of 128 samples from WikiText-2 training with 2048 sequence length to compute the Hessian-based error compensation, and (2) the per-column weight clipping ratios are selected via linear search over squared error on this same calibration set. The activation clipping ratio (0.9) and KV cache clipping ratio (0.95) were chosen from a small sweep on LLAMA2-7B (Appendix A.2, Table 5) and then applied uniformly to all model sizes and tasks. The paper provides no evaluation of how sensitive the results are to the choice of calibration set, its size, or its domain. All perplexity evaluation is on WikiText-2 test — which is in-domain with the GPTQ calibration set (both drawn from WikiText-2). The zero-shot tasks (PIQA, WinoGrande, HellaSwag, etc.) are out-of-domain relative to the calibration set, but no analysis of calibration set domain shift is presented.

The consequence. GPTQ with in-domain calibration can overestimate real-world performance if the deployment distribution differs from WikiText-2. If a practitioner is deploying on code, scientific text, or multilingual data, the GPTQ calibration on WikiText-2 may produce suboptimal clipping ratios and Hessian compensation, potentially degrading accuracy beyond the reported numbers. The fixed clipping ratios (0.9 for activations, 0.95 for KV cache) were tuned on LLAMA2-7B and may not be optimal for other model sizes or architectures — though the sweep in Table 5 shows low sensitivity (only 0.11 perplexity variation across the range 0.85–1.0 for activations), this was measured only on LLAMA2-7B and only for input quantization with everything else in FP16. The interaction of the clipping ratio with the full 4-bit quantization pipeline (weights + activations + KV cache simultaneously) could amplify sensitivity in ways not captured by the single-variable sweep. For the LLAMA-3 results, where degradation is already large, suboptimal clipping from using LLAMA-2-tuned hyperparameters might be contributing to the gap — but this is not investigated.

What evidence exists. The clipping ratio sweep in Appendix A.2 (Table 5) provides limited evidence of low sensitivity for LLAMA2-7B in isolation. Table 3 shows that RTN (no calibration data, no per-column clipping search) on LLAMA2-70B is only 0.35 perplexity behind GPTQ, suggesting calibration dependence is modest for large models. However, no experiment varies the calibration set domain (e.g., using C4, GitHub, or multilingual data instead of WikiText-2) and measures the impact, nor is there any evaluation of calibration set size sensitivity (how many samples are needed before GPTQ saturates?).

Mitigation status. The paper acknowledges implicitly that calibration dependence is a concern by providing the RTN results (Table 3, Table 9) as a calibration-free alternative. For 6–8 bits, RTN achieves lossless results, which fully mitigates this limitation. For 4-bit, the GPTQ-RTN gap is 2.27 perplexity on LLAMA2-7B (significant) but only 0.35 on LLAMA2-70B (minor), suggesting that the largest models are robust to removing calibration dependence. However, this pattern is shown only for LLAMA-2; the LLAMA-3 RTN results are not reported, so whether LLAMA-3-70B RTN would close the gap to GPTQ is unknown.

6.6 The Method Introduces a Latency-Throughput Tradeoff for the Attention Mechanism That Is Not Characterized or Discussed

The constraint. QuaRot inserts three types of online computation into the forward pass that were not present in the original model: (1) an online Hadamard transform before the FFN down-projection in every layer, (2) a head-wise Hadamard rotation on queries and keys after RoPE in every attention layer, and (3) a cross-head Hadamard transform after the attention computation in every layer. Each of these operations has a fixed per-token cost that does not scale with batch size or model dimension in the same way as the matrix multiplications they surround. The paper reports that the Hadamard transform overhead is ~6.5% of the INT4 matmul time for the down-projection (Table 14), but this is measured in isolation for a single matrix size. The cumulative latency impact across all layers, including the key/query/value rotations and cross-head transform, is never reported in a full per-token latency breakdown.

The consequence. The online Hadamard transforms, while individually cheap, are serial operations that must be completed before the subsequent quantized matmul can begin. In a latency-sensitive deployment (e.g., streaming token generation where each millisecond of per-token time matters), these transforms add fixed overhead that partially offsets the matmul speedup. The paper's prefill speedup numbers (1.97–3.33×) already include this overhead (the benchmarks measure full block forward pass time), but the distribution of time across operations — what fraction goes to the Hadamard transforms vs. the GEMM vs. the quantization/dequantization — is not reported. For a practitioner optimizing a deployment pipeline, knowing which operations are the bottleneck after QuaRot is essential for further optimization (e.g., whether to invest in faster Hadamard kernels or focus on GEMM tuning). The cross-head Hadamard transform ( $H_{n_h} \otimes I_{d_h}$ ) in particular requires a reshape operation that may have non-trivial cost on GPU due to memory layout changes; its individual cost is never benchmarked. The key/query rotation at each decoding step — though argued to be cheap because it applies only to the single new query — still adds a fixed per-token overhead that, for very short sequences, could be comparable to the attention computation itself.

What evidence exists. Table 14 benchmarks the down-projection matmul with and without Hadamard, showing 6.5–7.0% overhead for that specific operation. Table 15 benchmarks the attention decode with and without Hadamard for the KV cache (the numbers include the key/query/value rotations). But there is no per-operation latency breakdown showing the relative cost of each Hadamard transform, each quantization step, each dequantization step, and each matmul. This makes it difficult to identify optimization priorities or to understand how the latency profile changes across different model sizes and sequence lengths.

Mitigation status. Partially addressed by the integrated benchmarks (Tables 14–17) which include all overheads in the final speedup numbers. A practitioner can trust the aggregate speedup figures for the specific hardware and configurations tested. However, the lack of per-operation breakdown means that extrapolating to different hardware (with different relative costs for matmul vs. memory vs. element-wise operations) is unreliable, and bottleneck analysis for further optimization is unsupported.

7. Implications and Future Directions

How This Work Changes the Landscape

QuaRot introduces a methodological reframing of the LLM quantization problem that shifts attention from quantizer design to representation design. Before this work, the field's response to activation outliers was uniformly reactive: build quantizers sophisticated enough to tolerate them (LLM.int8()'s mixed-precision, SmoothQuant's per-channel scaling, Atom's complex mixed-precision kernels, QUIK's selective high-precision channels). Each of these approaches added engineering complexity — irregular memory layouts, custom kernels, hyperparameters controlling which channels receive special treatment — and none achieved uniform 4-bit end-to-end inference without some form of outlier accommodation. The conceptual contribution of QuaRot is the demonstration that this entire line of work was solving a self-imposed problem: outliers are an artifact of the representation basis, not an intrinsic property of the computation, and a lossless orthogonal rotation can eliminate them without changing what the model computes.

This is not a paradigm shift in the Kuhnian sense — the underlying mathematics (Hadamard transforms, computational invariance) were known, and rotations for weight quantization were introduced by QuIP and QuIP#. But it is a diagnostic reframing with substantial practical consequences. The key insight is that the three points where QuaRot inserts rotations — the residual stream (Stage 1a), the FFN intermediate activations (Stage 1b), and the attention key/value heads (Stages 1c–1d) — correspond exactly to the three points where nonlinear operations (RMSNorm, SiLU gating, and softmax attention) can re-concentrate signal energy into specific dimensions. This transforms the outlier problem from a mysterious empirical nuisance into a predictable consequence of transformer architecture: nonlinearities create structure in the representation, and rotations destroy it. Future work on activation quantization can now operate with this diagnostic: if outliers persist after a rotation, a nonlinear operation downstream of the rotation is re-creating them, and an additional rotation is needed there.

The paper also resolves a contradiction in the KV cache quantization literature. Prior work (KVQuant, KIVI) documented that keys exhibit severe outliers requiring complex handling while values are somewhat easier, and built increasingly elaborate quantization schemes specific to the KV cache. QuaRot demonstrates that these outliers are eliminated by the same mechanism — head-wise Hadamard rotations — as activation outliers. The KV cache problem was never a separate problem; it was the same problem occurring in a different tensor. The fact that 4-bit KV cache quantization with QuaRot achieves 5.51 WikiText-2 perplexity on LLAMA2-7B (only 0.04 above the FP16 baseline of 5.47, Table 6) using simple asymmetric per-group quantization — while prior specialized KV cache methods needed per-channel or non-uniform schemes for similar accuracy — confirms this unification. The implication is that the field can stop developing separate quantization strategies for different tensors (weights, activations, KV cache) and instead apply a unified rotation-then-quantize pipeline, with rotation points chosen based on network topology rather than tensor identity.

Perhaps the most underappreciated implication is what this work reveals about the intrinsic compressibility of transformer representations. The fact that a randomized Hadamard rotation — which spreads information uniformly across all dimensions — enables lossless 6- and 8-bit quantization with no calibration data and no hyperparameter tuning (Table 9: 8-bit RTN on LLAMA2-70B achieves 3.33 perplexity, identical to FP16) suggests that the FP16 representation in pretrained LLMs is highly redundant in an information-theoretic sense. The model stores information in a sparse, structured basis (with outliers) that happens to be terrible for uniform quantization, but the actual information content is low-dimensional enough to survive aggressive uniform compression once the basis is randomized. This has implications beyond quantization: it suggests that the outlier structure is not serving a useful computational purpose (the model works identically when it's removed) and may be an artifact of training dynamics (possibly related to how gradients flow through RMSNorm and attention). Investigating why pretrained LLMs develop outlier-rich representations when outlier-free representations are equally expressive could yield insights into optimization and architecture design.

The work also redirects research priorities away from quantizer complexity and toward representation preprocessing. The evidence is stark: SmoothQuant at 4-bit achieves 83.12 WikiText-2 perplexity on LLAMA2-7B (Table 1) despite sophisticated per-channel smoothing, while QuaRot achieves 6.10 with the simplest possible quantizer (round-to-nearest with a fixed clipping ratio). The 2–3 orders of magnitude gap comes entirely from the representation transformation, not the quantization algorithm. A research program that invested in increasingly complex quantizers to close this gap would be solving the wrong problem. The paper makes a compelling case that future effort should go into: (a) understanding why certain architectures (LLAMA-3, Phi-3) are more resistant to rotation-based outlier elimination, (b) designing rotation strategies tailored to specific architectural components (mixture-of-experts, alternative attention patterns), and (c) developing cheaper rotation primitives (faster Hadamard variants, structured random rotations with similar incoherence properties).

Finally, the paper establishes a new performance frontier for uniform 4-bit inference that recalibrates expectations for what is achievable. Before QuaRot, the best 4-bit joint quantization methods either collapsed completely (SmoothQuant) or required high-precision channels (QUIK: 256 outlier features, Atom: 128). QuaRot achieves better accuracy than all prior methods with zero high-precision channels and uniform quantization throughout. This is a qualitative threshold: it means that end-to-end 4-bit inference on commodity hardware (with standard INT4 Tensor Core instructions) is now demonstrably viable for LLAMA-2 class models, without requiring hardware support for mixed-precision or sparse computation. The door is open for hardware designers to optimize for uniform low-bit inference knowing that the algorithmic barrier has been breached.

Follow-Up Research This Work Enables

Diagnosing and repairing the LLAMA-3 sensitivity gap. The most pressing open question from the paper is why LLAMA3-70B loses 3.80 WikiText-2 perplexity under 4-bit QuaRot while LLAMA2-70B loses only 0.47 — an 8× difference (Tables 1 and 11). A strong follow-up would systematically isolate the source by measuring: (a) the transformation error (perplexity of the rotated-but-unquantized model vs. the original — is the rotation itself lossy in finite precision?), (b) the outlier distribution before and after rotation at each of the three rotation points (FFN intermediate activations, attention values, attention keys — do outliers persist after rotation in LLAMA-3 where they don't in LLAMA-2?), (c) the per-component quantization sensitivity (quantize only weights, only activations, only KV cache, and pairwise combinations to identify which component is driving the degradation), and (d) whether architectural differences — LLAMA-3 uses a different tokenizer (128K vocabulary vs. 32K for LLAMA-2), different training data mixture, and potentially different initialization or optimization — correlate with the sensitivity. A particularly informative negative result would be: if LLAMA-3 outliers are eliminated by the rotations (the activation distributions look as clean as Figure 1 right) but the perplexity still degrades, then the problem is not outliers but some other aspect of the representation (e.g., the rotations might be scrambling structure that the model actually uses for computation, violating the assumption that any orthogonal basis is equally good). This would refine our understanding of when computational invariance holds in practice vs. in theory.

Adaptive rotation placement guided by outlier diagnostics, not fixed architectural heuristics. QuaRot places rotations at three fixed points based on reasoning about where nonlinearities occur. But this reasoning has not been empirically validated: do outliers actually re-emerge after RMSNorm? After the SiLU gate? After softmax attention? A follow-up study could measure the activation distribution (kurtosis, max-to-median ratio, or a formal outlier metric) at every point in the transformer forward pass — after each linear layer, after each normalization, after each activation function, after attention — both in the original model and after each QuaRot rotation. This would produce an "outlier map" of the transformer, showing exactly where the representation drifts away from uniformity. The hypothesis is that QuaRot's three rotation points correspond to the three peaks in this map. If additional peaks are found (e.g., after the attention output but before the residual add, or in the residual stream immediately after the residual add), then additional rotations might further improve quantizability. If some of QuaRot's rotations are at points where no outliers exist (the distribution is already uniform), then they are unnecessary overhead and could be removed. The deliverable would be a diagnostic procedure — analogous to profiling a program before optimizing — that tells a practitioner exactly which layers and which points within layers need rotation for a given model architecture, rather than applying a one-size-fits-all template. This would directly address the LLAMA-3 sensitivity gap: the diagnostic might reveal that LLAMA-3's outlier patterns differ from LLAMA-2's in structure (not just magnitude), explaining why the QuaRot template underperforms.

Combining QuaRot with 2-bit weight quantization methods (QuIP#, AQLM) for end-to-end sub-4-bit inference. QuaRot achieves 4-bit end-to-end quantization; the natural extension is to 3-bit or 2-bit. The paper already shows that 3-bit KV cache is viable (Table 6: 3-bit keys + 3-bit values achieves 5.68 perplexity on LLAMA2-7B with KV cache only quantized, a 0.21 loss), and that 3-bit weight quantization with QuaRot rotations achieves 6.09 perplexity on LLAMA2-7B (Table 7, weight-only). A strong follow-up would combine QuaRot's activation and KV cache rotations with an advanced 2-bit weight quantization method like QuIP# (Tseng et al., 2024) or AQLM (Egiazarian et al., 2024), targeting 2-bit weights + 4-bit activations + 3-bit KV cache. The key question: do the QuaRot rotations, which were designed for activation outlier elimination, also improve the incoherence of weight matrices in a way that complements lattice-codebook or vector quantization methods? QuIP# already uses incoherence processing (random orthogonal rotations applied to weights) independently of QuaRot's rotations; combining them might be redundant or might provide additional benefit. The experiment would compare: (a) QuIP# alone (rotations for weight quantization, FP16 activations), (b) QuaRot alone (rotations for activations + GPTQ for weights), and (c) the combination, measuring both perplexity and zero-shot accuracy. A negative result — the combination provides no benefit over QuIP# alone — would suggest that weight and activation incoherence are largely independent and that separate rotation schemes for each are sufficient. A positive result would suggest that a unified rotation strategy targeting both weights and activations simultaneously is more effective, potentially enabling uniform sub-4-bit inference that neither method achieves alone.

Online difficulty estimation with adaptive bit-width — spending quantization budget where it matters. QuaRot quantizes all layers and all tokens uniformly to 4 bits. But the sensitivity to quantization likely varies across layers and across tokens — some layers may be more quantization-sensitive than others, and some tokens (e.g., those carrying rare information) may need higher precision than others. A strong follow-up would develop a lightweight "quantization difficulty" predictor — perhaps a small network or a simple statistic computed from the activation distribution — that estimates, for each layer and each token, the minimum bit-width needed to preserve the output within some tolerance. During inference, the system could dynamically allocate a quantization budget: easy layers/tokens get 2–3 bits, difficult layers/tokens get 4–6 bits, and the average bit-width meets a target (e.g., 4 bits). This is analogous to the compute-optimal test-time scaling strategy, but for quantization precision rather than generation budget. QuaRot makes this newly tractable because the rotated activations have simple, predictable distributions (approximately Gaussian), making it easier to estimate quantization error analytically (e.g., from the variance of the distribution) without requiring empirical measurement for every token. The experiment would measure the accuracy-vs.-average-bit-width Pareto frontier for mixed-precision QuaRot compared to uniform 4-bit, and quantify how much additional compression (or accuracy recovery) is achievable. A key challenge: current hardware has limited support for dynamic per-token bit-width in GEMM kernels (Tensor Cores expect uniform bit-width for a given matmul), so the practical impact depends on whether the adaptive scheme can be implemented with layer-wise or block-wise granularity (coarse enough for hardware efficiency, fine enough for accuracy gains).

Extending computational invariance to mixture-of-experts and non-transformer architectures. The computational invariance argument relies on the pre-norm transformer structure where RMSNorm divides by the norm and orthogonal rotations preserve the norm. Mixture-of-experts (MoE) architectures add a routing mechanism between the FFN and the residual stream — the router selects which experts to activate based on the hidden state. If the hidden state is rotated by Q, the router's input changes from X to XQ, and unless the router is also transformed (or is rotationally invariant), the expert selection will change, breaking computational invariance. Investigating whether routers can be made invariant to orthogonal rotations — perhaps by applying the same rotation to the router weights or by designing routers that operate on rotation-invariant features (e.g., norms, pairwise distances) — would extend QuaRot to MoE models. Similarly, non-transformer architectures (state-space models like Mamba, linear attention models like RetNet) have different computational structures where the "computational invariance" principle might or might not apply; identifying which architectures admit lossless rotation-based outlier elimination and which do not would establish the scope boundary of the approach. The experiment for MoE would be: (1) apply QuaRot's between-block rotation (Stage 1a) to an MoE model, (2) measure whether the routing decisions change (what fraction of tokens get routed to different experts?), (3) if routing changes significantly, measure the perplexity degradation and investigate whether the degradation comes from the routing change or from something else. A negative result — MoE routing is fundamentally incompatible with orthogonal rotations — would establish an important boundary condition for the approach. A positive result — perhaps by rotating the router weights as well — would extend QuaRot's applicability to the growing class of MoE models (Mixtral, DeepSeek-MoE, etc.).

Training with rotations from initialization to produce natively quantization-friendly models. QuaRot applies rotations post-hoc to a pretrained model. But if rotations make the representation uniformly distributed without harming expressiveness (as evidenced by the 8-bit RTN lossless results), then training from scratch with rotations baked into the architecture should produce models that are natively easy to quantize, without needing the post-hoc transformation step. The experiment would: (1) take a transformer training codebase, insert the QuaRot rotation scheme as a fixed part of the forward pass (the rotations are not learned, they're fixed Hadamard matrices), (2) train a model from scratch with these rotations in place (so the model learns in the rotated basis), (3) measure whether the trained model achieves the same perplexity as an unrotated baseline at equal compute, and (4) measure whether the rotated-trained model can be quantized to 4 bits with even lower degradation than post-hoc QuaRot (since the model never learned the outlier-rich basis in the first place). This draws inspiration from Xi et al. (2023), who used exact Hadamard transformations during training for 4-bit training, but extends it to the full QuaRot scheme (including attention rotations) and evaluates quantizability at inference time rather than training dynamics. A positive result — rotated training produces models that quantize to 4 bits with <0.1 perplexity loss — would make quantization a training-time concern rather than a post-training patch, simplifying deployment to a single step (just quantize the already-rotated weights). A negative result — rotated training hurts convergence or final perplexity — would suggest that the outlier-rich basis serves an optimization purpose during training (e.g., outliers might help gradient flow through certain layers), which would be a fundamental insight about transformer training dynamics.

Practical Applications and Downstream Use Cases

On-device deployment of LLMs with tight memory budgets. The most direct application is deploying large models on consumer hardware with limited GPU memory. A single NVIDIA RTX 3090 has 24 GB of VRAM. LLAMA2-70B in FP16 requires approximately 140 GB just for the weights (70B parameters × 2 bytes), making it impossible to load. With 4-bit weight quantization (70B × 0.5 bytes = 35 GB) plus 4-bit KV cache (reducing per-token cache size by 4×), the model becomes feasible on multi-GPU consumer setups or high-memory workstations. QuaRot's 3.89× peak memory saving during decoding (Table 17) means that for a given memory budget, either ~4× larger models can be served, or ~4× longer context lengths can be supported. This is particularly impactful for applications where users want to run models locally for privacy or latency reasons — QuaRot makes LLAMA2-70B class models accessible on hardware that previously could only run 7B or 13B models. The 99% zero-shot accuracy retention on 70B (Table 2) means the quality is nearly indistinguishable from FP16 for most tasks, making the memory savings genuinely usable rather than a degraded fallback.

High-throughput batch inference serving with cost reduction proportional to speedup. For API providers or enterprises running large-scale batch inference (evaluating prompts across millions of examples), the 3.33× prefill speedup on LLAMA2-70B (Table 16, batch size 64) translates directly to ~3× cost reduction: the same hardware can process ~3× more prompts per unit time, or equivalently, the same throughput can be achieved with ~3× fewer GPUs. The speedup increases with batch size (2.16× at batch size 1, 3.33× at batch size 64 for LLAMA2-70B), meaning the method is particularly well-suited for throughput-oriented deployments that can batch many requests together. The caveat is that the 4-bit KV cache is slower than FP16 at small batch sizes (Table 15: 45% slower at batch size 1 for LLAMA2-7B attention), so the deployment should use adaptive precision — INT4 for batched inference, FP16 for interactive single-query serving — or accept a latency penalty for the memory savings. The 8-bit RTN results (complete losslessness with no calibration data, Table 9) provide a simpler alternative for deployments that need guaranteed no-loss compression: 8-bit QuaRot-RTN can be applied to any model with zero tuning and achieves identical accuracy to FP16, while still providing 2× memory and potentially 2× speedup (though 8-bit speedup numbers are not reported in the paper).

Long-context inference where the KV cache dominates memory. For applications requiring very long context windows (e.g., 32K–128K tokens for document analysis, codebase understanding, or multi-turn dialogue), the KV cache becomes the dominant memory consumer. With FP16 KV cache, a single 128K-token sequence in LLAMA2-7B (32 heads, 128 dim, 2 bytes per element) requires approximately 32 heads × 128 dim × 128K tokens × 2 bytes × 2 (keys + values) = 2 GB for the KV cache alone for one sequence. With 4-bit quantization, this drops to 0.5 GB — a 4× reduction. QuaRot's KV cache quantization achieves this with only 0.04 perplexity loss (Table 6: 5.51 vs. 5.47 on LLAMA2-7B), meaning long-context quality is essentially preserved. For batch serving with many concurrent long-context requests, the memory savings compound. This enables either serving more concurrent users with the same hardware, or extending the maximum context length on existing hardware. The post-RoPE caching design (the rotation is applied once when the key is generated, not redone at every step) is particularly important here: the cost of rotating keys does not scale with sequence length during decoding, making the approach practical for very long contexts where pre-RoPE caching (as in KVQuant) would incur prohibitive per-step overhead.

Edge deployment of instruction-tuned models for real-time interaction. For edge devices (laptops, mobile phones, embedded systems) running instruction-tuned models for real-time interaction, both memory and compute are constrained. QuaRot's combination of weight compression (reducing model loading time and memory footprint), activation quantization (accelerating the prompt processing phase when the user submits a query), and KV cache quantization (reducing memory pressure during token generation) addresses all three bottlenecks simultaneously. For a model like Phi-3-mini (3.8B parameters, evaluated in Appendix A.9), 4-bit QuaRot achieves 7.85 WikiText-2 perplexity (vs. 6.35 FP16 — a 1.50 loss) and 66.34% zero-shot average (vs. 73.11% — a 6.77% loss). Whether this quality is acceptable depends on the application; for many edge use cases (on-device assistants, text classification, simple Q&A), a 6–7% accuracy reduction might be acceptable for the memory and speed benefits. For higher-quality requirements, 6-bit QuaRot on Phi-3-mini achieves 6.63 perplexity (only 0.28 loss) and 72.67% zero-shot accuracy (0.44% loss) — essentially lossless. The key deployment architecture: use 6-bit QuaRot-RTN (calibration-free) for the weights, 8-bit for activations, and 4-bit for the KV cache, getting a mix of speed and accuracy tuned to the specific edge device constraints.

When to Prefer This Method

The paper provides clear empirical guidance on when QuaRot is the right tool versus when simpler alternatives suffice, though it does not frame it as an explicit decision rule. The following is grounded in the paper's own results:

Prefer QuaRot for 4-bit end-to-end inference on LLAMA-2 class models when uniform quantization is required. QuaRot is uniquely positioned as the only method (as of publication) that achieves usable 4-bit joint quantization (weights + activations + KV cache) with zero high-precision channels on LLAMA-2 (Table 1). If the deployment requires 4-bit throughout — for maximum speedup from INT4 Tensor Cores and maximum memory savings — and the model is LLAMA-2 or similar, QuaRot with GPTQ weight quantization is the only demonstrated path. For LLAMA2-70B, the quality loss is minimal (0.47 perplexity, 1% zero-shot accuracy).
Prefer QuaRot-RTN for lossless 6- or 8-bit compression when calibration data is unavailable or undesirable. Table 9 demonstrates that 8-bit RTN on LLAMA2-70B is literally lossless (3.33 perplexity matches FP16, 77.17% zero-shot accuracy exceeds 77.07% baseline), and 6-bit RTN is nearly so. No calibration data, no hyperparameter search, no GPTQ — just apply the QuaRot transformations (5 minutes on an A100, Section 5), round the weights to the nearest integer in 6- or 8-bit representation, and deploy. This is a strong practical recommendation for any deployment where guaranteed no-loss compression is needed.
Prefer GPTQ weight-only quantization (without QuaRot) if activations remain in FP16 and only weight memory savings matter. QuaRot's main contribution is solving the activation outlier problem to enable joint quantization. If the deployment only needs to compress weights (keeping activations in FP16), standard GPTQ without rotations achieves competitive weight-only perplexity (Table 7: GPTQ A16W4 is 8.25 on LLAMA2-7B vs. QuaRot-GPTQ at 5.60 — but note QuaRot helps significantly here too). The decision reduces to whether the implementation complexity of QuaRot's transformation pipeline is justified by the additional weight-only gains, which are substantial at 4-bit (2.65 perplexity improvement on 7B) but diminish at higher bit-widths.
Be cautious with QuaRot on models not in the LLAMA-2 family, and run a diagnostic evaluation first. The LLAMA-3 results (Tables 11–12: 3.80 perplexity loss on 70B) show that QuaRot's effectiveness is not universal. For a new model architecture, the paper provides no predictive diagnostic, so the practitioner should: (a) measure the 4-bit perplexity with RTN weight quantization (cheap, no calibration) to estimate the best-case degradation without GPTQ optimization, (b) measure the transformation error (perplexity of the rotated unquantized model vs. original) to check whether the rotation itself is lossy in the model's native precision, and (c) compare against a simple baseline like 4-bit GPTQ weight-only with FP16 activations (which avoids the activation quantization risk entirely) to determine whether the additional speedup from activation quantization is worth the potential accuracy degradation.
For interactive single-query inference (batch size 1), consider keeping the KV cache in FP16 or using hybrid precision for recent tokens. The attention decode micro-benchmarks (Table 15) show that 4-bit KV cache is 45% slower than FP16 at batch size 1 on LLAMA2-7B. In latency-sensitive applications (chatbots, code completion), using a 4-bit KV cache may regress per-token generation time. The memory savings might still be worth it if the alternative is not being able to serve the model at all (e.g., memory-constrained edge deployment), but for cloud serving where memory is plentiful, FP16 KV cache with 4-bit weights-only may provide a better latency-accuracy-memory Pareto point.