Training Transformers with 4-bit Integers

ArXiv: 2306.11987

🎯 Pitch

Transformers can be trained using only 4-bit integer arithmetic, achieving up to 2.2× speedup over FP16 on existing GPUs—no custom number formats needed. The key is a Hadamard transform that tames destructive activation outliers in the forward pass, combined with a sampling technique that exploits extreme gradient sparsity in the backward pass to preserve accuracy.

---

1. Executive Summary

This paper proposes a hardware-friendly INT4 training algorithm for transformers that implements all matrix multiplications with integer arithmetic. Evaluated on language model fine-tuning (BERT, GLUE/SQuAD), machine translation (Transformer-base, WMT 14 En-De), and image classification (ViT, CIFAR/ImageNet), the method combines a Hadamard quantizer for forward propagation (applying a block-diagonal Hadamard transform to suppress activation outliers before quantization, reducing their numerical range) with bit splitting and leverage score sampling for backpropagation (splitting gradients into higher and lower 4-bit components, then sampling only the most informative rows via an importance-sampling procedure based on per-row norms). The proposed INT4 linear operators achieve up to 2.2× speedup over FP16 matrix multiplication and up to 35.1% end-to-end training speedup, while maintaining competitive accuracy—recovering 5.5 percentage points over the LSQ+LUQ 4-bit baseline on GLUE with BERT-base. The method's hardware compatibility distinguishes it from prior 4-bit training approaches, establishing that custom numerical formats (FP4, logarithmic) are not required for accurate low-precision training only when specialized quantizers address activation outliers in the forward pass and gradient sparsity in the backward pass.

2. Context and Motivation

The Core Problem: Training Neural Networks at Ultra-Low Precision Without Custom Hardware

The fundamental question this paper tackles is: can we train transformer neural networks using only 4-bit integer arithmetic for all matrix multiplications, while maintaining competitive accuracy and achieving real speedup on existing GPU hardware? This matters because training modern neural networks—particularly large language models and vision transformers—is enormously computationally expensive. A single training run for a large model can consume millions of GPU-hours and emit hundreds of tons of CO₂. Reducing the numerical precision of training from the standard 16-bit floating-point (FP16) to 4-bit integers (INT4) would, in principle, slash both computation and memory requirements by roughly 4×.

However, this transition has proven surprisingly difficult. The paper identifies the central tension: "Training with an ultra-low INT4 precision is challenging" (Section 1). The challenge is not merely about getting training to run at 4 bits—it's about getting it to converge to the same accuracy as full-precision training, and doing so on hardware that people actually have.

Why This Problem Matters Now

The paper's motivation rests on several converging trends that make INT4 training both timely and practically significant:

The scaling imperative. As models grow larger (BERT-Large at 340M parameters, GPT-3 at 175B, and beyond), training costs scale proportionally. FP16 training with mixed precision (Micikevicius et al., 2018) has been the workhorse for years, and FP8 training recently became available on Nvidia's H100 GPU through the Transformer Engine (Nvidia, 2023). But each step down in precision—FP32 → FP16 → FP8—has required either new hardware capabilities or carefully designed numerical formats. Pushing to 4 bits represents the next frontier, but it's a frontier where the quantization error becomes so severe that training can easily destabilize or diverge entirely.

The hardware compatibility gap. Prior 4-bit training methods (Sun et al., 2020; Chmiel et al., 2021) demonstrated that training at 4 bits is conceptually possible, achieving reasonable accuracy on several benchmarks. However, they relied on custom numerical formats—FP4 (4-bit floating-point) or logarithmic number systems—that are "not supported by contemporary hardware" (Section 1). This creates a frustrating situation: the algorithms exist on paper but cannot actually accelerate training on real GPUs, which have INT8 and INT4 tensor core instructions but no FP4 or log-format support. The paper's explicit goal is to bridge this gap—to design an algorithm that both trains accurately and maps onto existing INT4 tensor core hardware (Section 1):

"Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs."

The environmental and economic case. The "Broader Impacts" section states the motivation directly: "Our algorithm can improve efficiency and reduce the energy consumption of training neural networks, which helps reduce the carbon footprint caused by deep learning." While this is a standard framing in efficient ML papers, it reflects a genuine urgency—training a single large model can emit as much carbon as several cars over their lifetimes, and any reduction in the per-FLOP energy cost has compounding benefits at scale.

The Specific Technical Gap: Why Naïve 4-Bit Quantization Fails

The paper frames the problem through two specific failure modes that prevent straightforward application of existing quantization techniques to 4-bit training:

1. Activation outliers in forward propagation. Section 3.2 explains that transformer activations exhibit a characteristic "feature-wise structure" where a small number of dimensions (columns of the activation matrix) contain entries that are dramatically larger in magnitude than all others. Figure 1(a) shows this vividly: the histogram of activation values has a long tail of outliers, with some entries being orders of magnitude larger than the typical value.

This creates a fundamental trade-off in static quantization. The learned step size quantizer (LSQ, Esser et al., 2019) uses a single scalar $s_X$ to scale all entries of an activation matrix into the integer range $[-7, 7]$ (for 4 bits). If $s_X$ is large enough to accommodate the outliers without truncation, then the vast majority of normal-valued entries get squeezed into just one or two quantization bins—they lose almost all their information. If $s_X$ is small enough to give fine granularity to normal entries, the outliers get clamped (truncated), destroying information that the paper notes transformers "tend to store... in these outliers" (Section 3.2).

This is not a minor edge case. The paper shows in Section 5.2 (Figure 3a) that if you simply maintain the top 1% of activation entries in full precision (the "outlier" ablation), accuracy recovers dramatically compared to uniform 4-bit quantization. The outliers are the bottleneck for forward propagation.

2. Gradient imprecision and structural sparsity in backpropagation. Section 4.1 identifies a second, independent challenge: activation gradients $\nabla Y$ exhibit extreme "structural sparsity." Figure 2 shows that a small number of tokens (rows of $\nabla Y$) have very large gradient norms, while the majority of tokens have gradients near zero. This arises from overparameterization—the model fits most training examples well (producing near-zero gradients), while a few "hard" examples generate substantial gradient signals.

Quantizing such a sparse, heavy-tailed distribution to 4 bits with a uniform quantizer is inaccurate: either the large gradients get truncated (losing critical optimization signal), or the small gradients get coarsely approximated (injecting noise that can slow or destabilize training). The paper phrases this insight as an efficiency opportunity (Section 4.2): "many rows of the gradient are so small that they have little impact on the parameter gradient, yet they waste abundant computation." The key is to redirect computational budget from the small, uninformative gradients to representing the large gradients more accurately.

Why these problems are harder for training than inference. The paper draws an important distinction in Section 3.1 (Remark) between quantization-aware training (QAT) for inference and fully quantized training (FQT). QAT can use a pre-trained full-precision model as initialization, employ knowledge distillation, adopt multi-stage training pipelines, and approximate discrete quantizers with continuous functions during training. FQT has none of these luxuries: it must train from scratch (or from a pre-trained checkpoint, but without a full-precision teacher guiding every step), converge as fast as full-precision training, and use truly discrete integer arithmetic. This makes 4-bit FQT substantially more challenging than 4-bit QAT, and explains why the field had not previously achieved hardware-compatible INT4 training.

Where Prior Approaches Fall Short

The paper positions itself against three categories of prior work, identifying specific limitations of each:

Fully quantized training at higher bit-widths. The FQT literature has progressively reduced training precision: FP16 mixed precision (Micikevicius et al., 2018), FP8 (Wang et al., 2018; Sun et al., 2019), INT32+INT8 (Banner et al., 2018), and INT8+INT5 (Chen et al., 2020). These methods work well but leave a factor of 2–4× efficiency on the table compared to INT4. The paper does not argue these methods are flawed—rather, the techniques developed for 8-bit training do not directly generalize to 4 bits because the quantization noise floor rises dramatically with each bit removed. As Section 2 notes, the "vast numerical range of the gradient and the optimization issues of training quantized networks from scratch" become acute at 4 bits.

Existing 4-bit training methods: accurate but hardware-incompatible. Two prior works reached the 4-bit frontier:

• Sun et al. (2020) ("Ultra-low" in the paper's experiments) trained networks with INT4 activations and weights but FP4 gradients. The FP4 format provides non-uniform quantization levels that better capture the heavy-tailed gradient distribution, but FP4 tensor cores do not exist on current GPUs.
• Chmiel et al. (2021) ("LUQ") proposed logarithmic unbiased quantization using a custom 4-bit logarithmic numerical format. Logarithmic quantization places more quantization levels near zero (where most values lie) and fewer at large magnitudes, which is well-suited to distributions like gradients. However, like FP4, this custom format cannot leverage INT4 tensor core instructions.

Table 1 quantifies the accuracy gap: on GLUE with BERT-base, LSQ+LUQ achieves only 75.29 average score versus 82.67 for full-precision training—a 7.4-point degradation. On BERT-large, the drop is catastrophic: 55.93 versus 84.57, nearly 29 points. These methods prove 4-bit training is possible but leave two problems unsolved: the accuracy gap is too large for practical deployment, and the custom formats prevent any actual speedup.

Post-training quantization and outlier-handling methods: not designed for training. Section 3.2 acknowledges several works on handling activation outliers, including Outlier Suppression (Wei et al., 2022) and SmoothQuant (Xiao et al., 2022). These methods were developed for post-training quantization (PTQ) or QAT for inference, typically targeting 6–8 bits. The paper notes they "seldom successfully deal with ultra-low 4-bit training." SmoothQuant, for instance, migrates quantization difficulty from activations to weights by applying per-channel scaling factors, but the paper's ablation (Appendix C.3, Table 4) shows that SmoothQuant combined with LSQ achieves zero accuracy at 2–5 bits on CoLA. These PTQ techniques address a different problem—compressing an already-trained model—and do not account for the dynamic distribution shifts, gradient noise accumulation, and optimization challenges of training from scratch.

How This Paper Positions Itself

The paper frames its contribution through a specific design philosophy that sets it apart from prior work: specialized quantizers that exploit the structure of transformers, rather than general-purpose numerical formats. As stated in Section 1:

"This MM form allows us to design more flexible quantizers, which better approximate FP32 matrix multiplications by utilizing specific structures of the activations, weights, and gradients in transformers."

This is a different intellectual approach. Instead of asking "what numerical format minimizes quantization error for arbitrary tensors?", the paper asks "what are the specific properties of activations and gradients in transformers, and how can we design quantizers that exploit those properties?" The answer leads to two specialized techniques:

• Hadamard quantization (HQ) for forward propagation: rather than trying to represent outliers more accurately in the original space, apply a Hadamard transform to "spread" outlier information across many matrix entries, making the transformed matrix easier to quantize uniformly. This exploits the feature-wise structure of activation outliers.
• Bit splitting + leverage score sampling (LSS) for backpropagation: rather than using a non-uniform quantizer to capture the heavy-tailed gradient distribution, split gradients into higher and lower 4-bit components (an INT8 representation), then probabilistically drop the small components using importance sampling. This exploits the row-wise sparsity of gradients.

The paper also explicitly positions itself within the framework of randomized numerical linear algebra (RandNLA, Drineas and Mahoney, 2016), citing it in Section 1. The leverage score sampling technique directly descends from RandNLA's importance sampling methods for approximate matrix multiplication, giving the approach theoretical grounding beyond empirical heuristics. Proposition 4.1 formalizes the variance of the LSS estimator and shows it is minimized when sampling probabilities are proportional to the leverage scores $c_i = \|\nabla^{\updownarrow}_{Y i,:}\| \|X^{\updownarrow}_{i,:}\|$.

A crucial aspect of the paper's positioning is that it targets transformers specifically, not all neural networks. Section 6 (Limitations) explicitly acknowledges this: "The main limitation of this work is that it can only accelerate models with a large portion of matrix multiplications (linear layers), but can not accelerate convolution layers." This is not a failure—it's a deliberate scoping. Transformers consist almost entirely of matrix multiplications (in fully-connected layers and attention BMMs), making them an ideal target for MM-focused quantization. The techniques exploit transformer-specific properties (feature-wise outliers from LayerNorm interactions, token-wise gradient sparsity from overparameterization) that may not transfer to CNNs. The paper also notes in the Conclusions that extension to "other MM-only architectures, such as MLP-Mixer, graph neural networks, and recurrent neural networks" is a natural future direction.

The Hardware Compatibility Thesis

Perhaps the paper's most distinctive position is its insistence on contemporary hardware compatibility as a first-class design constraint. Section 1 states this directly:

"Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs."

This is not merely a practical convenience—it reflects a belief that the gap between algorithmic innovation and hardware reality has hindered the adoption of low-precision training. The paper backs this claim with concrete implementation: a prototype CUDA operator using Nvidia's cutlass library that achieves 2.2× speedup over FP16 matrix multiplication on an RTX 3090 (Section 5.3, Figure 4). The authors are candid that this implementation is "not fully optimized" and "a fully optimized implementation requires heavy engineering," but the existence of a working prototype that actually accelerates training (35.1% end-to-end on BERT, Figure 5) validates the thesis that INT4 training can be both accurate and fast.

This distinguishes the work from papers that report simulated efficiency (counting theoretical INT4 operations without implementing them) or that rely on hardware that doesn't exist yet. The constraint of fitting into INT4 tensor core instructions—with their specific data layout requirements (RowMajor/ColumnMajor, alignment to multiples of 32 for K dimension)—shapes the algorithm design, as discussed in Appendix A.6.

3. Technical Approach

3.1 Reader Orientation

This paper is a systems-and-algorithms paper that designs specialized quantizers for forward and backward propagation in transformers to enable training with 4-bit integer matrix multiplications. The core idea is to analyze the specific structure of activations (feature-wise outliers) and gradients (token-wise sparsity) in transformers, then design targeted quantization techniques—a Hadamard transform for the forward pass and bit splitting with importance sampling for the backward pass—that preserve accuracy while mapping efficiently onto existing INT4 tensor core hardware.

3.2 Big-Picture Architecture (Diagram in Words)

The system replaces every floating-point linear operation in a transformer training loop with a 4-bit integer counterpart, while leaving non-linear operations (GeLU, LayerNorm, softmax, etc.) in FP16. The architecture has four major components that interlock across the forward and backward passes:

1. Hadamard Quantizer (HQ) — for forward propagation: applies a block-diagonal Hadamard transform to activations and weights before quantization, spreading outlier information across neighboring entries so that a uniform INT4 quantizer can represent both normal values and outliers without catastrophic truncation. Operates on matrices of shape $X \in \mathbb{R}^{N \times D}$ (activations, where $N$ is the number of tokens and $D$ is the model dimension) and $W \in \mathbb{R}^{C \times D}$ (weights, where $C$ is the output dimension).

2. Learned Step Size Quantizer (LSQ) — the core quantizer embedded within HQ: a static (input-independent) scalar quantizer with a learnable scale parameter $s_X$ or $s_W$ that maps FP16 values into the integer range $[-7, 7]$. The scale is optimized jointly with the model parameters during training.

3. Bit Splitting and Leverage Score Sampling (LSS) — for backward propagation: dynamically quantizes the output gradient $\nabla Y \in \mathbb{R}^{N \times C}$ by splitting it into higher and lower 4-bit components (creating an effective INT8 representation), then probabilistically discards approximately half the tokens (rows) based on importance sampling weights derived from per-row norms, using the saved computation to represent the remaining rows more accurately.

4. Prototype CUDA Operator — a hardware implementation using Nvidia's cutlass library that packs INT4 data, executes tensor-core INT4 matrix multiplications, and handles the non-trivial data layout transformations (RowMajor to ColumnMajor) and alignment constraints (K-dimension multiples of 32) required by the hardware.

Information flows through the system in the standard training loop order: input activations → HQ forward propagation (Hadamard transform + LSQ quantization + INT4 MM + dequantization) → nonlinear operations in FP16 → loss computation → LSS backward propagation (gradient splitting + leverage score computation + probabilistic row sampling + INT4 MM + dequantization) → weight updates. The quantization step sizes ($s_X$, $s_W$) are themselves updated via gradient descent, with their gradients computed efficiently by reusing the MMs already performed.

3.3 Roadmap for the Deep Dive

• First, the formal problem setup in Section 3: the matrix multiplication form that all linear operations in transformers take, and why focusing on $Z = XW^\top$ captures the entire acceleration opportunity.
• Second, the Learned Step Size Quantizer (LSQ) in Section 3.1 — the foundational quantization primitive that both forward and backward methods build upon, including why it is static rather than dynamic and how it interfaces with INT4 hardware.
• Third, the activation outlier problem in Section 3.2 — what outliers are structurally, why they catastrophically degrade 4-bit training, and why prior PTQ/QAT solutions do not transfer to FQT.
• Fourth, the Hadamard Quantizer (HQ) in Section 3.3 — the forward-pass solution: the block-diagonal Hadamard transform, the quantization procedure, and the computational cost model.
• Fifth, the backward propagation formulation in Section 4, covering the gradient structure, the sparsity observation (Section 4.1), and then the LSS mechanism (Section 4.2) — bit splitting, leverage score computation, probabilistic sampling, and the variance-minimizing probability derivation — for both weight gradients and activation gradients.
• Sixth, implementation details and hardware considerations, including how the algorithms map onto cutlass INT4 tensor core operations, the data packing scheme, the quantization overhead breakdown, and the cold-start heuristic for training from scratch.

3.4 Detailed, Sentence-Based Technical Breakdown

Overview: The Paper as a Specialized Quantizer Design

This is an algorithm design paper whose core contribution is two specialized quantization procedures — one for forward propagation (HQ) and one for backward propagation (LSS) — that together enable all matrix multiplications in transformer training to run in INT4 arithmetic with competitive accuracy and real hardware speedup. The two procedures are independent in mechanism but unified by a common philosophy: exploit known structural properties of transformer tensors (feature-wise outliers in activations, token-wise sparsity in gradients) to design quantizers that are more accurate than generic alternatives, while constraining all operations to INT4 to ensure compatibility with existing GPU tensor cores.

---

The Matrix Multiplication Form and Why It Covers Everything

Section 3 opens by establishing that all linear operations in transformers can be written as matrix multiplications (MMs) of the form:

$Z = XW^\top, \quad \text{where } Z \in \mathbb{R}^{N \times C}, \; X \in \mathbb{R}^{N \times D}, \; W \in \mathbb{R}^{C \times D}$

Here $X$ is the activation matrix with $N = ST$ rows (batch size $S$ times sequence length $T$) and $D$ columns (the model hidden dimension), $W$ is the weight matrix with $C$ output and $D$ input dimensions, and $Z$ is the output.

This covers fully-connected layers directly. For attention layers, which involve batch matrix multiplications (BMMs) over multiple heads, the same techniques apply with minor extensions — the Hadamard transform operates independently per batch using a repeated block-diagonal structure, and leverage score sampling operates per batch element (Appendix A.1 confirms this). The paper explicitly states: "we leave the discussion of BMMs in Appendix A.1," meaning the main text focuses on the conceptually simpler MM case and the extensions are straightforward.

Why this focus matters: by reducing the acceleration problem to MM acceleration, the paper can design quantizers that operate at the granularity of entire matrices, exploiting global structural properties (column-wise outliers, row-wise sparsity) that per-element or per-vector quantizers would miss.

---

Learned Step Size Quantizer (LSQ) as the Foundational Primitive

Section 3.1 introduces the Learned Step Size Quantizer (LSQ, from Esser et al., 2019) as the basic quantization building block that both forward and backward procedures use. LSQ is a static, per-tensor quantizer with a learnable scale parameter.

The LSQ mechanism. Given a floating-point matrix $X$, LSQ produces an integer matrix through:

$\text{int}_{s_X}(X) := \left\lfloor \text{clamp}\left(\frac{X}{s_X}, -Q_N, Q_P\right) \right\rceil$

where $s_X \in \mathbb{R}$ is a learnable scalar (the step size or quantization scale), $\text{clamp}(\cdot, a, b)$ restricts each element to the range $[a, b]$, and $\lfloor \cdot \rceil$ rounds to the nearest integer. The division $X / s_X$ is performed elementwise.

For 4-bit quantization, the paper sets $Q_N = Q_P = 7$, giving the signed integer range $\{-7, -6, \ldots, 6, 7\}$ — fifteen representable values plus zero, for sixteen total quantization levels (4 bits).

The integer matrix is dequantized back to floating-point by multiplying by the step size: $\text{float}(\text{int}_{s_X}(X)) = s_X \cdot \text{int}_{s_X}(X) \approx X$. This is an affine mapping (scale only, no zero-point offset) centered at zero.

What this enables downstream. With LSQ applied to both $X$ and $W$, the original matrix multiplication $Z = XW^\top$ can be approximated as:

$Y = XW^\top \approx s_X s_W \cdot \text{int}_{s_X}(X) \cdot \text{int}_{s_W}(W)^\top$

Only the INT4 MM $\text{int}_{s_X}(X) \cdot \text{int}_{s_W}(W)^\top$ needs to run on the tensor cores; the scalar $s_X s_W$ is applied as a single floating-point multiply at the end.

Why LSQ is static rather than dynamic. Dynamic quantization methods compute the scale per input (e.g., as a function of the current tensor's range), which requires scanning the tensor at runtime to find min/max values. Static quantization fixes $s_X$ as a learned parameter that does not depend on the current input. The paper states: LSQ is "cheaper than dynamic quantization methods, which need to compute the quantization scale dynamically per iteration" (Section 3.1). This matters for training throughput — computing dynamic scales adds overhead proportional to the tensor size that grows with model scale, while static scales add only a negligible scalar parameter per quantized tensor.

How LSQ parameters are learned. The step size $s_X$ is a trainable parameter optimized jointly with the model weights via stochastic gradient descent. The straight-through estimator (STE, Bengio et al., 2013) is used for the non-differentiable rounding operation: the derivative of $\lfloor \cdot \rceil$ is treated as 1 everywhere (i.e., $\lfloor x \rceil' = 1$), so gradients flow through the quantizer as if it were the identity function. The step size receives gradients through the chain rule (detailed in Appendix A.3), with the gradient involving the difference between the quantized and unquantized values scaled by the quantizer's sensitivity.

The difference between FQT and QAT (the "Remark" in Section 3.1). The paper explicitly distinguishes fully quantized training (FQT) from quantization-aware training (QAT) for inference:

1. Initialization: QAT can start from a converged full-precision model and/or use knowledge distillation from a full-precision teacher. FQT must train from scratch (or fine-tune a pre-trained model) while operating under quantization noise.
2. Convergence speed: QAT can use expensive multi-stage training pipelines without worrying about training time, since inference-only deployment is the goal. FQT must converge as fast as full-precision training to be useful as a training accelerator — if 4-bit training takes 3× more iterations, the per-iteration speedup is wasted.
3. Quantizer differentiability: QAT can approximate the discrete quantizer with continuous functions during training (e.g., differentiable soft quantization, Gong et al., 2019), which cannot be implemented with integer arithmetic and therefore cannot accelerate training itself. FQT must use truly discrete integer operations.

This distinction explains why simply taking a QAT method designed for 4-bit inference and applying it to FQT fails — QAT methods are not designed for the optimization and speed requirements of training.

---

The Activation Outlier Problem

Section 3.2 identifies activation outliers as the primary bottleneck for forward propagation at 4 bits. The empirical evidence comes from Figure 1(a), which shows a histogram of activation values from a BERT-base-uncased model: the distribution is concentrated near zero but has a long tail of values that are orders of magnitude larger. The paper describes these as "outlier entries, which are much larger in magnitude than other entries."

Structural nature of the outliers. The key observation is that outliers are not randomly scattered — they have a "feature-wise structure" (Section 3.3): "They are typically concentrated on a few dimensions, i.e., only a few columns of $X$ are significantly larger than others." This means the outlier phenomenon is a property of specific feature channels (columns in the activation matrix), likely arising from LayerNorm interactions (as noted by Outlier Suppression, Wei et al., 2022, and SmoothQuant, Xiao et al., 2022).

Why outliers break 4-bit quantization. The step size $s_X$ in LSQ controls a fundamental trade-off between quantization granularity and representable range. The representable range is $[-7 s_X, 7 s_X]$ — any value outside this range is clamped (truncated) to the boundary. The granularity — the spacing between adjacent integer levels — is $s_X$, which controls how precisely normal-magnitude values can be represented.

If $s_X$ is set large enough to include the outliers without clamping, then the granularity $s_X$ becomes coarse relative to typical activation magnitudes. Most entries, which are orders of magnitude smaller than the outliers, would all map to the same quantization bin (typically zero or near-zero), losing their information content entirely.

If $s_X$ is set small to give fine granularity to typical entries, the outliers exceed the representable range and get truncated to $\pm 7 s_X$, destroying the information they carry. The paper states: "the transformers tend to store information in these outliers, and such truncation would seriously harm accuracy" (Section 3.2). The ablation in Section 5.2 confirms this: a method that simply keeps the top 1% largest activation entries in full precision ("outlier" in Figure 3a) recovers accuracy close to full-precision levels, proving that outlier truncation is the dominant failure mode.

Why this is worse for fine-tuning. The paper notes that "the outlier problem is particularly significant when the training task is to fine-tune a pre-trained model on some new downstream tasks, since the pre-train model contains more outliers than random initialization" (Section 3.2). A pre-trained model has already developed the feature-wise outlier structure during pre-training; fine-tuning must preserve these information-carrying outliers while adapting the model, making quantized fine-tuning especially sensitive to outlier handling.

Why prior PTQ methods do not solve this for FQT. Section 3.2 briefly reviews Outlier Suppression, SmoothQuant, and Outlier Channel Splitting. These methods were designed for post-training quantization (compressing an already-trained model for deployment, not training). At 4 bits, they fail for FQT because: (1) they are not designed to handle the interaction between quantization noise and gradient-based optimization over many steps; (2) they were developed for 6–8 bits where the quantization noise floor is lower; (3) methods like SmoothQuant that migrate difficulty from activations to weights can work at 8 bits but the weight matrices at 4 bits cannot absorb the transferred difficulty without their own accuracy degradation. Table 4 in Appendix C.3 confirms that SmoothQuant + LSQ achieves zero accuracy at 2–5 bits on CoLA — the weight quantization becomes the bottleneck.

---

Hadamard Quantization (HQ): The Forward Propagation Solution

Section 3.3 proposes the Hadamard Quantizer (HQ) to solve the outlier problem. The core idea is beautifully simple: don't quantize the activation in its original space — first apply a linear transformation that "spreads out" the outliers, quantize in the transformed space, then transform back.

The Hadamard Transform: What It Is and Why It Works

The Hadamard matrix is a recursively defined orthogonal matrix. For $k \geq 0$:

$H_0 = [1], \quad H_k = \frac{1}{\sqrt{2}} \begin{bmatrix} H_{k-1} & H_{k-1} \\ H_{k-1} & -H_{k-1} \end{bmatrix}$

where $H_k$ is a $2^k \times 2^k$ matrix.

The Hadamard matrix has three critical properties for this application: (1) it is orthogonal ($H_k H_k^\top = I$), so it preserves the Frobenius norm of any matrix it multiplies — no information is lost; (2) it is symmetric ($H_k = H_k^\top$), so forward and inverse transforms are identical; and (3) its entries are all $\pm 2^{-k/2}$, meaning multiplying by $H_k$ involves only additions and subtractions (no true multiplications, since the $\pm 1/\sqrt{2^k}$ scaling can be factored out).

The outlier-spreading mechanism. Consider a worst-case scenario: a single coordinate row vector $e_i^\top \in \mathbb{R}^{2^k}$ where only dimension $i$ has value 1 and all others are 0 — a pure outlier. When multiplied by $H_k$, the result is $e_i^\top H_k = 2^{-k/2} \mathbf{1}_{2^k}$, where $\mathbf{1}_{2^k}$ is a $2^k$-dimensional all-ones vector. The outlier has been completely amortized into an equal-magnitude vector across all dimensions — the ideal case for uniform quantization because every entry has identical magnitude.

In practice, activation outliers are not pure delta functions, but Figure 1(b) demonstrates the practical effect: after Hadamard transformation, the activation histogram no longer has the long tail — it's much more concentrated and quantization-friendly.

The block-diagonal structure. Applying a full $D \times D$ Hadamard matrix would require $D$ to be an exact power of 2, which is rarely the case for transformer hidden dimensions (768, 1024, 4096). The paper uses a block-diagonal Hadamard matrix:

$H = \text{BlockDiag}(H_k, \ldots, H_k) \in \mathbb{R}^{D \times D}$

where $D$ must be a multiple of $2^k$, and there are $D / 2^k$ blocks along the diagonal. This means the Hadamard transform operates independently on disjoint blocks of $2^k$ dimensions each. The block size $2^k$ controls the trade-off between outlier suppression power (larger blocks spread outliers more broadly) and computational cost (more operations per block). The paper proposes an adaptive algorithm to choose $k$ dynamically for each activation based on the outlier scale (Appendix A.5), with typical values around $k = 5$ ($2^5 = 32$ dimensions per block) when $C$ and $D$ range from 768 to 4096.

The HQ Quantization Procedure

Instead of quantizing $X$ and $W$ directly, HQ quantizes their Hadamard-transformed versions:

$X = (XH)H^\top \approx s_X \cdot \text{int}_{s_X}(XH) \cdot H^\top$ $W = (WH)H^\top \approx s_W \cdot \text{int}_{s_W}(WH) \cdot H^\top$

The key observation is that when these quantized approximations are multiplied to compute $Y = XW^\top$, the Hadamard transforms cancel:

$Y = XW^\top \approx s_X s_W \cdot \text{int}_{s_X}(XH) \cdot H^\top \cdot H \cdot \text{int}_{s_W}(H^\top W^\top)$ $= s_X s_W \cdot \text{int}_{s_X}(XH) \cdot \text{int}_{s_W}(H^\top W^\top)$

where the cancellation $H^\top H = I$ follows from orthogonality (and symmetry: $H^\top = H$ for Hadamard matrices). The result is an INT4 MM between the transformed-and-quantized activation and the transformed-and-quantized weight, with no Hadamard transform needed at inference/MM time — only the dequantization scalar multiplication $s_X s_W$.

What this computes operationally. The HQ matrix multiplication procedure (HQ-MM) has four steps:

1. Compute $XH$ and $H^\top W^\top$ in FP16 (the Hadamard transforms).
2. Quantize both transformed matrices to INT4 using LSQ with learned scales $s_X$ and $s_W$.
3. Multiply the two INT4 matrices using tensor-core INT4 GEMM.
4. Dequantize the INT32 accumulation result to FP16 by multiplying by $s_X s_W$.

Computational cost. Step 1 costs $\mathcal{O}(2^k N (D + C))$ FP16 multiply-accumulates (MACs) — the factor $2^k$ comes from the fact that each block of $2^k$ dimensions requires a $2^k \times 2^k$ Hadamard transform, and there are $D/2^k$ blocks, so the total cost per row of $X$ is proportional to $D \cdot 2^k$ (each block's transform is essentially a $2^k \times 2^k$ matrix multiply). Steps 2 and 4 cost $\mathcal{O}(N(D + C))$ FP16 MACs for the elementwise scale-and-round operations. Step 3 costs $\mathcal{O}(NDC)$ INT4 MACs.

The critical comparison is that an FP16 MM costs $\mathcal{O}(NDC)$ FP16 MACs. HQ-MM retains the INT4 MM at $\mathcal{O}(NDC)$ but replaces FP16 MACs with INT4 MACs, while adding $\mathcal{O}(2^k N (D + C))$ FP16 MACs for the Hadamard transforms. Since $2^k \ll D$ and $2^k \ll C$ (the paper notes typical $k = 5$, giving block size 32, while $D$ and $C$ are 768–4096), the Hadamard overhead is a small fraction of the total computation. The net result: "our HQ-MM is still much cheaper than an FP16 MM" (Section 3.3).

Choosing the block size adaptively. Appendix A.5 describes how $k$ is chosen per activation. For each candidate $k$, the quantization error is estimated as:

$\text{MSE}(\bar{X}_k, X) \times \text{MSE}(\bar{W}_k, W)$

where $\bar{X}_k = s_X \cdot \text{int}_{s_X}(XH_k) \cdot H_k^\top$ is the Hadamard-quantized-and-dequantized approximation of $X$, and similarly for $\bar{W}_k$. The $k$ that minimizes this product of MSEs is selected. For fine-tuning, $k$ is chosen once at the start and fixed. For pre-training, where distributions shift significantly, $k$ (and the LSQ scales) are re-initialized after the first 2 epochs.

Why this design over alternatives. The alternative approach — maintaining outlier entries in full precision — is the "outlier" ablation in Figure 3(a). While accurate, it is "hardware-unfriendly" because it requires sparse or mixed-precision operations that do not map well onto tensor cores. The Hadamard approach is elegant because it transforms the representation to make uniform quantization work, rather than building a more complex quantizer. This leverages existing, fast, uniform INT4 tensor core GEMM without modification.

A second alternative — more granular quantization like per-token or per-channel — is explored in Appendix C.3 (Table 4). Without HQ, even per-token + per-channel + SmoothQuant achieves zero accuracy at 2–5 bits on CoLA. With HQ, these more granular methods provide only marginal benefit over per-tensor LSQ (e.g., HQ alone: 54.6 at 4 bits; HQ + per-token + per-channel: 54.1 at 4 bits). The Hadamard transform is the necessary component; granularity alone is insufficient.

---

Backpropagation Setup and Gradient Computation

Section 4 opens by deriving the gradients required for backpropagation through the HQ forward pass. The linear operator defined in Eq. (3) has four inputs: activation $X$, weight $W$, and the two step sizes $s_X$, $s_W$. Given the output gradient $\nabla_Y \mathcal{L}$ (the gradient of the loss with respect to the MM output $Y$), the chain rule with the straight-through estimator ($\lfloor x \rceil' = 1$) yields:

$\nabla_W = s_X \left( \nabla_Y^\top \hat{X} \circ I_W \right) H^\top$ $\nabla_X = s_W \left( I_X \circ \nabla_Y \hat{W} \right) H^\top$

where $\hat{X} = \text{int}_{s_X}(XH)$ is the INT4-quantized transformed activation, $\hat{W} = \text{int}_{s_W}(WH)$ is the INT4-quantized transformed weight, $I_X = \mathbb{I}(-Q_N \leq X/s_X \leq Q_P)$ is a binary indicator matrix (1 for entries that are not clamped, 0 for clamped entries), and $I_W$ is defined analogously.

What these equations compute. $\nabla_W$ is the gradient of the loss with respect to the weight matrix — it's what gets subtracted from $W$ during the optimizer step (scaled by the learning rate). $\nabla_X$ is the gradient with respect to the input activation — it's propagated backward to earlier layers.

Three types of operations. The paper decomposes these gradient computations into three categories:

1. Element-wise multiplication of a 0/1 matrix ($I_X$ or $I_W$) with another matrix. This is cheap — it's just masking with negligible FLOPs.

2. Multiplication by the block-diagonal Hadamard matrix $s_W H^\top$ (or $s_X H^\top$). This is the same Hadamard transform used in the forward pass, costing $\mathcal{O}(2^k N (D + C))$ FP16 MACs. The paper classifies this as "low time complexity" relative to the main MM.

3. Multiplication of $\nabla_Y$ (FP16) with an INT4 matrix $\hat{X}$ or $\hat{W}$ — specifically, $\nabla_Y^\top \hat{X}$ for the weight gradient and $\nabla_Y \hat{W}$ for the activation gradient. These are the computationally expensive "type 3" MMs that the paper's LSS method accelerates by quantizing $\nabla_Y$ to INT4.

The paper notes that the step size gradients $\nabla_{s_W}$ and $\nabla_{s_X}$ (derived in Appendix A.3) reuse the "type 3" MMs already computed, so they add negligible extra cost.

---

Structural Sparsity of Gradients

Section 4.1 presents the key empirical observation that motivates LSS: activation gradients $\nabla_Y$ exhibit extreme structural sparsity along the token dimension. Figure 2(a) shows the histogram of per-row norms $\|(\nabla_Y)_{i,:}\|$ — a few rows have large norms, while most rows have norms near zero. Figure 2(b) reinforces this: the cumulative sum of the top $X$ largest row norms as a percentage of the total norm sum rises sharply — a small fraction of rows accounts for most of the total gradient magnitude.

Why this happens. The paper attributes this to "the heavy overparameterization of modern neural networks" (Section 4.1). During training, the network operates in the overparameterized regime where it fits most training examples well. Well-fitted examples produce activation gradients close to zero (the model has little to learn from them), while a few "hard" examples — unusual inputs, ambiguous cases, examples the model hasn't yet mastered — produce large gradients. For fine-tuning, the gradient is "always sparse during the whole training process" because the pre-trained model already fits most data; for pre-training, the sparsity "quickly emerges after only a few training epochs" as the model rapidly learns the easy examples.

Why this matters for quantization. If we quantize all rows of $\nabla_Y$ uniformly to 4 bits, two problems arise simultaneously: (1) the large-gradient rows, which carry the most important optimization signal, cannot be represented accurately in only 4 bits — they get truncated or coarsely approximated, potentially degrading convergence; (2) the small-gradient rows, which are essentially noise, consume the same amount of computation as the informative rows, wasting INT4 MACs that could be used to better represent the important gradients.

The paper's key insight: "Rather than computing these small gradients, it is better to save the computational resource for calculating the residuals of the larger gradients" (Section 4.2, slightly rephrased for clarity). The proposal is to drop some small-gradient rows entirely and use the freed computation to represent the retained large-gradient rows with higher precision.

---

Bit Splitting and Leverage Score Sampling (LSS): The Backward Propagation Solution

Section 4.2 develops the LSS method, which has two components: bit splitting (BS), which represents $\nabla_Y$ as higher and lower 4-bit components (an effective INT8 representation), and leverage score sampling (LSS), which probabilistically drops approximately half of these components to keep the total INT4 operation count unchanged.

Bit Splitting (BS): Creating an INT8 Representation from INT4 Operations

Bit splitting decomposes the FP16 gradient matrix into two INT4 matrices:

$\nabla_Y \approx s^\uparrow \nabla_Y^\uparrow + s^\downarrow \nabla_Y^\downarrow$

where $s^\uparrow, s^\downarrow \in \mathbb{R}$ are floating-point scalars, and $\nabla_Y^\uparrow, \nabla_Y^\downarrow$ are INT4 matrices representing the higher and lower 4 bits respectively.

The operational procedure. BS is implemented in two quantization steps:

1. Quantize $\nabla_Y$ to INT4 normally: $\nabla_Y \approx s^\uparrow \nabla_Y^\uparrow$, where $s^\uparrow$ is the LSQ scale and $\nabla_Y^\uparrow$ contains the quantized values in $\{-7, \ldots, 7\}$.

2. Quantize the residual (the error of the first quantization) to INT4: $\nabla_Y - s^\uparrow \nabla_Y^\uparrow \approx s^\downarrow \nabla_Y^\downarrow$, where $s^\downarrow$ is a second LSQ scale for the residual.

The result is an effective INT8 representation where the higher 4 bits capture the coarse magnitude and the lower 4 bits capture the fine detail. This is not a true INT8 format — it's two separate INT4 representations that, when combined with their respective scales, approximate the original FP16 value with 8 bits of precision. The advantage over simply quantizing to INT4 is substantially higher accuracy; the disadvantage is that it doubles the number of INT4 MMs required.

Formalizing the Weight Gradient with BS

For the weight gradient, the "type 3" MM is $\nabla_Y^\top \hat{X}$, where $\nabla_Y \in \mathbb{R}^{N \times C}$ and $\hat{X} \in \mathbb{Z}^{N \times D}$ (INT4). Applying BS:

$\nabla_Y^\top \hat{X} \approx (s^\uparrow \nabla_Y^{\uparrow \top} + s^\downarrow \nabla_Y^{\downarrow \top}) \hat{X} = \nabla_Y^{\updownarrow \top} X^{\updownarrow}$

where the paper introduces a stacked notation:

$\nabla_Y^{\updownarrow} = \begin{bmatrix} s^\uparrow \nabla_Y^\uparrow \\ s^\downarrow \nabla_Y^\downarrow \end{bmatrix}^\top \in \mathbb{R}^{2N \times C}, \quad \hat{X}^{\updownarrow} = \begin{bmatrix} \hat{X} \\ \hat{X} \end{bmatrix} \in \mathbb{Z}^{2N \times D}$

What this stacked representation means operationally: the original MM (one INT8-times-INT4 MM) is implemented as two INT4 MMs: $s^\uparrow \nabla_Y^{\uparrow \top} \hat{X}$ and $s^\downarrow \nabla_Y^{\downarrow \top} \hat{X}$. The results are summed after dequantization. This doubles the INT4 operation count from $NCD$ to $2NCD$ INT4 MACs compared to naive 4-bit quantization.

Leverage Score Sampling (LSS): Cutting the Cost Back Down

LSS reduces the cost of the BS formulation back to approximately $NCD$ INT4 MACs (the same as naive quantization) by probabilistically dropping approximately half of the $2N$ rows from the stacked matrices. The key idea is that many rows correspond to small gradients and can be discarded with minimal impact on the MM result.

The MM as a sum of rank-1 matrices. The product $\nabla_Y^{\updownarrow \top} X^{\updownarrow}$ can be decomposed as a sum of $2N$ rank-1 outer products:

$\nabla_Y^{\updownarrow \top} X^{\updownarrow} = \sum_{i=1}^{2N} (\nabla_Y^{\updownarrow \top})_{:,i} (X^{\updownarrow})_{i,:} = \sum_{i=1}^{2N} \nabla W_i$

where $\nabla W_i = (\nabla_Y^{\updownarrow \top})_{:,i} (X^{\updownarrow})_{i,:} \in \mathbb{R}^{C \times D}$ is the contribution of row $i$ to the total weight gradient. Each $\nabla W_i$ is a rank-1 matrix formed by the outer product of a column of $\nabla_Y^{\updownarrow \top}$ (a $C$-dimensional vector) and a row of $X^{\updownarrow}$ (a $D$-dimensional vector).

The importance sampling formulation. LSS assigns each component $i$ a probability $p_i \in [0, 1]$ satisfying $\sum_{i=1}^{2N} p_i = N$ (the total probability mass equals the number of rows in the original matrix). A random mask $m_i \sim \text{Bernoulli}(p_i)$ is drawn independently for each $i$, and a diagonal mask matrix $\tilde{M}$ is constructed:

$\tilde{M} = \text{diag}\left(\frac{m_1}{p_1}, \ldots, \frac{m_{2N}}{p_{2N}}\right)$

The MM is then approximated by masking the stacked representations:

$\nabla_Y^{\updownarrow \top} X^{\updownarrow} \approx \nabla_Y^{\updownarrow \top} \tilde{M} X^{\updownarrow} = \sum_{i=1}^{2N} \frac{m_i}{p_i} (\nabla_Y^{\updownarrow \top})_{:,i} (X^{\updownarrow})_{i,:}$

The factor $1/p_i$ compensates for the dropped components — when a component with $p_i = 0.5$ is included, its contribution is doubled to account for the fact that it's representing approximately two components.

Why this is unbiased. The expectation of the masked MM equals the original MM:

$\mathbb{E}\left[\nabla_Y^{\updownarrow \top} \tilde{M} X^{\updownarrow}\right] = \nabla_Y^{\updownarrow \top} \mathbb{E}[\tilde{M}] X^{\updownarrow} = \nabla_Y^{\updownarrow \top} X^{\updownarrow}$

because $\mathbb{E}[m_i/p_i] = p_i/p_i = 1$. The $\text{diag}$ notation means $\tilde{M}$ is diagonal with the specified entries, so $\mathbb{E}[\tilde{M}] = I$.

Why the cost is approximately $NCD$ INT4 MACs. In expectation, $\sum_i p_i = N$ and each $m_i$ is Bernoulli with mean $p_i$, so $\mathbb{E}[\sum_i m_i] = N$. This means approximately $N$ of the $2N$ rows are kept (on average), and each kept row contributes to an INT4 MM. The total INT4 operations are proportional to the number of kept rows times $C \times D = NCD$ — the same as naive INT4 quantization, but with the kept rows represented in INT8 precision via BS.

Minimizing the Variance: Deriving the Leverage Score

The quality of the LSS approximation depends on how the sampling probabilities $\{p_i\}$ are chosen. Proposition 4.1 provides the variance of the LSS estimator for the weight gradient:

$\text{Var}\left[\sum_{i=1}^{2N} \frac{m_i}{p_i} (\nabla_Y^{\updownarrow \top})_{:,i} (X^{\updownarrow})_{i,:}\right] = \sum_{i=1}^{2N} \frac{1 - p_i}{p_i} \|\nabla_{Y i,:}^{\updownarrow}\|^2 \|X_{i,:}^{\updownarrow}\|^2$

where $\text{Var}[X] := \mathbb{E}[\|X - \mathbb{E}[X]\|_F^2]$ is the expected squared Frobenius norm of the deviation from the mean.

Where this formula comes from. The proof (Appendix B.1) uses the fact that each term in the sum is independent (the $m_i$ are independent Bernoulli variables), so the variance of the sum is the sum of the variances. For a single term, $\text{Var}[(m_i/p_i) A_i] = \text{Var}[m_i/p_i] \cdot \|A_i\|_F^2$ since the matrix $A_i = (\nabla_Y^{\updownarrow \top})_{:,i} (X^{\updownarrow})_{i,:}$ is deterministic. The variance of the scaled Bernoulli is $(1-p_i)/p_i$, giving the result.

The leverage score. Define $c_i := \|\nabla_{Y i,:}^{\updownarrow}\| \cdot \|X_{i,:}^{\updownarrow}\|$. This is called the leverage score — a measure of how much row $i$ contributes to the total MM, combining the magnitude of the gradient row and the activation row.

Why proportional sampling minimizes variance. By the Cauchy-Schwarz inequality:

$\sum_{i=1}^{2N} \frac{c_i^2}{p_i} = \left(\sum_{i=1}^{2N} \frac{c_i^2}{p_i}\right) \left(\sum_{i=1}^{2N} p_i\right) \geq \left(\sum_{i=1}^{2N} c_i\right)^2$

where $\sum_{i=1}^{2N} p_i = N$ is the total probability budget. The inequality becomes equality when $p_i \propto c_i$ (i.e., $p_i = N \cdot c_i / \sum_j c_j$). This means the variance-minimizing strategy is to sample rows with probability proportional to their leverage score — rows that contribute more to the MM result are more likely to be kept, rows that contribute little are likely to be dropped.

Intuition: When the leverage scores $\{c_i\}$ are highly diverse — which Figure 2 shows they are, with a few large scores and many near-zero scores — LSS can approximate the MM accurately with far fewer rows because it concentrates the sampling budget on the rows that matter most.

The Practical LSS Procedure for Weight Gradients

The paper distills the analysis into a concrete procedure (LSS-MM for weight gradients):

1. Quantize $\nabla_Y$ with BS to obtain $\nabla_Y^\uparrow$ and $\nabla_Y^\downarrow$ in INT4.
2. Compute the leverage scores $c_i = \|\nabla_{Y i,:}^{\updownarrow}\| \cdot \|X_{i,:}^{\updownarrow}\|$ in FP16. The norms are computed from the quantized INT4 matrices to save computation (Appendix A.2).
3. Sample masks $\{m_i\}$ according to the probabilities derived from $\{c_i\}$ via the normalization procedure in Appendix A.2, which handles the case where the proportional $p_i$ would exceed 1 (impossible for a probability) by iteratively clamping and renormalizing.
4. Sample rows of $\nabla_Y$ and $\hat{X}$ based on the masks — rows with $m_i = 1$ are kept, rows with $m_i = 0$ are discarded.
5. Compute two INT4 MMs: $\nabla_Y^{\uparrow \top} \tilde{M}^\uparrow \hat{X}$ and $\nabla_Y^{\downarrow \top} \tilde{M}^\downarrow \hat{X}$, where $\tilde{M}^\uparrow$ and $\tilde{M}^\downarrow$ are the top-left and bottom-right $N \times N$ submatrices of $\tilde{M}$. Each MM operates on the sampled (kept) rows.
6. Dequantize the INT32 results and sum them to obtain the FP16 weight gradient.

The overhead (steps 1–4) costs $\mathcal{O}(NC + ND)$ in total: computing norms is linear in the matrix sizes, and sampling is linear in $N$. Step 5 costs approximately $2NCD$ INT4 MACs in expectation (as argued above). Step 6 is a constant-cost scalar multiply-and-add.

Activation Gradient via LSS

The activation gradient computation is structurally similar but with one important difference: the roles of the gradient and the INT4 matrix are swapped. The "type 3" MM is $\nabla_Y \hat{W}$, where $\nabla_Y \in \mathbb{R}^{N \times C}$ and $\hat{W} \in \mathbb{Z}^{C \times D}$ (INT4). Applying BS:

$\nabla_Y \hat{W} \approx (s^\uparrow \nabla_Y^\uparrow + s^\downarrow \nabla_Y^\downarrow) \hat{W} = s^\uparrow \nabla_Y^\uparrow \hat{W} + s^\downarrow \nabla_Y^\downarrow \hat{W}$

This can be written as a stacked MM by observing that $\nabla_Y^\uparrow \hat{W} + \nabla_Y^\downarrow \hat{W} = [I \; I] \begin{bmatrix} s^\uparrow \nabla_Y^\uparrow \\ s^\downarrow \nabla_Y^\downarrow \end{bmatrix} \hat{W}$, or more compactly:

$\nabla_Y \hat{W} \approx \left(\hat{I}^{\updownarrow} \nabla_Y^{\updownarrow}\right) \hat{W}$

where $\hat{I}^{\updownarrow} = [I \; I] \in \mathbb{R}^{N \times 2N}$ is a horizontal concatenation of two $N \times N$ identity matrices, and $\nabla_Y^{\updownarrow} = \begin{bmatrix} s^\uparrow \nabla_Y^\uparrow \\ s^\downarrow \nabla_Y^\downarrow \end{bmatrix} \in \mathbb{R}^{2N \times C}$.

What is different from the weight gradient case. For the weight gradient, we sampled rows of the stacked gradient and activation. For the activation gradient, we sample columns of the stacked identity matrix $\hat{I}^{\updownarrow}$. Since $\hat{I}^{\updownarrow}$ is just a concatenation of identity matrices, sampling its columns is equivalent to sampling rows of $\nabla_Y^{\updownarrow}$ — we are again deciding which rows of the gradient to process.

The decomposition into a sum of smaller MMs is:

$\hat{I}^{\updownarrow} \nabla_Y^{\updownarrow} = \sum_{i=1}^{2N} \hat{I}_{:,i}^{\updownarrow} \nabla_{Y i,:}^{\updownarrow} = \sum_{i=1}^{2N} \hat{\nabla Y}_i$

where $\hat{I}_{:,i}^{\updownarrow}$ is column $i$ of the stacked identity (either a unit vector in the first $N$ dimensions with $s^\uparrow$ weight, or in the second $N$ dimensions with $s^\downarrow$ weight), and $\nabla_{Y i,:}^{\updownarrow}$ is row $i$ of the stacked gradient.

The leverage score for activation gradients. Proposition B.2 (Appendix B.2) derives the variance:

$\text{Var}\left[\sum_{i=1}^{2N} \frac{m_i}{p_i} \hat{I}_{:,i}^{\updownarrow} \nabla_{Y i,:}^{\updownarrow}\right] = \sum_{i=1}^{2N} \frac{1 - p_i}{p_i} \|\nabla_{Y i,:}^{\updownarrow}\|^2$

Here the leverage score simplifies to $c_i := \|\nabla_{Y i,:}^{\updownarrow}\|$ — just the norm of the gradient row, since the "activation" component is from the identity matrix and has norm 1.

Why this is simpler. The activation leverage score only depends on the gradient magnitude, not on the weight matrix. This makes it cheaper to compute and means the same sampling mask can potentially be reused for both weight and activation gradient computations (the paper notes this optimization in Appendix A.6: "We omit the Quantization and Leverage Score Calculating process in LSSAct, and use the same value as LSSWeight to accelerate the training process").

Handling Invalid Probabilities

A technical detail: the proportional $p_i \propto c_i$ can produce probabilities larger than 1, which is invalid for a Bernoulli distribution. Appendix A.2 describes an iterative algorithm to handle this:

1. Start with $p_i^0 \propto c_i$, normalized to sum to $N$.
2. Clamp all $p_i$ to $[0, 1]$: $p_i^1 = \min(1, \max(0, p_i^0))$.
3. If any $p_i$ were clamped to 1, the total probability mass $\sum_i p_i^1 < N$. Scale up the remaining non-clamped probabilities to restore the total sum to $N$.
4. Repeat steps 2–3 until all $p_i \in [0, 1]$ (the process terminates in at most $\mathcal{O}(N)$ iterations because each iteration reduces the number of clamped values).

The paper also notes (Appendix A.6) that in practice, a simpler heuristic works equally well: simply keep rows whose leverage score (or gradient norm) is above zero, discarding rows with zero norm. Since many rows have exactly zero or near-zero gradients due to the structural sparsity, this achieves a similar effect without the iterative normalization.

---

Hardware Implementation Considerations

Appendix A.6 provides concrete details on the GPU implementation, which shapes the algorithm design in several ways:

INT4 data packing. GPUs do not have a native INT4 data type — the smallest addressable unit is INT8. To use INT4 tensor cores, data must be packed: two adjacent INT4 values are combined into one INT8 byte using (data[1] << 4) | (data[0] & 15). This packing happens on the host/GPU before the GEMM and is a non-trivial but necessary step for hardware compatibility.

Data layout constraints. Nvidia's cutlass INT4 tensor core GEMM requires the first input matrix to be RowMajor and the second to be ColumnMajor. PyTorch tensors are RowMajor by default, so the second matrix must be transposed and made contiguous (via .t().contiguous()), which is "very time-consuming and needs further optimization in the future" (Appendix A.6). The paper acknowledges this as an engineering limitation of the prototype.

K-dimension alignment. INT4 tensor core GEMM requires the inner dimension $K$ (the shared dimension between the two input matrices) to be a multiple of 32 for memory alignment. This is not a problem in forward propagation where dimensions are naturally multiples of model architecture sizes, but in backward propagation, the row sampling can produce matrices with non-aligned $K$. The solution is to zero-pad the sampled matrices so that $K$ becomes a multiple of 32.

Quantization overhead breakdown. Figure 6 (Appendix C.5) decomposes the time spent in HQ-MM and LSS-MM into sub-operations. In HQ-MM, the dominant cost is the INT4 GEMM itself, with the Hadamard transform and data packing contributing smaller fractions. In LSS (weight), the quantization and leverage score computation take measurable time; the authors suggest operator fusion could reduce this overhead.

The cold-start problem. Training from scratch (random initialization) presents an additional challenge: the distributions of weights and activations change rapidly in early iterations, making the LSQ scale parameters unstable. The paper's remedy (Appendix A.4): during the first few iterations, do not learn the step sizes. Instead, use a heuristic to set $s_X = 2 \cdot \text{mean}(X) / \sqrt{Q_p}$, where $Q_p = 7$ for 4-bit, and the mean is computed per tensor. This provides a reasonable initial scale that adapts to the changing distribution without the instability of joint optimization from scratch. For pre-training, this heuristic period lasts through the first 2 epochs, after which the Hadamard block size and LSQ scales are re-initialized.

Design Choice Summary

The paper's approach reflects several deliberate design decisions:

• Static over dynamic quantization for forward propagation: reduces per-iteration overhead at the cost of requiring learned scales. The scales are cheap to learn (scalar parameters, gradient computation reuses existing MMs) and converge along with the model.
• Transformation over granularity for outlier handling: the Hadamard transform addresses outliers by reshaping the distribution, rather than by using per-channel or per-token quantization that would complicate the GEMM. This preserves a simple, uniform INT4 tensor core operation.
• Importance sampling over uniform compression for gradients: rather than uniformly compressing all gradient rows to 4 bits, LSS uses importance sampling to allocate precision where it matters. This exploits the structural sparsity that overparameterization provides.
• Probabilistic dropping over deterministic thresholding for row selection: LSS uses Bernoulli sampling with inverse-probability weighting rather than hard thresholding. This preserves unbiasedness — the expected gradient equals the true gradient — which is important for stochastic optimization convergence.

4. Key Insights and Innovations

Innovation 1: A Structural-Correction Approach to Ultra-Low-Precision Training Rather Than a Numerical-Format Approach

The paper's most fundamental intellectual move is reframing the 4-bit training problem from a numerical format design problem to a structural distribution-correction problem. Prior work on low-precision training — from FP16 mixed precision (Micikevicius et al., 2018) through FP8 (Wang et al., 2018; Sun et al., 2019) to 4-bit FP4 (Sun et al., 2020) and logarithmic quantization (Chmiel et al., 2021) — approached the problem by asking: what is the optimal numerical representation for the values that tensors take? The answer was always a new format: wider dynamic range, non-uniform quantization levels, logarithmic spacing. Each format better captured the heavy-tailed distributions of activations and gradients, but each required custom hardware that did not exist.

This paper inverts the question. Instead of asking what format fits these tensors?, it asks can we transform the tensors so that a standard INT4 format works? The distinction is profound. A numerical format approach treats the statistical properties of tensors as fixed constraints that the hardware must accommodate. A structural-correction approach treats those statistical properties as artifacts of the tensor's organization that can be altered through principled linear transformations (Hadamard) or exploited through importance sampling (LSS).

This shift is not merely a different way of achieving the same goal — it changes what kinds of techniques are admissible. Under the format-design paradigm, the only degree of freedom is the mapping from bit patterns to real values. Under the structural-correction paradigm, one can use any invertible transformation of the representation space, any sparsity-exploiting sampling scheme, any technique from the RandNLA toolbox. The paper explicitly invokes this intellectual lineage in Section 1 by citing Drineas and Mahoney (2016) on randomized numerical linear algebra, signaling that this is not just engineering but an application of a mature mathematical framework to a new domain.

The practical consequence is that this single conceptual shift enables hardware compatibility without accuracy sacrifice. The Hadamard transform corrects the activation distribution so that uniform INT4 quantization — the format that GPUs actually implement — produces accurate results. The LSS procedure corrects the gradient computation by reallocating precision budget, again using only INT4 operations. The paper's Table 1 demonstrates that this structural approach recovers 5.5 points of GLUE accuracy over LSQ+LUQ (a format-based 4-bit method) on BERT-base, and an extraordinary ~29 points on BERT-large where the format-based method collapsed entirely. The figure is not just a metric improvement — it validates the entire reframing by showing that correcting for structure dominates trying to accommodate it.

This is a fundamental shift rather than an incremental refinement. It opens a new design space where training quantizers are paired with structure-exploiting transforms, and it suggests that other structural properties of neural network tensors (beyond feature-wise outliers and token-wise sparsity) might be similarly exploitable.

---

Innovation 2: Hadamard Transforms as Quantization-Friendly Representations — Not Compression, but Redistribution

The second innovation is the specific structural correction chosen for forward propagation: the Hadamard transform as a mechanism for outlier redistribution rather than outlier accommodation or removal. This is a genuinely non-obvious choice that distinguishes the paper from every prior approach to activation outliers in neural network quantization.

To appreciate why this is novel, consider how the field previously handled activation outliers in low-precision settings:

• Outlier Suppression (Wei et al., 2022) identified that LayerNorm amplifies outliers and proposed Gamma Migration and Token-Wise Clipping — methods that modify the network to prevent outliers from forming in the first place.
• SmoothQuant (Xiao et al., 2022) proposed per-channel scaling factors that migrate the quantization difficulty from activations to weights, effectively trading off activation precision for weight precision.
• Outlier Channel Splitting (Zhao et al., 2019) duplicated outlier-containing channels, splitting their magnitude across multiple channels to reduce per-channel range.
• The paper's own "outlier" ablation (Section 5.2, Figure 3a) simply kept the top 1% of activation entries in FP16 — selective high-precision preservation.

All these approaches treat outliers as a problem of representation range: the values are too large relative to typical entries, so either reduce their magnitude (Suppression, SmoothQuant, Splitting) or represent them in higher precision (the ablation). The Hadamard quantizer does neither. It does not change the network architecture, does not reduce outlier magnitudes, and does not allocate more bits to outliers. Instead, it applies a linear transformation that redistributes the outlier's energy across neighboring dimensions.

The paper's illustrative example in Section 3.3 — showing that a pure one-hot outlier vector $e_i^\top$ becomes a uniform all-ones vector $2^{-k/2} \mathbf{1}_{2^k}$ after Hadamard transformation — captures the conceptual essence. The outlier isn't suppressed; its total energy is preserved (the Frobenius norm is invariant under orthogonal transforms). But that energy is shared equally among $2^k$ entries, each of which is now at a magnitude comparable to typical activation values. The transformed tensor is quantization-friendly not because anything was removed, but because the information was reorganized.

This is a more subtle and powerful idea than it first appears. It works because the Hadamard transform's action on feature-wise outliers is not random — it's mathematically principled. Hadamard matrices are the discrete analog of Fourier transforms, converting "spiky" signals in the original basis into "flat" signals in the transformed basis. The feature-wise structure of outliers (concentrated in specific columns) is exactly the kind of spiky signal that Hadamard transforms are designed to handle.

The practical significance is validated by Figure 1(b): the post-Hadamard activation histogram is dramatically more concentrated, and the ablation in Figure 3(a) shows HQ recovering accuracy comparable to the hardware-unfriendly "keep-1%-in-FP16" method. But the conceptual significance is broader: it demonstrates that the right basis can convert a hard quantization problem into an easy one without losing information. This principle — transform, quantize, inverse-transform — is not limited to Hadamard matrices or activation outliers. Any structured distribution mismatch between tensor statistics and quantizer design might be addressable through an appropriate choice of basis.

This is an incremental refinement of quantization technique but a fundamentally new idea in the training quantization literature. The Hadamard transform has been used in signal processing and compressed sensing for decades, and its orthogonality properties are textbook material, but applying it as a quantization pre-processing step in neural network training — and discovering that the transforms cancel in the matrix multiplication, leaving no runtime overhead in the INT4 GEMM — is the intellectual contribution.

---

Innovation 3: Gradient Sparsity as a Budget-Reallocation Opportunity — Turning a Statistical Nuisance into a Computational Advantage

The third innovation is the paper's treatment of gradient structural sparsity not as a problem to be solved, but as an opportunity to reallocate representational precision. Prior work on low-precision training viewed gradient sparsity — the observation that some gradients are large while most are near zero — as an obstacle: a heavy-tailed distribution that makes uniform quantization inaccurate. The standard response was to design non-uniform quantizers (logarithmic, FP4) that could simultaneously capture the small and large values.

This paper takes the radically different view that the small gradients are not worth computing at all. Section 4.2 states the core insight directly: "many rows of the gradient are so small that they have little impact on the parameter gradient, yet they waste abundant computation." Rather than trying to represent small gradients more accurately (the format-design approach), the paper proposes to drop them entirely and use the saved computation to represent the large gradients with twice the precision — 8 bits via bit splitting rather than 4 bits.

This is a conceptual inversion. The field had been asking: how do we quantize a heavy-tailed distribution to 4 bits? The paper asks: given 4 bits of budget per element on average, but the ability to allocate bits non-uniformly across elements, how should we distribute precision to minimize the error in the downstream matrix multiplication? The answer — via Proposition 4.1 — is to sample elements with probability proportional to their leverage score, which combines gradient magnitude and activation magnitude into a single importance measure.

The novelty of this move is that it reframes gradient quantization as a variance-minimization problem in approximate matrix multiplication, connecting directly to the RandNLA literature. The leverage score is not a heuristic — it is the variance-optimal importance weight derived from first principles (the Cauchy-Schwarz proof in Proposition 4.1). This gives the method theoretical grounding that most quantization heuristics lack.

The practical consequence is demonstrated in Figure 3(b): on the challenging CoLA dataset, LSS at 4 bits achieves accuracy competitive with LUQ (a custom logarithmic format) while using only standard INT4 arithmetic. But the deeper significance is that overparameterization is not just a property of neural networks — it's a resource that can be exploited for efficient training. The structural sparsity that emerges from overparameterization (most examples fit well, a few are hard) is not an accident; it's a direct consequence of modern network design, and this paper shows how to convert that property into computational savings.

This innovation is fundamental in concept but incremental in its technical machinery (importance sampling for matrix multiplication is a standard RandNLA technique). The contribution is recognizing that training gradients have exactly the structure that makes leverage score sampling effective, and designing the bit-splitting mechanism to convert the sampling budget into higher effective precision.

---

Innovation 4: The Hardware-Compatibility Thesis as a First-Class Scientific Claim

The fourth innovation is less an algorithmic contribution and more a methodological stance that the paper defends empirically: the claim that hardware compatibility is not merely an engineering afterthought but a binding constraint that should shape algorithm design from the start. This matters because the 4-bit training literature had bifurcated into two disconnected tracks: (1) algorithmic papers that demonstrated training in simulated low-precision environments with custom formats, and (2) hardware papers that optimized implementations of higher-bitwidth training (FP16, INT8). The former could not realize speedups on existing hardware; the latter could not reach 4-bit precision.

This paper argues — through its design choices and empirical validation — that the separation is unnecessary. Every aspect of the proposed method is constrained by what INT4 tensor cores can execute: uniform quantization levels (not logarithmic), signed 4-bit integers in the range [-7, 7], matrix multiplication as the primitive operation, and specific data layout requirements (RowMajor/ColumnMajor, K-dimension multiples of 32). The Hadamard transform is chosen not only because it suppresses outliers, but because it preserves a uniform INT4 GEMM as the core operation. Bit splitting is chosen not only because it provides 8-bit effective precision, but because it decomposes into two INT4 MMs that tensor cores can execute natively.

The paper provides concrete evidence that this constraint was worth imposing: Figure 4 shows the prototype INT4 operators achieving 2.2× speedup over FP16 GEMM on an RTX 3090, and Figure 5 shows up to 35.1% end-to-end training speedup on BERT. This is not a simulated efficiency — it's a measurement on real hardware that anyone can purchase. The speedup numbers are not overwhelming (35% is meaningful but not transformative), but they validate the core thesis: hardware-compatible 4-bit training that maintains accuracy is achievable.

The broader impact of this stance is that it provides a template for future low-precision training research. Instead of designing algorithms and numerical formats in the abstract and hoping hardware eventually catches up, the paper models a design process where the hardware's capabilities and constraints are front-loaded into the algorithmic problem formulation. This is a methodological contribution rather than a technical one, but it is arguably as important: it changes what kind of research the field values and how it evaluates contributions.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a broad spectrum of tasks: natural language understanding (GLUE dev-set, Wang et al., 2018; SQuAD v1 and v2, Rajpurkar et al., 2016, 2018; Adversarial QA, Bartolo et al., 2020; SWAG, Zellers et al., 2018; CoNLL-2003, Sang and De Meulder, 2003), machine translation (WMT 14 En-De, Bojar et al., 2014), and image classification (CIFAR-10, CIFAR-100, Krizhevsky et al., 2009; ImageNet1k). For language tasks, the paper uses the standard splits from HuggingFace datasets; for machine translation, it uses the Fairseq recipe's split; for vision, it uses standard train/test splits with ViT checkpoints pre-trained on ImageNet21k (Dosovitskiy et al., 2020) and fine-tuned on the target datasets.

• Base model(s). The paper evaluates BERT-base-uncased and BERT-large-uncased (Kenton and Toutanova, 2019) for language understanding, Transformer-base (Vaswani et al., 2017) for machine translation, ViT-B/32, ViT-L/32, and ViT-L/16 (Dosovitskiy et al., 2020) for vision fine-tuning, and DeiT-Small (Touvron et al., 2021) for vision pre-training. The models span approximately 110M (BERT-base, ViT-B/32) to 340M (BERT-large) parameters, representing the scale range where 4-bit training could plausibly be deployed on consumer GPUs. The choice is deliberate: these models are "representative of the capabilities of many contemporary" transformer architectures (Section 5), and they are small enough that full-precision baselines can be reproduced multiple times for statistical reliability.

• Metrics. For GLUE, the paper reports the standard metric per task (Matthews correlation for CoLA, Pearson correlation for STS-B, accuracy for RTE/MRPC/SST-2/QNLI/QQP/MNLI) and the average across all tasks. For SQuAD and Adversarial QA, it reports F1 score. For SWAG and CoNLL-2003, it reports accuracy. For machine translation, it reports both the original BLEU score (Papineni et al., 2002) and the SacreBLEU score (Post, 2018), since different baselines use different metrics. For image classification, it reports top-1 accuracy. All metrics except machine translation are reported as the mean and standard deviation over 3 independent runs (subscripts in Table 1).

• Baselines. The paper compares against four baselines:

  • FP — full-precision training in FP16 mixed precision (Micikevicius et al., 2018), the upper bound on achievable accuracy.
  • INT8 — INT8 training following Banner et al. (2018), representing the previous generation of hardware-compatible fully quantized training.
  • LSQ+LUQ — a 4-bit baseline combining LSQ (Esser et al., 2019) for forward activation/weight quantization with logarithmic unbiased quantization (Chmiel et al., 2021) for backward gradient quantization. This is the most relevant existing 4-bit method and the primary point of comparison.
  • Ultra-low — Sun et al. (2020)'s FP4 training method. The paper only reports its performance on the machine translation task from the original paper since no public implementation is available. Note that this baseline uses FP4 gradients, making it hardware-incompatible.

• Generation budget / compute accounting. The paper does not use a "generation budget" concept since it accelerates training rather than inference. Instead, efficiency is measured in two ways: operator throughput (Figure 4, comparing FP16 GEMM vs. HQ and LSS MM throughput in TFLOPs on an RTX 3090) and end-to-end training throughput (Figure 5, comparing FP16 PyTorch AMP vs. INT4 training wall-clock time on 8× A100 GPUs). The operator throughput is measured at various matrix sizes to capture how quantization overhead varies with dimensions. The training throughput varies hidden size, intermediate size, and batch size to understand scaling behavior. For accuracy comparisons, all methods use the identical training recipe (optimizer, scheduler, hyperparameters) — the only variable is the numerical precision of the linear operators.

• Cross-validation / statistical protocol. For most tasks (GLUE, SQuAD, Adversarial QA, SWAG, CoNLL-2003, CIFAR), each configuration is run 3 times with different random seeds, and the standard deviation is reported as a subscript in Table 1. For large-scale tasks (machine translation, ImageNet1k fine-tuning of ViT-L models, DeiT pre-training), only single runs are reported due to computational cost. There is no cross-validation or held-out strategy selection reported — the method is applied uniformly with its default hyperparameters, not adapted per-task. This means the results reflect the method's out-of-the-box performance rather than its best-case after per-task tuning.

---

Main Quantitative Results

Language Model Fine-Tuning

The headline result for language understanding is in Table 1: on GLUE with BERT-base, HQ+LSS achieves 80.81 average score compared to 82.67 for FP (a 1.86-point degradation) and 75.29 for LSQ+LUQ (a 5.52-point improvement over the existing 4-bit baseline). On BERT-large, the gap is more dramatic: HQ+LSS achieves 82.25 vs. 84.57 for FP (2.32-point degradation) and 55.93 for LSQ+LUQ (a 26.32-point improvement — LSQ+LUQ essentially collapses on the larger model).

The full GLUE breakdown (Appendix C.2, Table 2) reveals where the accuracy recovery comes from. On BERT-base, the tasks where LSQ+LUQ degrades most severely are precisely those where HQ+LSS provides the largest gains:

• CoLA: LSQ+LUQ drops to 18.76 (from 56.89 FP), while HQ+LSS recovers to 52.46 — a 33.7-point improvement. CoLA is a linguistic acceptability task requiring fine grammatical distinctions, making it sensitive to information loss.
• RTE: LSQ+LUQ achieves 56.80, HQ+LSS recovers to 62.45 (vs. 64.80 FP). RTE is a textual entailment task requiring precise logical reasoning.
• STS-B: LSQ+LUQ drops to 84.31, HQ+LSS recovers to 87.77 (vs. 88.14 FP). This is a semantic similarity task where subtle activation differences matter.
• MNLI: LSQ+LUQ achieves 81.79, HQ+LSS recovers to 83.59 (vs. 84.52 FP). MNLI is the largest GLUE task and tests broad natural language inference.

On easier tasks like SST-2 (sentiment analysis) and QQP (paraphrase detection), LSQ+LUQ already performs reasonably well (90.37 and 89.26 respectively), and HQ+LSS provides smaller gains (to 92.49 and 89.80). This pattern — large gains on difficult tasks requiring precise representations, smaller gains on easier tasks — is consistent with the outlier hypothesis: difficult tasks store more information in activation outliers, making outlier-tolerant quantization more beneficial.

On BERT-large, the pattern is even starker. LSQ+LUQ achieves 0.00 on CoLA (complete failure), 33.42 on MNLI (vs. 86.71 FP), and 63.18 on QNLI (vs. 92.26 FP). HQ+LSS recovers to 53.46 (CoLA), 85.86 (MNLI), and 91.53 (QNLI). The paper hypothesizes that "for those challenging tasks, there is more information stored in the outliers, which results in a larger gap between our method and LSQ+LUQ" (Appendix C.2). The BERT-large model has more capacity to develop outlier-heavy representations, making uniform 4-bit quantization more damaging.

For SQuAD v1 and v2 (Table 1), HQ+LSS achieves 87.60 and 74.63 F1 respectively, compared to 88.32 and 76.04 for FP and 85.75 and 71.02 for LSQ+LUQ. The 1.85- and 3.61-point improvements over LSQ+LUQ are substantial but leave a 0.72- and 1.41-point gap to FP.

On Adversarial QA — a deliberately challenging reading comprehension dataset designed to be hard for models — HQ+LSS achieves 38.70 F1 vs. 40.99 FP and 31.85 LSQ+LUQ, a 6.85-point improvement. On SWAG (commonsense reasoning), HQ+LSS achieves 77.49 accuracy vs. 79.84 FP and 70.79 LSQ+LUQ, a 6.70-point improvement. On CoNLL-2003 (named entity recognition), HQ+LSS achieves 91.90 vs. 93.38 FP and 87.63 LSQ+LUQ, a 4.27-point improvement.

Across all 9 language tasks evaluated (GLUE average × 2 model sizes + SQuAD × 2 + Adversarial QA + SWAG + CoNLL-2003), HQ+LSS outperforms LSQ+LUQ on every single one. The improvement is larger on harder tasks and larger models, consistent with the outlier hypothesis.

Machine Translation (Pre-Training)

Table 1 reports the WMT 14 En-De results. The paper notes an important metric discrepancy: the baseline Ultra-low (Sun et al., 2020) reports the original BLEU score of 27.5 for FP and 25.4 for their method (2.1-point degradation), while LSQ+LUQ reports 27.17 BLEU. The Fairseq recipe used in this paper's reproduction reports the more conservative SacreBLEU score of 26.5 for FP.

HQ+LSS achieves 25.57 SacreBLEU, representing approximately 1.0-point degradation from FP (26.5) under this metric. The paper states this is "smaller than 2.1 of Ultra-low and higher than 0.3 reported in the LUQ paper" (Section 5.1). However, these comparisons are not on the same metric — the 0.3 figure from the LUQ paper likely refers to the original BLEU score, making direct comparison impossible from the reported numbers. The paper's wording "HQ+LSS still performs comparably with existing methods for this pretraining task, and it supports contemporary hardware" is appropriately cautious.

The machine translation task is notable because it is the only pre-training-from-scratch task evaluated (as opposed to fine-tuning a pre-trained checkpoint). The cold-start heuristic (Appendix A.4) is used here, where the Hadamard block size and LSQ scales are re-initialized after epoch 2. The fact that the method converges to reasonable accuracy on a task where distributions shift dramatically during early training validates the cold-start procedure.

Image Classification

Table 1 reports vision results across three datasets and multiple model scales.

CIFAR-10: HQ+LSS achieves 98.36 for ViT-B/32 (vs. 98.77 FP, 97.76 LSQ+LUQ) and 98.47 for ViT-L/32 (vs. 98.98 FP, 98.38 LSQ+LUQ). The degradation from FP is 0.41 and 0.51 points respectively, compared to 1.01 and 0.60 for LSQ+LUQ. CIFAR-10 is a relatively easy dataset (10 classes, high accuracy), so the absolute room for improvement over LSQ+LUQ is limited, but HQ+LSS still provides a measurable benefit.

CIFAR-100: HQ+LSS achieves 89.78 for ViT-B/32 (vs. 91.94 FP, 88.63 LSQ+LUQ) and 91.13 for ViT-L/32 (vs. 93.07 FP, 90.97 LSQ+LUQ). The paper notes that "INT8 already has ~1% accuracy degradation, which shows its difficulty" — even 8-bit training loses 0.95 points on CIFAR-100 with ViT-B/32. HQ+LSS degrades 2.16 points from FP compared to 3.31 for LSQ+LUQ, recovering 1.15 points.

ImageNet1k: Across three model scales:

• ViT-B/32: HQ+LSS 79.18 vs. 81.88 FP and 77.25 LSQ+LUQ (1.93-point improvement)
• ViT-L/32: HQ+LSS 80.06 vs. 81.62 FP and 77.41 LSQ+LUQ (2.65-point improvement)
• ViT-L/16: HQ+LSS 82.61 vs. 84.55 FP and 82.40 LSQ+LUQ (0.21-point improvement)

The trend is nuanced: for ViT-B/32 and ViT-L/32, HQ+LSS provides substantial gains over LSQ+LUQ (1.93 and 2.65 points). For ViT-L/16, the largest vision model tested, LSQ+LUQ already performs reasonably well (82.40 vs. 84.55 FP), and HQ+LSS adds only 0.21 points. This could indicate that LSQ+LUQ's logarithmic gradient quantization becomes relatively more effective at larger model scales (where gradients may be better-behaved), or it could reflect that ViT-L/16's higher resolution patch size (16×16 vs. 32×32) changes the activation statistics in ways that affect both methods differently.

DeiT-Small pre-training on ImageNet1k: HQ+LSS achieves 69.18 vs. 73.10 FP and 69.96 LSQ+LUQ. This is the only result where HQ+LSS underperforms LSQ+LUQ (by 0.78 points). The paper does not comment on this reversal, but it occurs in the only vision pre-training setting (as opposed to fine-tuning), suggesting that the cold-start heuristic or the Hadamard block size adaptation may be less effective for vision pre-training than for language pre-training (where the WMT result showed competitive performance).

Computational Efficiency (Operator and End-to-End Speedup)

Operator-level speedup (Figure 4). On an Nvidia RTX 3090 (142 FP16 TFLOPs peak, 568 INT4 TFLOPs peak), the paper compares FP16 GEMM throughput against HQ (forward), LSSWeight (weight gradient), LSSAct (activation gradient), and their average (INT4) across multiple matrix sizes. Key findings:

• At the largest matrix sizes tested, INT4 operators reach up to 2.2× the throughput of FP16 GEMM.
• At smaller matrix sizes, the speedup is reduced because the Hadamard transform, quantization, and data packing overhead constitute a larger fraction of total time.
• The individual operators show different scaling: HQ-MM achieves the highest speedup at large sizes, while LSS operators have additional overhead from leverage score computation and sampling that reduces their peak throughput.

End-to-end training speedup (Figure 5, Table 3 in Appendix C.4). On 8× A100 GPUs training BERT and GPT-style models:

• For BERT-Large (encoder-only), speedup ranges from -25.5% (slowdown at small scale: hidden=2560, intermediate=10240, batch=2048) to 35.1% (at hidden=8960, intermediate=35840, batch=480). The speedup increases with model width because the INT4 GEMM dominates more of the total computation.
• For GPT2-base (decoder-only), speedup ranges from -27.7% (slowdown at hidden=2560) to 26.5% (at hidden=9600). The pattern is similar: larger models benefit more.
• The slowdown at small scales occurs because the quantization overhead (Hadamard transforms, data packing, leverage score computation, sampling) dominates when the matrix multiplications are small. The paper explicitly acknowledges this: "our implementation is not fully optimized" (Section 5.3) and notes that operator fusion and other optimizations could reduce this overhead.

Time proportion analysis (Figure 6, Appendix C.5). Breaking down where time is spent in each operator:

• In HQ-MM: the INT4 GEMM consumes the majority of time, followed by the Hadamard transform. Data packing and dequantization are minor.
• In LSSWeight: quantization (bit splitting) and leverage score computation take non-trivial fractions. The INT4 GEMM still dominates but less so than in HQ.
• In LSSAct: the quantization and leverage score are reused from LSSWeight, so the overhead is reduced.

Cross-GPU performance (Figures 7 and 8, Appendix C.5). The paper also reports operator throughput on T4 and A100 GPUs, showing that the speedup is consistent across GPU generations with different CUDA capabilities (sm_75 for T4, sm_80 for A100, sm_86 for RTX 3090).

---

Ablation Studies and Robustness Checks

Forward propagation methods on CoLA (Figure 3a): This ablation fixes backpropagation in FP16 and varies only the forward quantization method.

• LSQ (uniform 4-bit quantization): achieves very low accuracy (~20% Matthews correlation at 4 bits), confirming that naïve forward quantization fails.
• Outlier (keep top 1% activation entries in FP16): achieves accuracy close to full precision (~55%). This validates the paper's central claim that outliers are the primary cause of forward quantization error — if outliers are preserved, the rest can be quantized aggressively. However, this method is "hardware-unfriendly" because it requires sparse or mixed-precision operations.
• HQ (Hadamard quantization): achieves accuracy comparable to the outlier method (~53–54%). This validates that the Hadamard transform effectively addresses the outlier problem without requiring custom hardware support.
• The fact that HQ matches the outlier preservation method is a strong result — it means the Hadamard transform captures essentially all the information that the outliers carry, just redistributed rather than preserved in place.

Backward propagation methods on CoLA (Figure 3b): This ablation fixes forward propagation in FP16 and varies only the gradient quantization method.

• Minimax (uniform 4-bit quantization with min/max range): accuracy degrades severely at 4 bits and below.
• LUQ (logarithmic unbiased quantization, Chmiel et al., 2021): performs well across bit widths — the paper's primary competitor for gradient quantization.
• LSS (leverage score sampling): at bit widths above 2, achieves accuracy "comparable and even slightly higher than LUQ" (Section 5.2).
• This is significant because LSS uses only standard INT4 arithmetic, while LUQ requires custom logarithmic formats. The comparable accuracy validates the importance-sampling approach as a hardware-compatible alternative to non-uniform quantization.

INT4 precision choice (Figure 3b): The figure sweeps bit widths from 1 to 8. Both LUQ and LSS show monotonic improvement with bit width, with LSS closely tracking LUQ. At 1–2 bits, both fail; at 3 bits, both provide partial accuracy; at 4 bits and above, both approach full-precision performance. This confirms 4 bits as a reasonable operating point — the minimum width where gradient quantization is feasible with these methods.

Quantization granularity methods on CoLA (Table 4, Appendix C.3): This extensive ablation tests whether more granular quantization (per-token, per-channel) or smoothing techniques (SmoothQuant) can substitute for or complement HQ.

• Per-tensor LSQ: 0 accuracy at 2 bits, 9.16 at 3 bits, 24.2 at 4 bits, 51.4 at 8 bits. The jump from useless (3 bits) to partially working (4 bits) to nearly full-precision (8 bits) shows the sensitivity to bit width.
• Per-token quantization: 0 at 2–3 bits, 15.3 at 4 bits — better than per-tensor at 3 bits but still far from usable.
• Per-token + per-channel + SmoothQuant: 0 at 2–5 bits. This is the key negative result: SmoothQuant, which works well at 8 bits, completely fails at 4 bits. The paper's explanation: "the motivation of LSQ is to learn a trade-off between outliers and inliers, while smoothquant aims to sacrifice the precision of inliers in order to exactly maintain the information of outliers. When the bitwidth is high, this is not a problem... But when the bitwidth is low, such sacrifice will cause severe problems."
• HQ alone: 0 at 2 bits, 45.2 at 3 bits, 54.6 at 4 bits, 58.4 at 8 bits. Note that HQ at 3 bits (45.2) already outperforms LSQ at 8 bits (51.4) — this is a striking demonstration of the Hadamard transform's effectiveness.
• HQ + per-token + per-channel: 48.4 at 3 bits, 54.1 at 4 bits. The marginal benefit over HQ alone (45.2 → 48.4 at 3 bits, 54.6 → 54.1 at 4 bits) is small or negative, confirming that granularity does not substitute for the Hadamard transform.
• HQ + per-token + per-channel + SmoothQuant: 0 at 2–3 bits. SmoothQuant actually harms HQ at low bit widths — the weight quantization cannot absorb the difficulty that SmoothQuant transfers from activations.

This ablation conclusively demonstrates that the Hadamard transform is the necessary and sufficient mechanism for forward quantization below 8 bits. More granular quantizers or smoothing techniques cannot replace it, and they provide at best marginal complementary benefits.

Adaptive Hadamard block size (cold-start): Appendix A.4 and A.5 describe the cold-start heuristic (not learning step sizes for the first few iterations, using mean-based initialization) and the adaptive block size selection (minimizing quantization MSE). The paper does not provide an ablation comparing adaptive vs. fixed block size or warm-start vs. no warm-start. The effectiveness of these design choices must be inferred from the fact that the overall method works on both fine-tuning (where distributions are stable) and pre-training (where the cold-start and re-initialization are employed after epoch 2).

LSS sampling implementation shortcut (Appendix A.6): The paper notes that the theoretically optimal iterative sampling procedure (Appendix A.2) is replaced in practice by a simpler heuristic: "simply selecting elements whose leverage score is bigger than 0 can also work well, even better than our proposed algorithm in some cases." No ablation is provided comparing the optimal and heuristic sampling strategies, but the paper reports this as a practical implementation detail.

---

Critical Assessment

Claim 1: HQ+LSS achieves competitive accuracy with full-precision training across diverse tasks.

The evidence in Table 1 provides strong support for this claim on fine-tuning tasks with moderate-sized models (BERT-base, BERT-large, ViT-B/32, ViT-L/32). Across 9 language tasks and multiple vision configurations, HQ+LSS closes most of the accuracy gap between LSQ+LUQ and FP, reducing the degradation to typically 1–2 points on GLUE average and 1–3 points on vision tasks. This is a genuine achievement — prior 4-bit methods lost 5–29 points on these benchmarks.

However, the evidence is weaker in specific settings:

• BERT-large fine-tuning: While HQ+LSS dramatically outperforms LSQ+LUQ (82.25 vs. 55.93 on GLUE average), the 2.32-point gap to FP (84.57) is larger than for BERT-base (1.86-point gap). The claim of "competitive" accuracy is true — the model trains and achieves reasonable performance — but the gap is measurable and statistically significant given the reported standard deviations.

• Pre-training from scratch: Only two pre-training results are reported: WMT machine translation (Transformer-base) and DeiT-Small on ImageNet1k. On WMT, HQ+LSS achieves ~1.0 SacreBLEU degradation, which the paper correctly characterizes as "comparable" to existing methods. On DeiT-Small, HQ+LSS actually underperforms LSQ+LUQ by 0.78 points (69.18 vs. 69.96). This is the only negative result in the main experiments, and the paper does not discuss it — a notable omission. The pre-training evidence is thin: two tasks, small models (Transformer-base, DeiT-Small), and only one run each (no standard deviations).

• Larger-scale training: The paper does not evaluate on models larger than BERT-large (340M parameters) or ViT-L/16. Section 6 (Limitations) acknowledges that "the proposed method cannot yet work well for those extremely large models such as OPT-175B. To the best of our knowledge, even INT8 training is still an open problem for these large models." This is an honest and important boundary condition. The accuracy claims should be understood as applying to the scale range tested (~100M–350M parameters), with no evidence for or against scaling to billions of parameters.

• Standard deviation and statistical reliability: For most tasks, the paper reports mean and standard deviation over 3 runs. However, for the most computationally expensive configurations (machine translation, ViT-L on ImageNet1k, DeiT pre-training), only single runs are reported. The standard deviations on GLUE tasks range from 0.05 (SST-2) to 1.46 (CoLA) for HQ+LSS on BERT-base, meaning the 1.86-point gap to FP on GLUE average may be within 1–2 standard deviations for some individual tasks. The paper could strengthen its case by reporting confidence intervals or by running the larger experiments multiple times.

Claim 2: The Hadamard quantizer solves the activation outlier problem for 4-bit forward propagation.

The evidence in Figure 3(a) and Table 4 strongly supports this claim for the specific case of BERT fine-tuning on CoLA. HQ matches the accuracy of keeping the top 1% of activations in FP16 — the "outlier" ablation that represents an upper bound on outlier-handling effectiveness. Table 4 further demonstrates that HQ alone (without granular quantizers or smoothing) achieves 54.6 Matthews correlation at 4 bits, while the best non-HQ method (per-token + LSQ) achieves only 27.8.

However, several caveats:

• The evidence is from a single task (CoLA) and a single model family (BERT). Figure 3(a) only shows CoLA results, not a sweep across GLUE tasks or other architectures. The improvement on other tasks is shown only through the full HQ+LSS system (Table 1), not through a forward-only ablation. It is possible that the outlier problem is less severe on some tasks and that HQ provides diminishing benefits. The CoLA task was chosen precisely because it is challenging and outlier-sensitive — this makes it a good stress test but potentially overstates the average benefit.

• The "outlier" upper bound is hardware-unfriendly but not uniquely defined. The 1% threshold is arbitrary — would 0.5% be sufficient? 2%? The paper does not ablate this threshold. The fact that HQ matches the 1%-preservation method is convincing, but the absolute upper bound on outlier-handling methods is unknown.

• No ablation comparing HQ to other outlier-handling transforms. The Hadamard matrix is one orthogonal transform among many (Fourier, DCT, random orthogonal matrices). Is there something special about Hadamard, or would any orthogonal matrix work? The paper argues theoretically that Hadamard is particularly effective for extreme outliers (the one-hot to all-ones example), but does not empirically test alternatives. An ablation with a random orthogonal matrix would strengthen the claim that Hadamard is specifically well-suited.

• The block size $k$ is chosen adaptively, but no sensitivity analysis is provided. How much does accuracy vary with block size? Is the MSE-based selection criterion (Appendix A.5) close to optimal? For fine-tuning, $k$ is fixed after initial selection — does re-optimizing $k$ periodically improve results?

Claim 3: LSS with bit splitting achieves gradient quantization accuracy competitive with custom numerical formats (LUQ) using only INT4 arithmetic.

Figure 3(b) supports this claim directly for CoLA — LSS matches or slightly exceeds LUQ at bit widths above 2. This is a significant result because it shows that importance sampling can substitute for non-uniform quantization.

However:

• The comparison is only shown for backward propagation in isolation. Figure 3(b) fixes forward propagation to FP16. The interaction between forward quantization error and backward quantization error is not explored — it's possible that when combined, the errors interact non-linearly. The full HQ+LSS results in Table 1 show competitive accuracy, but the marginal contribution of LSS over a simpler gradient quantizer in the combined setting is not isolated.

• No ablation comparing LSS to a simple best-of-N or top-K deterministic selection. The paper's sampling approach is unbiased due to inverse-probability weighting, but is unbiasedness necessary for training? A deterministic threshold (keep the top-K gradient rows by norm, discard the rest, no reweighting) would be biased but potentially simpler and faster. No such ablation is provided.

• The leverage score computation introduces overhead that is not fully quantified in isolation. Figure 6 shows time breakdown at one matrix size, but the overhead as a function of matrix dimensions is not comprehensively characterized. At what model scales does the sampling overhead outweigh the INT4 GEMM savings? The end-to-end slowdown at small scales (Figure 5, Table 3) suggests the overhead is substantial for small models — it's unclear whether this is primarily from LSS or from HQ.

Claim 4: The method achieves real hardware speedup on contemporary GPUs.

This claim is supported by Figures 4, 5, 7, and 8, but with important qualifications:

• The 2.2× operator speedup is an upper bound, achieved at the largest matrix sizes. At smaller, more common sizes, the speedup is lower. The end-to-end speedup of 35.1% reflects this — it's the speedup averaged over all linear operations in the training loop across their actual sizes, not the peak kernel speedup.

• The prototype implementation is explicitly not optimized. Appendix A.6 lists several sources of overhead: explicit transposes for ColumnMajor layout ("very time-consuming"), no operator fusion, K-dimension zero-padding for sampled matrices. An optimized implementation would likely achieve higher speedups. However, this also means the current speedup numbers are not necessarily representative of what a production implementation would achieve. They are a lower bound in one sense (optimizations would improve speedup) but potentially an upper bound in another sense (the paper chooses matrix sizes where the overhead fraction is small).

• The end-to-end speedup is measured on fine-tuning, not pre-training. Table 3 shows speedup for fine-tuning BERT-Large and GPT2-base on the SuperGLUE RTE dataset. Pre-training involves different computation patterns (larger batches, different sequence lengths, different ratios of forward to backward computation) and may show different speedup characteristics. The paper does not report end-to-end training time for the machine translation or DeiT pre-training tasks.

• Memory savings are not evaluated. INT4 training should reduce memory usage by approximately 4× for the quantized tensors (activations, weights, gradients), but the paper does not measure peak memory usage during training. This is a significant omission — for large-model training, memory constraints often matter more than FLOPs. If INT4 training reduces memory enough to use larger batch sizes or avoid gradient checkpointing, the effective speedup could be much larger than 35%.

What experiments would strengthen the paper:

• Model scale sweep on a single task. Evaluate HQ+LSS on the same architecture (e.g., BERT) at scales from 100M to 1B+ parameters, measuring both accuracy degradation and speedup vs. FP. This would address the critical question: does accuracy degradation increase, decrease, or stay constant with model scale?
• Ablation of Hadamard transform alternatives. Compare HQ to a random orthogonal matrix, a DCT, and a learned transformation on CoLA. This would clarify whether the Hadamard choice is critical.
• Deterministic top-K gradient sampling vs. stochastic LSS. This would test whether unbiasedness matters for training convergence.
• Memory footprint measurements. Report peak GPU memory for FP16 vs. INT4 training at various batch sizes.
• More pre-training results. Pre-training is where training acceleration matters most (fine-tuning is relatively cheap). The single WMT and DeiT-Small results are insufficient to claim general pre-training effectiveness.
• Standard deviations for large-scale experiments. The lack of error bars on the machine translation and ImageNet1k ViT-L results makes it impossible to assess whether the observed gaps are statistically significant.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Likely Dominates the Inference Budget

The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so requires generating and scoring 2048 samples per prompt from the base model, then binning prompts into difficulty quintiles based on the pass@1 rate or PRM average score (Section 3.2). This cost is enormous — 2048 samples is 4–8× larger than the largest test-time compute budgets studied (256–512 generations), meaning the difficulty estimation itself consumes more computation than the actual problem-solving strategies being optimized.

The paper acknowledges this explicitly in Section 3.2: "our experiments do not account for this cost largely for simplicity." This is not a minor bookkeeping issue — it means the headline 4× efficiency gains over best-of-N are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment where difficulty must be estimated for each incoming prompt, the total cost would be difficulty estimation + strategy execution, and the former could easily dominate the latter, potentially eliminating or reversing the reported efficiency advantage.

The paper attempts to address this by moving from oracle difficulty (requiring ground-truth correctness checks) to predicted difficulty (using only the PRM's final-answer score distribution). While the predicted difficulty bins largely track the oracle bins (the curves overlap in Figures 4 and 8), this does not reduce the sample cost — it only removes the need for ground-truth labels. The paper explicitly frames this as an exploration-exploitation tradeoff (Section 3.2) and flags it as "a key avenue for future work," suggesting that pretraining or fine-tuning models to directly predict difficulty from question text could eliminate the estimation cost. However, no such model is developed or evaluated, so the current results represent an idealized upper bound on what is achievable without the estimation overhead.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

Across all methods — PRM search, iterative revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5, the bottom quintile of the base model's pass@1 rate) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% accuracy, and at high inference-to-pretraining ratios (R ≫ 1), PRM search shows a −52.9% relative disadvantage compared to simply training a ~14× larger model (Figure 1, bottom-right bar chart).

This reveals a fundamental boundary condition: test-time compute can amplify and refine existing capability but cannot create capability from nothing. If the base model's pass@1 rate is near zero on a problem class — meaning the model essentially never produces a correct solution in its initial samples — then no amount of search, revision, or adaptive allocation can find a correct answer, because there is no correct answer in the proposal distribution to discover. The FLOPs-matched analysis in Section 7 quantifies this boundary: for hard problems (bins 4–5), pretraining compute is almost always more effective than test-time compute, and the advantage of test-time compute that exists on easy-to-medium problems disappears or reverses.

The paper is transparent about this limitation, including it in the Section 7 takeaway box: "test-time and pretraining compute are not 1-to-1 exchangeable; test-time compute is powerful when problems are within the base model's reach but cannot compensate for fundamental capability gaps that larger pretraining would address." However, this transparency does not mitigate the practical consequence: the method offers no path forward for genuinely novel, out-of-distribution, or fundamentally challenging reasoning that exceeds the base model's training distribution. For such problems, pretraining scale remains the only viable path, and the compute-optimal framework provides no guidance on how to bridge that gap.

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate with No Principled Solution

Section 6.1 reports a significant practical problem with the revision model: approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This occurs because the model was trained exclusively on sequences where all in-context answers are incorrect followed by a correct target — during training, it never sees examples of what to do when the current answer is already correct. At test time, when the chain happens to produce a correct answer, the model has no learned behavior for recognizing this and simply continuing to produce revisions, which can corrupt the correct answer.

The paper mitigates this with majority voting or verifier-based selection across the entire chain of revisions — picking the best answer from any point in the chain rather than always taking the final revision. This is an effective patch (Figure 6 shows that sequential with best-of-N weighted selection outperforms parallel, despite the reversion problem), but it is an imperfect one. The selection mechanism can fail — if the verifier or majority vote incorrectly prefers a wrong answer over a correct one, the reversion problem still degrades accuracy. More importantly, this patch does not address the root cause: the model has not learned when to stop revising.

The ReST^EM experiment in Appendix K (Figure 16) further highlights the fragility of revision training. When the authors attempted to optimize the revision model using RL-style training with on-policy data collection, performance degraded substantially with sequential revisions — at 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection exacerbates spurious correlations in revision data." This negative result demonstrates that the positive revision results depend on specific training choices (offline data construction, edit-distance-based incorrect-correct pairing) and are not robust to straightforward attempts at improvement. The paper does not explore more principled solutions, such as training the model to output a "stop" token when no revision is needed, or incorporating correct-answer examples in the training data to teach the model to recognize and preserve correct outputs.

The Method Is Evaluated on a Single Benchmark and a Single Model Family — Generalizability Is Unverified

All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The MATH benchmark consists exclusively of competition-level math problems requiring multi-step symbolic reasoning with clean, verifiable answers. This is a specific and narrow domain. The paper does not evaluate on code generation (HumanEval, MBPP), logical reasoning (ARC, FOLIO), scientific question answering, or any task requiring factual recall rather than inference. It does not evaluate on open-ended generation tasks where correctness is subjective or multi-dimensional, and where verifier training would require fundamentally different approaches.

Several aspects of the findings could be model- or benchmark-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution and error patterns — a model with different calibration properties, different reasoning strategies, or different types of mistakes might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (GPT, PaLM, LLaMA, etc.). The structural properties of MATH problems — formalizable correctness, clear step-by-step solution structure, the presence of process-based ground truth — may be essential to the method's success and may not transfer to messier reasoning domains.

The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assumption, not a finding. The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation (Section 3.2), means the compute-optimal policy is selected based on approximately 50 questions per fold per difficulty bin. This is a small sample for policy learning — the selected strategies may not be robust or may reflect noise in a particular split. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed differences between strategies (or between oracle and predicted difficulty bins) are statistically reliable at this sample size.

The FLOPs-Matched Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. However, the larger model is trained by scaling only parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the compute-optimal pretraining paradigm where both data and parameters are scaled equally (Hoffmann et al., 2022). The paper acknowledges this: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7).

This is a consequential choice. A Chinchilla-optimal model trained with 14× more total FLOPs — scaling both parameters and training tokens — would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R ≪ 1 in Figure 1, revisions bar chart) may shrink or reverse against a properly compute-optimal larger model.

Furthermore, the 14× larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no verifier-guided selection, no revision chains. This creates an asymmetric comparison: the smaller model receives sophisticated test-time compute strategies while the larger model receives none. A fairer comparison would allocate some of the larger model's inference budget to test-time compute as well — for instance, giving the larger model best-of-8 or best-of-16 selection, which would consume a small fraction of its per-token inference cost but could substantially improve its accuracy. The paper's comparison addresses a specific question (can test-time compute on a small model beat raw greedy decoding from a larger model?), but does not answer the more policy-relevant question (given a total FLOPs budget spanning both training and inference, how should compute be split between model scale and test-time strategies for both models?). The current results should therefore be interpreted as an upper bound on the advantage of test-time compute over pretraining, with the true advantage likely smaller when both sides of the comparison are treated symmetrically.

The Method Cannot Be Applied to Tasks Without Clean Correctness Signals and Closed-Form Answers

The entire pipeline — PRM training, revision model training, difficulty estimation, and answer selection — depends on the existence of ground-truth answers with exact string matching via the MATH grading function (Appendix G). The PRM is trained using Monte Carlo rollouts where the supervision signal is whether a rollout reaches the correct final answer (Section 5.1). The revision model training data is constructed by identifying which of 64 sampled responses are correct and which are incorrect (Section 6.1). Difficulty estimation requires computing pass@1 rates, which requires knowing which answers are correct (or using the PRM as a proxy, which itself was trained using ground-truth correctness). The best-of-N weighted selection mechanism relies on identifying which solutions arrive at the same final answer and summing their scores — a procedure that assumes answers can be deterministically clustered by exact match.

Many important real-world applications lack such clean correctness signals. Open-ended generation tasks (dialogue, creative writing, summarization), complex multi-step planning tasks (where success is partial or multi-dimensional), and tasks requiring subjective evaluation (helpfulness, harmlessness, style) cannot be reduced to exact-match answer checking. Extending the compute-optimal framework to these settings would require fundamentally different approaches to verifier training (e.g., learned reward models trained on human preferences, as in RLHF), difficulty estimation (no pass@1 oracle exists), and answer selection (no majority voting over exact-match clusters). The paper does not address these extensions, and the current method is effectively limited to tasks with formalizable, verifiable correct answers — a restriction that covers math, code (with unit tests), and certain types of logical reasoning, but excludes a large fraction of potential LLM applications. The authors do not discuss this scope limitation, though it is implicit in the choice of MATH as the sole benchmark and the reliance on exact-answer verification throughout the methodology.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes a new design paradigm for low-precision neural network training: structural-correction quantization rather than numerical-format quantization. Prior to this work, the field's approach to reducing training precision below 8 bits was to design custom numerical formats—FP4 (Sun et al., 2020), logarithmic number systems (Chmiel et al., 2021)—that could better accommodate the heavy-tailed distributions of activations and gradients. These methods demonstrated that 4-bit training was conceptually possible but could not deliver actual speedups because the custom formats lacked hardware support. The paper's central methodological shift is to invert the problem: instead of asking what numerical format fits the tensors, ask how to transform the tensors so that a standard, hardware-supported INT4 format works.

This reframing has consequences beyond the specific techniques proposed. It opens a design space where quantization is paired with principled linear transformations or sampling schemes that correct for structural mismatches between tensor statistics and quantizer design. The Hadamard quantizer demonstrates that applying an orthogonal transform before quantization can convert an unquantizable distribution into a quantizable one without losing information—the transform's orthogonality preserves norms, and the matrices' structure causes the transforms to cancel in the matrix multiplication, eliminating runtime overhead. The leverage score sampling procedure demonstrates that importance sampling from the RandNLA literature can substitute for non-uniform quantization, achieving comparable gradient accuracy using only uniform INT4 operations. Both techniques point to a broader principle: the right structural correction can make uniform quantization work for distributions that would otherwise require non-uniform formats.

The paper also resolves a practical contradiction that had stalled progress in 4-bit training. The two prior 4-bit methods—Ultra-low (Sun et al., 2020) and LUQ (Chmiel et al., 2021)—demonstrated training accuracy but required custom hardware, while INT8 training methods (Banner et al., 2018) delivered hardware speedup but could not reach 4 bits. The field lacked a method that was simultaneously accurate at 4 bits and implementable on existing GPUs. This paper provides that method, and in doing so, validates a thesis that had been implicit but unproven: hardware compatibility need not be sacrificed for accuracy if quantization is designed around tensor structure rather than around numerical representation. The 2.2× operator speedup and 35.1% end-to-end training speedup on contemporary GPUs (Figures 4–5) are not just efficiency numbers—they are existence proofs that hardware-compatible 4-bit training is achievable without accuracy collapse.

The paper also reframes gradient sparsity from a nuisance into a resource. Prior work treated the heavy-tailed, sparse nature of gradients as an obstacle to uniform quantization—hence the need for logarithmic or FP4 formats. This paper recognizes that gradient sparsity, driven by overparameterization (Section 4.1: most examples fit well, producing near-zero gradients; a few hard examples produce large gradients), is an opportunity to reallocate precision budget. By dropping approximately half the gradient rows and using the saved computation to represent the retained rows at 8-bit effective precision via bit splitting, LSS turns a statistical property that breaks uniform quantizers into one that enables more accurate gradient computation within the same INT4 operation budget. This insight—that sparsity can be exploited for precision rather than merely for FLOP reduction—connects to broader trends in efficient ML (mixture-of-experts, dynamic sparsity) but applies them at the level of quantization rather than architecture.

The paper does not resolve all tensions in low-precision training. In particular, it establishes a sharp boundary condition: the method targets transformers specifically, and the authors explicitly acknowledge that it "cannot accelerate convolution layers" (Section 6, Limitations). The feature-wise outlier structure that Hadamard quantization exploits arises from LayerNorm interactions in transformer architectures; CNNs have different activation statistics and different computational patterns. This scoping is simultaneously a strength (the techniques are tailored to exploit transformer-specific properties) and a limitation (the paradigm may not extend to the broader space of neural network architectures without discovering architecture-appropriate structural corrections).

Follow-Up Research This Work Enables

Characterizing which orthogonal transforms are optimal for activation quantization in different architectures. The paper uses the Hadamard transform and argues theoretically that it is effective for extreme feature-wise outliers (the one-hot to all-ones example, Section 3.3). But the Hadamard is one choice among many—discrete cosine transforms, Fourier transforms, random orthogonal matrices, and learned transformations are all candidates. A systematic study comparing these transforms on activation quantization error across different model architectures (BERT, GPT, ViT, T5) and tasks would clarify whether the Hadamard is uniquely well-suited or whether any orthogonal matrix works. The key experiment: for each transform, measure the post-transform activation histogram skewness/kurtosis and correlate with downstream 4-bit training accuracy on a standard benchmark like GLUE. If random orthogonal matrices perform similarly to Hadamard, the effect is just decorrelation; if Hadamard is substantially better, its specific structure (equal-magnitude entries, recursive block structure) matters.

Scaling HQ+LSS to large language models (1B+ parameters) and characterizing the accuracy-efficiency Pareto frontier. The current evaluation stops at BERT-large (340M parameters). Section 6 acknowledges that "even INT8 training is still an open problem" for models like OPT-175B. A natural follow-up would apply HQ+LSS to incremental model scales (BERT-base → BERT-large → 1B → 3B → 7B parameters) on a single task (e.g., language modeling perplexity on WikiText or C4) and measure both accuracy degradation relative to FP16 and training throughput. The critical question: does the accuracy gap widen, narrow, or remain constant with scale? Larger models might be more robust to quantization noise (overparameterization provides redundancy) or more sensitive (outliers become more extreme). The cold-start heuristic and adaptive block size selection (Appendix A.4–A.5) would need re-evaluation at scale—the first-2-epochs heuristic may not transfer. A negative result (accuracy collapse above some scale threshold) would be as informative as a positive one, since it would delineate the current method's applicability boundary.

Training a lightweight difficulty/outlier predictor that eliminates the cold-start heuristic and enables dynamic block-size adaptation per layer. The paper's adaptive block size selection (Appendix A.5) computes quantization MSE for candidate block sizes at initialization (or after epoch 2 for pre-training), then fixes the choice. A more dynamic approach would train a small predictor network (or even a simple linear probe on activation statistics) that takes a layer's current activation distribution as input and predicts the optimal Hadamard block size $k$ for the next training step. The training signal would come from the actual quantization MSE observed with different $k$ values during a profiling phase. This would allow per-layer, per-step adaptation as distributions shift during training, potentially reducing the accuracy gap on pre-training tasks where distributions change rapidly. A concrete experiment: compare fixed-$k$ vs. predicted-$k$ on DeiT-Small pre-training (where HQ+LSS underperformed LSQ+LUQ by 0.78 points, Table 1) and on WMT machine translation; measure both final accuracy and the frequency/timing of block-size changes.

Ablating whether unbiasedness in LSS is necessary for training convergence by comparing stochastic importance sampling to deterministic top-K selection. LSS uses Bernoulli sampling with inverse-probability weighting to produce an unbiased gradient estimator (Proposition 4.1). But stochastic gradient descent already uses noisy gradient estimates—is unbiasedness of the quantization step specifically important, or would a biased but lower-variance deterministic selection (keep the top-K gradient rows by leverage score, discard the rest, no reweighting) work equally well or better? A clean comparison on CoLA and GLUE: LSS vs. deterministic top-K with the same compute budget, measuring both convergence speed (steps to reach a given accuracy) and final accuracy. If deterministic top-K performs comparably, it would simplify implementation and eliminate the overhead of random mask generation and inverse-probability scaling. If LSS is substantially better, it validates the theoretical framework and suggests unbiasedness matters beyond the standard SGD noise tolerance.

Combining HQ+LSS with existing distributed training optimizations (ZeRO, pipeline parallelism, gradient checkpointing) and measuring the compound efficiency gains. The paper reports 35.1% end-to-end speedup on BERT-Large fine-tuning on 8× A100 GPUs (Figure 5), but this is a single-node, standard data-parallel setting. Modern large-model training combines multiple efficiency techniques—ZeRO stage 3 (Rajbhandari et al., 2020), pipeline parallelism (Huang et al., 2019), gradient checkpointing, and activation offloading—each of which changes the ratio of computation to communication to memory access. How does INT4 training interact with these techniques? Specifically: (a) INT4 reduces the memory footprint of activations and gradients, potentially reducing the need for activation checkpointing/offloading and its associated recomputation overhead; (b) INT4 reduces the volume of gradient communication in data-parallel settings (gradients are 4× smaller), which could alleviate communication bottlenecks; (c) pipeline parallelism's bubble overhead might change relative to the per-layer speedup from INT4. A systematic evaluation training a 1B–3B parameter GPT-style model with Megatron-DeepSpeed or similar, measuring throughput with and without HQ+LSS under various parallelism configurations, would quantify the compound benefit.

Stress-testing HQ+LSS on adversarial or out-of-distribution data to identify fragility in the outlier and sparsity assumptions. The Hadamard quantizer relies on the assumption that activation outliers have a feature-wise structure (concentrated in specific columns) that Hadamard transforms can amortize. The LSS sampler relies on gradients exhibiting token-wise sparsity due to overparameterization. What happens when these assumptions are violated? Adversarial inputs (designed to produce worst-case activation distributions without clean feature-wise structure) or out-of-distribution data (from a different domain than training) might produce activation and gradient patterns that HQ+LSS handles poorly. A concrete experiment: fine-tune BERT-base with HQ+LSS on a standard GLUE task, then evaluate on an adversarial version of the same task (e.g., ANLI for natural language inference, or text with systematically perturbed inputs designed to activate outlier channels uniformly rather than in a feature-wise pattern). Measure whether the accuracy gap between HQ+LSS and FP16 widens under distribution shift. A negative result (HQ+LSS degrades disproportionately) would identify a fragility that practitioners need to understand; a positive result (the gap remains constant) would strengthen the case for deploying HQ+LSS in real-world settings with unpredictable inputs.

Practical Applications and Downstream Use Cases

Fine-tuning large pre-trained transformers on consumer or edge GPUs with limited memory. The paper's 35.1% training speedup on BERT-Large (Figure 5, Appendix C.4) and the implicit memory savings from INT4 activations and gradients (4× smaller than FP16) directly benefit practitioners fine-tuning moderate-to-large transformers on GPUs with constrained memory, such as the RTX 3090 (24 GB) or A4000 (16 GB). For a concrete scenario: fine-tuning BERT-large (340M parameters) on a text classification task with batch size 32 and sequence length 512 in FP16 uses approximately 18–20 GB of GPU memory (model + optimizer states + activations), fitting on an RTX 3090 but leaving little headroom. Moving activations and gradients to INT4 reduces the activation/gradient memory by ~4×, potentially freeing 3–5 GB, which could be reinvested in larger batch sizes (improving training stability) or longer sequences (handling document-level tasks). The accuracy penalty of 1–2 points on GLUE (Table 1) is acceptable for many applied settings where the alternative is either a smaller model or cloud GPU rental costs.

Training data generation and knowledge distillation pipelines where many model variants are fine-tuned. In scenarios where a large pre-trained model is fine-tuned hundreds or thousands of times—for hyperparameter sweeps, ensembling, or generating synthetic training data via many specialized models—the 1–2% training speedup matters cumulatively. The paper's results show that HQ+LSS preserves most of the accuracy of FP16 fine-tuning (80.81 vs. 82.67 GLUE average for BERT-base, Table 1) while the 2.2× operator speedup (Figure 4) directly reduces the cost per fine-tuning run. For an organization fine-tuning BERT-base on 10 downstream tasks with 50 hyperparameter configurations each (500 total runs, 3 epochs each, 2 hours per run in FP16), switching to HQ+LSS at 35% speedup saves approximately 260 GPU-hours—roughly $500 on cloud instances ($2/hour for an A100-equivalent), or a day of continuous training time on a single GPU.

On-device or privacy-preserving training where full-precision compute is unavailable or expensive. While the paper evaluates on server GPUs (RTX 3090, A100), the fundamental technique—replacing FP16 MMs with INT4 MMs—is even more impactful on hardware where FP16 throughput is limited relative to INT8/INT4 throughput, such as mobile GPUs, edge TPUs, or upcoming AI accelerators optimized for integer arithmetic. The paper's operator speedup on T4 GPUs (Figure 7, Appendix C.5) demonstrates that the speedup persists across GPU generations. For privacy-sensitive applications where training must occur on-device (federated learning, personalization of language models to user data), reducing the per-iteration training cost by 2× can be the difference between feasible and infeasible given battery, thermal, and latency constraints. The key limitation for this use case is the paper's focus on fine-tuning rather than pre-training—on-device training is typically fine-tuning of a pre-trained model, which aligns with the paper's strongest results.