The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm

ArXiv: 2507.18553

🎯 Pitch

GPTQ, the leading method for compressing LLMs to 4‑bit, is shown to be exactly Babai’s nearest plane algorithm — a classic lattice solver — which instantly gives the first tight, layer‑wise error bound and explains why the method works. The authors then design clipping‑free quantization schemes that rigorously satisfy the bound and outperform standard GPTQ, doubling inference speed on the resulting representations.

---

1. Executive Summary

This paper proves that GPTQ — the standard one-shot post-training quantization method for LLMs — is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) when executed back-to-front on the lattice defined by a layer's Hessian matrix. Working with the Qwen3 model family, the authors establish that GPTQ's error propagation step — previously described as a sequence of ad-hoc algebraic updates — gains an intuitive geometric interpretation as an orthogonal projection onto the nearest hyperplane of the lattice basis, and this equivalence imports Babai's tight worst-case error bound (trace of the LDL decomposition's diagonal matrix) into the no-clipping quantization regime. Leveraging this theoretical foundation, the paper introduces two overflow-tolerant methods — SSQR (scale-adjusted sparse-quantized representation with binary-searched per-channel scaling to control outlier density) and HPTQ (Huffman-encoded post-training quantization using entropy-guided scale selection to produce uneven-bitwidth representations) — that avoid weight clipping entirely, with HPTQ achieving Pareto-optimal perplexity on Qwen3-8B at 3.125 bits (10.34 WikiText-2 perplexity vs. 9.73 for BF16, outperforming standard GPTQ at the same bitwidth by 2.43 points) and SSQR delivering approximately 2× end-to-end inference speedup via a custom CUDA kernel on Ampere GPUs across multiple bitwidth and outlier-rate configurations, establishing that lattice-theoretic guarantees translate to practical accuracy gains only when clipping — which violates the CVP equivalence — is removed from the quantization pipeline.

2. Context and Motivation

The Core Problem: GPTQ Works Empirically, But No One Knows Why

The fundamental question this paper tackles is theoretical rather than empirical: why does GPTQ's greedy, coordinate-by-coordinate quantization procedure work so well globally, and what are its guarantees? GPTQ (Frantar et al., 2023) has become one of the standard methods for compressing LLM weights to 4-bit precision with minimal accuracy loss, yet its inner workings are described in the original paper as a sequence of algebraic operations with no clear geometric meaning and no worst-case performance bounds. The procedure picks a weight, quantizes it (with rounding or clipping), then optimally updates all remaining unquantized weights to compensate for the introduced error, then moves to the next weight. This is a greedy local rule — there is no a priori reason it should produce a globally good solution, and the original GPTQ paper provides no theoretical justification for why the accumulated error across hundreds of sequential greedy steps remains tightly controlled.

This gap is significant for two reasons. First, from an engineering perspective, the absence of error bounds means practitioners cannot predict how far the quantized layer's output will deviate from the original layer's output before running the algorithm. This forces trial-and-error calibration (varying group sizes, damping factors, and quantization orders) without principled guidance on which choices actually tighten the error envelope. Second, from a scientific perspective, the lack of geometric understanding obscures connections to decades of work on lattice algorithms that could directly inform the design of better quantizers. The paper's central insight is that GPTQ's ad-hoc algebraic steps are not ad-hoc at all — they are an instance of Babai's nearest plane algorithm (Babai, 1986) applied to the lattice induced by the layer's input Hessian, and this equivalence immediately imports approximation guarantees that have been known in the lattice theory community for nearly 40 years.

The Practical Stakes: LLM Deployment Economics

The practical motivation is the enormous cost of deploying large language models. As the authors note, generative pre-trained transformers contain hundreds of billions of parameters and impose massive computational and memory costs (Luccioni et al., 2024). Post-training quantization (PTQ) from 16-bit to 4-bit or lower representation is the de facto approach to fit these models onto affordable accelerators (Gholami et al., 2021). Every bit of precision that can be squeezed out of the weights without degrading accuracy translates to roughly proportional reductions in memory footprint, memory bandwidth pressure, and — when paired with efficient kernels — inference latency.

GPTQ's empirical success in the 4-bit regime made it a cornerstone of the LLM deployment ecosystem. However, as the paper demonstrates in its experiments (Section 5, Figure 4a), standard GPTQ degrades sharply at lower bitwidths — on Qwen3-8B, standard GPTQ at 3.125 bits achieves 12.77 WikiText-2 perplexity (vs. 9.73 for BF16), and at 2.125 bits it collapses to 57.51. The paper's geometric analysis reveals a key reason for this degradation: weight clipping — the practice of capping overflowed quantized values to the representable range of the quantization grid — severs the equivalence to Babai's algorithm and voids the error bound. This means GPTQ in its standard clipped form operates in a regime with no theoretical guarantees, and the greedy error propagation can amplify clipping-induced errors rather than compensating for them.

Conflicting Signals in Prior Theoretical Work

The paper positions itself relative to two bodies of prior work that provide partial but incomplete theoretical footing for GPTQ:

Second-order compression lineage. The idea of using Hessian information to guide parameter removal traces back to Optimal Brain Damage (LeCun et al., 1989) and Optimal Brain Surgeon (OBS) (Hassibi et al., 1993), which use second-order Taylor expansions of the loss to identify which weights can be removed with minimal impact. Optimal Brain Compression (OBC) (Frantar & Alistarh, 2022) generalizes this framework to the post-training setting and unifies structured pruning and quantization under a single exact solver — this is the direct predecessor of GPTQ. OBC's error propagation formula (Equation 2 in the paper) is mathematically optimal for each individual weight update, but OBC dynamically selects which weight to quantize next based on which choice minimizes the expected output error (Equation 1). This dynamic selection requires recomputing the inverse Hessian after each update, giving OBC quartic complexity in the input dimension — far too expensive for LLM-scale models with hidden dimensions in the thousands.

GPTQ's key innovation was to fix the quantization order in advance (e.g., front-to-back) so that the inverse Hessian is computed once and shared across all output channels, reducing complexity to cubic in the input dimension. But this raised an obvious question: does fixing the order — essentially discarding OBC's greedy dimension selection — break the optimality guarantees that OBC inherits from the OBS framework? The GPTQ paper provided no answer, and prior to the current work, there was no analysis of how much error the fixed-order greedy procedure accumulates.

QuIP's LDLQ formulation and guarantees. Chee et al. (2023) introduced QuIP, which proves an error guarantee for GPTQ and proposes LDLQ as an equivalent variant. This represented the first theoretical progress toward understanding GPTQ's behavior. However, QuIP's analysis works within the algebraic framework of GPTQ's weight-space updates — it does not reveal the geometric structure of the problem or connect to lattice algorithms. The current paper goes substantially further by showing that the entire procedure is an instance of a well-studied lattice algorithm with decades of analysis behind it, importing not just error bounds but also geometric intuition (the orthogonal walk through nested affine subspaces), connections to basis reduction techniques, and the relationship between quantization order and the Gram-Schmidt orthogonalization process.

Where Prior Approaches Fall Short

The paper identifies three specific limitations in the existing theoretical understanding:

No geometric interpretation. Prior to this work, GPTQ was understood purely algebraically — as a sequence of matrix-vector updates involving the LDL decomposition of the inverse Hessian. There was no geometric picture of what these updates mean in the activation space. The paper shows that each error propagation step is an orthogonal projection onto the nearest hyperplane of the lattice basis defined by the Hessian — providing an intuitive visualization (Figures 2 and 3) of why the greedy procedure works: it walks orthogonally through a nested sequence of affine subspaces, with each step's error being orthogonal to all subsequent projection directions.

No worst-case guarantees for the standard GPTQ. While QuIP provided some guarantees, standard GPTQ with clipping operates without any theoretical safety net. The paper shows that clipping breaks the CVP equivalence (the clipped quantization grid \mathbb{Z}^{\dagger} \subset \mathbb{Z} does not form a lattice), meaning that all of Babai's approximation guarantees are void when weights are forced into a bounded integer range. This explains a phenomenon that practitioners have observed but not understood: GPTQ's accuracy degrades more sharply than expected at very low bitwidths, because the clipping errors that become frequent at low bitwidths are not properly compensated by the error propagation mechanism.

No bridge to lattice theory. The CVP and lattice basis reduction are mature fields with decades of algorithmic development — LLL reduction (Lenstra et al., 1982), BKZ reduction (Kannan, 1987), and numerous CVP approximation algorithms. If quantization could be formulated as a lattice problem, the entire toolkit of lattice algorithms becomes available for designing better quantizers. The paper explicitly frames this as a two-way channel: "decades of CVP heuristics can refine practical quantizers, while the behavior of massive neural networks may, in turn, inspire new questions for lattice theory" (Section 6).

How This Paper Positions Itself

The paper's contribution is conceptual unification rather than a new algorithm per se. It shows that:

1. The L2 quantization problem is a CVP (Theorem 1, Section 4.1): minimizing \| \mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i \|_2 over integer vectors \mathbf{z}_i is exactly the classical closest vector problem on the lattice with basis \mathbf{B} = \mathbf{X} \text{diag}(\mathbf{s}_i) and target \mathbf{y} = \mathbf{X} \mathbf{w}_i. This equivalence holds for any factorization of the Hessian matrix, not just the original input matrix, enabling computational simplifications.

2. OBQ's error propagation is Babai's projection (Theorem 2, Section 4.2): Each OBQ error propagation step (Equation 2) is mathematically identical to projecting the target vector onto the nearest hyperplane orthogonal to a Gram-Schmidt vector of the basis — exactly what Babai's algorithm does. The paper provides both a geometric proof (using the geometry of the projection plane and the inverse basis, Figure 2) and an algebraic proof (Appendix B) establishing this equivalence rigorously.

3. GPTQ executed back-to-front is Babai's algorithm without basis reduction (Theorem 4, Section 4.3): Once the CVP equivalence and the OBQ-Babai connection are established, showing that GPTQ (with fixed order) equals Babai's algorithm (with fixed basis order) is a matter of aligning the iteration directions. GPTQ's standard front-to-back order becomes Babai's algorithm if the iteration direction is reversed — a superficial difference. The paper also proves that composing Babai's algorithm with additional GPTQ-style error propagation steps yields no improvement (Appendix B.4), confirming the equivalence is tight.

4. GPTQ inherits Babai's tight error bound in the no-clipping regime (Theorem 5, Section 4.4): The absolute quantization error per output channel satisfies \|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2 \leq \frac{1}{4} (\mathbf{T}^{-1} \mathbf{s}_i)^{\top} \mathbf{D} (\mathbf{T}^{-1} \mathbf{s}_i) where \mathbf{D} is the diagonal matrix of the LDL decomposition of the permuted Hessian, and the quantization order \mathbf{T} controls which entries of \mathbf{D} are multiplied by which scale factors. This bound is tight — equality is attained when the target weight vector lies at a corner of Babai's orthogonal hyper-cuboid — and provides the first principled guidance for choosing the quantization order (minimize \text{tr}(\mathbf{D}) under the permutation).

A crucial subtlety: the error bound holds only in the no-clipping regime where \mathbb{Z}^{\dagger} = \mathbb{Z} (unrestricted integers). The paper's practical contribution — the SSQR and HPTQ methods — is motivated directly by this observation. Standard GPTQ clips overflowed integers to the quantization grid's representable range, violating the CVP equivalence and voiding the bound. The paper's methods instead eliminate clipping entirely by representing overflowed weights as sparse outliers (SSQR) or by using Huffman coding to accommodate the full integer range at a variable bitwidth (HPTQ), thereby preserving the lattice structure and the error guarantee.

The Broader Research Agenda

By establishing that LLM weight quantization is a lattice problem, the paper opens specific research directions that go beyond its own experimental contributions:

• Basis reduction for quantization: LLL or BKZ basis reduction could find a shorter, more orthogonal lattice basis (Figure 1b) that reduces Babai's error bound, potentially enabling lower-bitwidth quantization with the same accuracy. However, as the paper notes, basis reduction is scale-sensitive — different scale vectors \mathbf{s}_i for different output channels would require different basis reductions, which is computationally prohibitive for batched LLM quantization. Developing scale-aware or scale-agnostic reduction techniques is left for future work.

• Connection to quantization order: The paper shows that the quantization order corresponds to the pivot order in the LDL decomposition of the Hessian, and that minimizing \text{tr}(\mathbf{D}) tightens the error bound. GPTQ's existing "act-order" heuristic (descending Hessian diagonal) is reinterpreted as a cheap approximation to this minimization. The paper's proposed "min-pivot" order (Algorithm 3), which greedily selects the minimum diagonal entry at each LDL step, is shown to reduce \text{tr}(\mathbf{D}) by 35-50% relative to baseline orders on Qwen3-8B (Table 2, Appendix C.3), though downstream accuracy gains are modest — consistent with the interpretation that act-order already captures most of the benefit when the Hessian is well-conditioned.

• Extension to clipping-aware guarantees: The current error bound applies only in the no-clipping regime. Extending the lattice framework to account for finite quantization grids — where the CVP becomes a bounded CVP or a CVP with box constraints — is identified as an immediate next step and is likely necessary for making the theoretical bounds tight at very low bitwidths where even the no-clipping methods still degrade (HPTQ at 2.125 bits on Qwen3-8B achieves 13.97 WikiText-2 perplexity, far above the BF16 baseline of 9.73).

In summary, the paper addresses a gap that is simultaneously theoretical (why does GPTQ work, and what are its guarantees?) and practical (how can we push to lower bitwidths without losing the theoretical safety net that clipping removes?). Its positioning is that GPTQ is not an isolated engineering trick but rather an instance of a classical lattice algorithm, and that recognizing this unlocks both rigorous guarantees and a roadmap for importing lattice-theoretic advances into LLM compression.

3. Technical Approach

3.1 Reader Orientation

This paper is a theoretical analysis paper that reinterprets an existing algorithm (GPTQ) through the lens of lattice geometry rather than proposing a new quantization method from scratch. The core idea is to prove that GPTQ — when executed in the right direction and without weight clipping — is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP), and that this equivalence imports tight worst-case error bounds that explain why and when GPTQ works, while also motivating practical no-clipping quantization schemes that preserve those guarantees. The "system" being analyzed is not a new piece of software but rather a mathematical framework that connects linear-layer quantization, the Hessian matrix of layer inputs, and lattice basis reduction through a chain of equivalences: quantization-as-CVP → OBQ-as-Babai-projection → GPTQ-as-Babai-without-reduction.

3.2 Big-Picture Architecture (Diagram in Words)

The theoretical framework has five major components, connected through a chain of equivalences rather than a data flow:

1. Quantization-CVP dictionary (Section 4.1, Table 1): A mapping that translates every concept from linear-layer quantization (input activations, scales, weights, integer representations, dequantized outputs, target activation vectors) into the language of lattice problems (basis vectors, basis stretches, lattice basis, integer coordinates, floating-point coordinates, external target vector). This dictionary is the Rosetta Stone that makes all subsequent equivalences possible.

2. OBQ's error propagation as Babai's projection (Section 4.2, Theorem 2): A geometric proof that the Optimal Brain Quantizer's per-step error correction formula — previously understood as an algebraic update derived from the inverse Hessian — is exactly the operation of projecting a target vector onto the nearest hyperplane of a lattice basis, which is the core step in Babai's nearest plane algorithm.

3. GPTQ-Babai equivalence modulo ordering (Section 4.3, Theorem 4): The central theorem showing that GPTQ (fixed quantization order) and Babai's algorithm (fixed basis order, no LLL reduction) produce identical results when their iteration directions are aligned. This requires a three-step algebraic proof (Appendix B) establishing that GPTQ's front-to-back weight updates are equivalent to Babai's back-to-front target projections under a change of variables.

4. Error bound inheritance (Section 4.4, Theorem 5): Once the equivalence is established, Babai's known tight error bound — the sum of squared lengths of the Gram-Schmidt vectors of the basis, which equals the trace of the diagonal matrix from the LDL decomposition of the permuted Hessian — transfers directly to GPTQ in the no-clipping regime. This bound provides a closed-form expression for the worst-case layer-wise L2 output error as a function of the quantization order and per-channel scales.

5. Quantization order optimization (Section 4.5, Algorithm 3): Since the error bound depends on the pivot order of the LDL decomposition, different quantization orders change the bound. The paper analyzes GPTQ's existing "act-order" heuristic (descending Hessian diagonal) and proposes a "min-pivot" order (greedy minimum diagonal at each LDL step) that consistently reduces the bound's trace term, though with modest downstream accuracy gains because the Hessian is typically well-conditioned.

The chain of reasoning flows forward: start with the quantization objective → recognize it as a CVP (Theorem 1) → show that OBQ's per-step correction is Babai's projection (Theorem 2) → note that OBQ's dimension selection minimizes the projection error at each step (Corollary 3) → prove that GPTQ, which fixes the order, is Babai's algorithm without basis reduction (Theorem 4) → import Babai's error bound (Theorem 5) → optimize the quantization order to tighten the bound (Section 4.5). The practical methods (SSQR, HPTQ, Section 5) are downstream applications that enforce no-clipping to preserve the lattice structure that makes the bound valid.

3.3 Roadmap for the Deep Dive

• First, the L2 quantization objective and its CVP formulation (Theorem 1): This establishes the mathematical language in which all subsequent equivalences are expressed. Understanding why minimizing \|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2 over integer \mathbf{z}_i is the same as finding the closest lattice point to a target vector is prerequisite to everything that follows.

• Second, the OBQ algorithm and its geometric interpretation (Theorem 2, Corollary 3): OBQ is GPTQ's slower but dynamically-ordered predecessor. Analyzing OBQ first makes GPTQ's behavior comprehensible because OBQ's dynamic dimension selection has a clean geometric meaning (choosing the nearest hyperplane), and GPTQ can then be understood as OBQ with a pre-committed order.

• Third, the algebraic proof of GPTQ-Babai equivalence (Theorem 4, Appendix B): This is the paper's main technical contribution. The three-step proof (rewriting GPTQ in terms of absolute error, reversing the iteration, mapping to Babai's variables) requires careful tracking of how the LDL decomposition, UDU decomposition, and Cholesky decomposition relate through permutations and matrix inverses.

• Fourth, the error bound and its dependence on quantization order (Theorem 5, Section 4.5): With the equivalence established, Babai's bound transfers automatically. The key practical insight is that the quantization order — previously a heuristic choice — now has a principled optimization criterion: minimize the trace of the LDL diagonal under permutation, weighted by the squared per-channel scales.

• Fifth, the Batched Babai Quantization algorithm (Algorithm 4, Appendix A): This algorithm translates Babai's nearest plane procedure into a computationally efficient batched form suitable for LLM-scale linear layers. Understanding how the Cholesky factor, permutation matrix, and per-channel scales interact is essential for seeing why GPTQ's computational efficiency (cubic rather than quartic) is preserved in the geometric formulation.

• Sixth, the no-clipping practical methods (Algorithms 9–12, Appendix D.1): SSQR and HPTQ are the engineering consequences of the theory. SSQR uses binary search on per-channel scales to control outlier density while avoiding clipping; HPTQ uses Huffman coding cost as a proxy for bitwidth in a similar binary search, producing uneven-bitwidth representations that stay within the integer lattice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical analysis paper that proves an equivalence between two algorithms from different fields and explores the consequences. The technical approach proceeds through a chain of definitions, lemmas, and theorems that build on each other cumulatively.

---

The Linear-Layer Quantization Problem (Formal Definition)

The paper begins by formally defining the quantization problem that GPTQ solves, establishing notation that will be used throughout all subsequent proofs. Understanding this definition precisely is critical because every theorem is stated in terms of these variables.

Problem statement (Section 3.1). Let $\mathbf{X} = [\mathbf{x}_1, \ldots, \mathbf{x}_n]^{\top} \in \mathbb{R}^{n \times c}$ be the sampled calibration input data — a matrix of $n$ input vectors (batch size), each of dimension $c$ (the input dimension of the linear layer). The rank condition $n \geq c = \text{rank}(\mathbf{X})$ ensures that the Hessian matrix $\mathbf{X}^{\top} \mathbf{X}$ is invertible (possibly with a damping factor added, as discussed below). Let $\mathbf{W} = [\mathbf{w}_1, \ldots, \mathbf{w}_r] \in \mathbb{R}^{c \times r}$ be the linear layer weights — $c$ input features, $r$ output features, with each column $\mathbf{w}_i \in \mathbb{R}^c$ being the weight vector for one output channel. Let $\mathbf{S} = [\mathbf{s}_1, \ldots, \mathbf{s}_r] \in \mathbb{R}_{\neq 0}^{c \times r}$ be the non-zero quantization scales — one scale per weight element, allowing for arbitrary grouping patterns (e.g., per-channel, per-group). The key assumption is that $\mathbf{S}$ is statically computed before any weight updates, using methods like AbsMax or MSE — this is standard in post-training quantization and means the scales are fixed inputs to the algorithm, not learned during quantization.

The unknown variable to be solved for is $\mathbf{Z} = [\mathbf{z}_1, \ldots, \mathbf{z}_r] \in \mathbb{Z}^{\dagger c \times r}$ — the quantized integer representations — where $\mathbb{Z}^{\dagger} \subseteq \mathbb{Z}$ is the quantization grid. In the clipping setting (standard GPTQ with e.g., INT4), $\mathbb{Z}^{\dagger} = \{-8, \ldots, -1, 0, 1, \ldots, 7\}$ — a bounded symmetric range. In the no-clipping setting (used for the theoretical analysis and the paper's proposed methods), $\mathbb{Z}^{\dagger} = \mathbb{Z}$ — all integers are allowed, which means the quantization grid forms a full lattice rather than a bounded subset. The dequantized weights are $\mathbf{Q} = [\mathbf{q}_1, \ldots, \mathbf{q}_r] \in \mathbb{R}^{c \times r}$ where each $\mathbf{q}_i = \text{diag}(\mathbf{s}_i) \mathbf{z}_i$ — the element-wise product of the scale vector and the integer vector for that output channel.

The optimization objective. The goal is to minimize the L2 error on the layer output:

$\text{argmin}_{\mathbf{z}_i \in \mathbb{Z}^{\dagger c}} \|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2 \quad \text{for all } 1 \leq i \leq r$

This decomposes across output channels because the total squared error separates as the sum of per-channel squared errors:

$\|\mathbf{X} \mathbf{Q} - \mathbf{X} \mathbf{W}\|_2^2 = \sum_{i=1}^r \|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2$

What this computes: For each output channel $i$, we want to find integer weights $\mathbf{z}_i$ such that, when scaled element-wise by $\mathbf{s}_i$ and multiplied by the input matrix $\mathbf{X}$, the resulting output activations are as close as possible (in Euclidean norm) to the original output activations $\mathbf{X} \mathbf{w}_i$. This is a per-channel optimization because the error contributions from different output channels are additive and independent — quantizing channel 1's weights does not affect channel 2's output error.

Why this form: The L2 objective on the layer output (rather than directly on the weights) captures what actually matters for the model's forward pass — two weight matrices that produce the same output activations on the calibration data are functionally equivalent, even if the weights themselves differ. Using the L2 norm makes the problem a least-squares integer programming problem, which is exactly the form of the closest vector problem in lattices. The decomposition across output channels is the reason GPTQ can process all channels simultaneously with a shared inverse Hessian — the optimization for different $\mathbf{w}_i$ uses the same input statistics $\mathbf{X}^{\top} \mathbf{X}$ and differs only in the scale vectors $\mathbf{s}_i$ and target weight vectors $\mathbf{w}_i$.

The damping factor. The paper includes a small damping term $\lambda \in \mathbb{R}_+$ when computing the Hessian matrix, so that $\mathbf{H} = \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}$ rather than $\mathbf{H} = \mathbf{X}^{\top} \mathbf{X}$. The typical choice is $\lambda = \frac{1}{100c} \sum_{j=1}^c (\mathbf{X}^{\top} \mathbf{X})[j, j] = \frac{1}{100c} \|\mathbf{X}\|_2^2$ — one hundredth of the average diagonal entry of the Hessian, or equivalently one hundredth of the normalized squared Frobenius norm of the input matrix. This damping ensures the Hessian is strictly positive definite (full rank) even when the calibration data has fewer samples than input dimensions or when columns are nearly collinear, preventing numerical instability in the Cholesky or LDL decomposition. The damping is a practical necessity inherited from GPTQ and does not fundamentally change the geometric interpretation — it simply adds a small isotropic regularizer to the lattice basis.

---

OBQ Algorithm: The Dynamic Predecessor (Section 3.1, Equations 1–2)

Before analyzing GPTQ, the paper reviews OBQ (Optimal Brain Quantizer), which is GPTQ's computationally more expensive predecessor. Understanding OBQ is essential because OBQ's per-step operations are exactly what GPTQ performs, just in a different order, and OBQ's dynamic dimension selection has a clean geometric interpretation that illuminates why GPTQ's fixed order works.

OBQ's dimension selection (Equation 1). At each step, let $\mathbf{J}_i$ (abbreviated $\mathbf{J}$) be the set of not-yet-quantized indices for output channel $i$. OBQ chooses the next dimension to quantize as:

$j \leftarrow \text{argmin}_{j \in \mathbf{J}} \frac{(\mathbf{q}_i[j] - \mathbf{w}_i[j])^2}{(\mathbf{X}[:, \mathbf{J}]^{\top} \mathbf{X}[:, \mathbf{J}])^{-1}[j, j]}$

where $\mathbf{q}_i[j]$ is the quantized (rounded) value for weight element $j$, $\mathbf{w}_i[j]$ is the original floating-point weight, and the denominator is the $j$-th diagonal entry of the inverse of the Hessian submatrix restricted to the unquantized dimensions.

What this computes: For each remaining unquantized dimension $j$, OBQ computes the squared rounding error $(\mathbf{q}_i[j] - \mathbf{w}_i[j])^2$ divided by the $j$-th diagonal entry of the inverse Hessian, and picks the dimension that minimizes this ratio. The denominator can be interpreted as the sensitivity of the output to changes in weight $j$, given that the other unquantized weights can be adjusted to compensate. A small diagonal entry in the inverse Hessian means the weight is in a direction where the Hessian has large curvature — changing this weight would cause large output changes that are hard to compensate for — so OBQ prioritizes quantizing weights in less sensitive directions first, leaving the more sensitive weights for later when the error propagation has more remaining degrees of freedom to absorb the quantization error.

Why this form: Corollary 3 (proved later) shows that this dimension selection criterion is exactly choosing the dimension whose nearest hyperplane is closest to the current target residual vector. In geometric terms, OBQ always projects onto the hyperplane that is nearest to the current residual, minimizing the projection error at each step. This is a greedy algorithm — it makes the locally optimal choice at each step without lookahead — and it is the source of OBQ's name: it performs "optimal brain surgery" by removing (quantizing) the least damaging weight first.

OBQ's error propagation (Equation 2). After quantizing dimension $j$, OBQ updates all remaining unquantized weights $\mathbf{w}_i[j']$ for $j' \in \mathbf{J}$ by:

$\Delta \mathbf{w}_i[j'] \leftarrow \frac{(\mathbf{X}[:, \mathbf{J}]^{\top} \mathbf{X}[:, \mathbf{J}])^{-1}[j', j]}{(\mathbf{X}[:, \mathbf{J}]^{\top} \mathbf{X}[:, \mathbf{J}])^{-1}[j, j]} (\mathbf{q}_i[j] - \mathbf{w}_i[j])$

where $\mathbf{q}_i[j] - \mathbf{w}_i[j]$ is the quantization error introduced at dimension $j$ (negative if the quantized value is smaller than the original), and the ratio of inverse Hessian entries determines how this error is distributed among the remaining unquantized weights.

What this computes: The error introduced by rounding weight $j$ is optimally compensated by adjusting all remaining unquantized weights $j'$ in proportion to the off-diagonal entry $[j', j]$ of the inverse Hessian divided by the diagonal entry $[j, j]$. This is the optimal linear compensation — it minimizes the increase in the L2 output error caused by the quantization of weight $j$, given that the other weights can be adjusted. The formula comes from solving a constrained least-squares problem: minimize the output error after adjusting the unquantized weights, subject to the constraint that weight $j$ is fixed to its rounded value.

Why this form: The ratio $(\mathbf{H}^{-1})[j', j] / (\mathbf{H}^{-1})[j, j]$ is the coefficient of the optimal linear predictor of the $j'$-th coordinate from the $j$-th coordinate under the Gaussian approximation implicit in the second-order Taylor expansion of the loss. In geometric terms (Theorem 2), this is exactly the projection of the quantization error onto the basis directions, as we will see.

Computational cost of OBQ. The dynamic dimension selection requires recomputing (or updating) the inverse Hessian of the remaining dimensions after each quantization step, because the denominator in Equation 1 changes as $\mathbf{J}$ shrinks. This gives OBQ $O(c^4)$ total complexity for a weight matrix of input dimension $c$ — each of the $c$ steps requires updating the $c \times c$ inverse Hessian, which is $O(c^3)$ if done naively, and there are $r$ output channels that require separate dimension selection (since different channels have different weight vectors and may benefit from different quantization orders). For LLM-scale models with $c$ in the thousands, this is prohibitively expensive.

---

GPTQ Algorithm: Fixed-Order Batched Quantization (Section 3.1, Algorithm 1)

GPTQ reduces OBQ's complexity from quartic to cubic by fixing the quantization order in advance — the dimensions are processed in a predetermined sequence (e.g., from 1 to $c$) that is the same for all $r$ output channels. This means the Hessian information (the ratios of inverse Hessian entries needed for error propagation) can be precomputed once using the LDL decomposition and shared across all output channels, eliminating the per-channel, per-step recomputation.

Algorithm 1 walkthrough (line-by-line, omitting the blocking mechanism which only affects memory access patterns and speed, not numerical results):

Inputs: $\mathbf{W} \in \mathbb{R}^{c \times r}$ (weights), $\mathbf{S} \in \mathbb{R}_{\neq 0}^{c \times r}$ (scales), $\mathbf{X} \in \mathbb{R}^{n \times c}$ (calibration inputs), $\mathbf{P} \in \{0, 1\}^{c \times c}$ (permutation matrix — the quantization order; the default is $\mathbf{P} = \mathbf{I}$, front-to-back), $\lambda \in \mathbb{R}_+$ (damping factor), $\mathbb{Z}^{\dagger} \subseteq \mathbb{Z}$ (quantization grid).

Line 3: Hessian computation. The damped Hessian is computed as $\mathbf{H} \leftarrow \mathbf{P}^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}) \mathbf{P}$. The permutation $\mathbf{P}$ reorders the rows and columns of the Hessian to match the desired quantization order before the LDL decomposition is computed.

Line 4: LDL decomposition. $\mathbf{L} \leftarrow \text{LDL}(\mathbf{H}^{-1})$ computes the LDL decomposition of the inverse of the permuted Hessian. An LDL decomposition factors a symmetric positive definite matrix $\mathbf{A}$ as $\mathbf{A} = \mathbf{L} \mathbf{D} \mathbf{L}^{\top}$ where $\mathbf{L}$ is lower triangular with ones on the diagonal, and $\mathbf{D}$ is a positive diagonal matrix. The lower triangular matrix $\mathbf{L}$ stores the ratios $(\mathbf{H}^{-1})[j', j] / (\mathbf{H}^{-1})[j, j]$ that appear in OBQ's error propagation formula — specifically, $\mathbf{L}[j', j] = (\mathbf{H}^{-1})[j', j] / (\mathbf{H}^{-1})[j, j]$ for $j' \geq j$, which is exactly the coefficient needed to propagate the quantization error from the $j$-th dimension to the $j'$-th dimension when processing in the permuted order.

Line 5: Permute weights and scales. $\mathbf{W}, \mathbf{S} \leftarrow \mathbf{P}^{-1} \mathbf{W}, \mathbf{P}^{-1} \mathbf{S}$ applies the inverse permutation to bring the weights and scales into the quantization order. After this step, the first row of $\mathbf{W}$ corresponds to the first dimension to be quantized, the second row to the second, and so on.

Lines 6–14: Main quantization loop. The algorithm initializes $\mathbf{Q} \leftarrow \mathbf{W}$ (dequantized weights start as the original weights, to be overwritten) and $\mathbf{Z} \leftarrow \mathbf{0}$ (integer representations start as zero). For each dimension $j$ from 1 to $c$ (the fixed quantization order):

• Line 9: Compute the floating-point value before rounding: $\boldsymbol{\zeta} \leftarrow \mathbf{W}[j, :] / \mathbf{S}[j, :]$ — element-wise division of the $j$-th row of weights by the $j$-th row of scales. This converts the weight values (which include accumulated error propagation from previous steps) back into the "integer space" by undoing the per-channel scaling. $\boldsymbol{\zeta}$ is a row vector of length $r$ (one value per output channel).

• Line 10: Quantize by rounding: $\mathbf{Z}[j, :] \leftarrow \text{Round}(\boldsymbol{\zeta}, \mathbb{Z}^{\dagger})$ rounds each element of $\boldsymbol{\zeta}$ to the nearest value in the quantization grid $\mathbb{Z}^{\dagger}$. In the clipping setting, values outside the representable range are clipped to the nearest boundary; in the no-clipping setting, any integer is allowed.

• Line 11: Dequantize: $\mathbf{Q}[j, :] \leftarrow \mathbf{Z}[j, :] * \mathbf{S}[j, :]$ — element-wise product of the rounded integers and the scales, producing the dequantized weight values for row $j$.

• Line 12: Compute the quantization error: $\boldsymbol{\varepsilon} \leftarrow \mathbf{Q}[j, :] - \mathbf{W}[j, :]$ — a row vector of length $r$ giving the error introduced by quantizing the $j$-th row, for each output channel.

• Line 13: Propagate the error: $\mathbf{W}[j:, :] \leftarrow \mathbf{W}[j:, :] + \mathbf{L}[j:, j] \boldsymbol{\varepsilon}$ — the error row vector $\boldsymbol{\varepsilon}$ is multiplied by the sub-column $\mathbf{L}[j:, j]$ (the $j$-th column of $\mathbf{L}$, from row $j$ downward) and added to all unquantized rows (rows $j$ through $c$). This is a rank-1 update: the vector $\mathbf{L}[j:, j]$ (of length $c-j+1$) outer-products with the row vector $\boldsymbol{\varepsilon}$ (of length $r$) and is added to the bottom submatrix of $\mathbf{W}$. Because $\mathbf{L}$ is lower triangular with unit diagonal, $\mathbf{L}[j, j] = 1$, which means the $j$-th row itself is updated by exactly $\boldsymbol{\varepsilon}$ — this cancels the quantization error in the weight space, resetting $\mathbf{W}[j, :]$ to its pre-quantization value plus compensation from previous steps, so that the error is "pushed forward" to the remaining unquantized weights.

After the loop, the outputs are permuted back to the original order: $\mathbf{Z}, \mathbf{Q} \leftarrow \mathbf{P} \mathbf{Z}, \mathbf{P} \mathbf{Q}$.

What this loop computes: At each step $j$, the algorithm (1) rounds the current weight values for row $j$ to the quantization grid, (2) records the error introduced, and (3) distributes this error among all subsequent rows $j' \geq j$ in proportion to the LDL factor $\mathbf{L}[j', j]$. This is the exact same optimal compensation as OBQ's Equation 2, but applied in a pre-determined row-by-row sweep rather than a dynamically chosen per-element order. The key computational saving is that $\mathbf{L}$ is computed once from the LDL decomposition of the full inverse Hessian, rather than being recomputed after each quantization step with a shrinking set of unquantized dimensions.

Why GPTQ processes all output channels simultaneously: The inner loop (lines 9–13) operates on entire rows of $\mathbf{W}$, $\mathbf{S}$, $\mathbf{Z}$, and $\mathbf{Q}$ — i.e., on all $r$ output channels at once — because the LDL factor $\mathbf{L}$ depends only on the input statistics $\mathbf{X}^{\top} \mathbf{X}$ and the quantization order $\mathbf{P}$, not on the weight values themselves. The error propagation coefficient for dimension $j'$ given quantization of dimension $j$ is the same for all output channels; only the error magnitude $\boldsymbol{\varepsilon}$ differs per channel. This is why GPTQ is "batched": the expensive matrix operations (LDL decomposition) are amortized across $r$ output channels, and the per-channel computation reduces to simple row operations.

Why the fixed order works (preview of the geometric interpretation): The fixed order is not arbitrary — it corresponds to choosing a basis for the lattice and processing the Gram-Schmidt vectors in that order. Theorem 4 will show that when the order is reversed (back-to-front), this is exactly Babai's nearest plane algorithm without basis reduction. The front-to-back order that GPTQ uses by default is simply the reverse of Babai's default order, and the paper's key insight is that this superficial difference is the only thing separating GPTQ from a classical lattice algorithm.

---

The Closest Vector Problem (CVP) and Babai's Algorithm (Section 3.2)

To establish the equivalence, the paper must first define the CVP and Babai's algorithm in the language of lattices, so that the mapping from quantization concepts to lattice concepts is precise.

CVP definition. Let $\mathbf{B} = [\mathbf{b}_1, \ldots, \mathbf{b}_c] \in \mathbb{R}^{n \times c}$ be a set of $c$ basis vectors, each of dimension $n$, with $n \geq c = \text{rank}(\mathbf{B})$ (full column rank — the basis vectors are linearly independent). Let $\mathbf{y} \in \mathbb{R}^n$ be an external target vector to approximate. Let $\mathbf{z} \in \mathbb{Z}^c$ be the unknown integer combination of basis vectors. The goal is to find the lattice point that is closest in Euclidean distance to the target:

$\text{argmin}_{\mathbf{z} \in \mathbb{Z}^c} \|\mathbf{B} \mathbf{z} - \mathbf{y}\|_2$

The lattice is the set of all integer linear combinations of the basis vectors: $\mathcal{L}(\mathbf{B}) = \{\mathbf{B} \mathbf{z} : \mathbf{z} \in \mathbb{Z}^c\}$. The CVP is NP-hard to approximate within any constant factor (van Emde Boas, 1981; Micciancio & Goldwasser, 2002; Dinur et al., 2003).

Babai's nearest plane algorithm (Algorithm 2). Babai's algorithm is a polynomial-time heuristic that solves CVP approximately by processing the basis vectors in reverse order (from the last to the first dimension), projecting the target vector onto the nearest hyperplane orthogonal to each Gram-Schmidt vector, and rounding the corresponding coefficient. The paper's pseudocode uses a conceptually equivalent but computationally simpler approach: instead of projecting the target onto hyperplanes, it moves the target along the basis direction toward the hyperplane through the origin, keeping the projection error in the updated target vector (which does not affect subsequent projections because the error is orthogonal to the remaining hyperplanes).

Algorithm 2 walkthrough:

Line 2: LLL basis reduction. $\mathbf{T} \leftarrow \text{LLL}(\mathbf{B})$ applies the Lenstra-Lenstra-Lovász (LLL) algorithm to find a unimodular transformation matrix $\mathbf{T}$ such that the reduced basis $\mathbf{A} = \mathbf{B} \mathbf{T}$ has shorter and more orthogonal vectors. LLL reduction improves the approximation ratio of Babai's algorithm from exponential to $2^{O(n)}$. The parameter delta defaults to $3/4$. This step is optional — Babai's algorithm without LLL reduction (with $\mathbf{T} = \mathbf{I}$) is still well-defined and corresponds to the version that the paper will connect to GPTQ.

Line 3: Apply the transformation. $\mathbf{A} \leftarrow \mathbf{B} \mathbf{T}$ — the reduced basis.

Line 4: QR decomposition. $\boldsymbol{\Phi} \leftarrow \text{QR}(\mathbf{A})$ computes the orthogonal matrix from the QR decomposition of $\mathbf{A}$. This is the same as the normalized Gram-Schmidt orthogonalization of the basis vectors.

Lines 5–12: Main iteration (back-to-front). Initialize the residual target $\mathbf{y}' \leftarrow \mathbf{y}$ and the integer solution $\mathbf{z} \leftarrow \mathbf{0}$. For $j$ from $c$ down to $1$ (reverse order):

• Line 9: Compute the coefficient for the $j$-th basis vector: $\zeta \leftarrow \langle \boldsymbol{\Phi}[:, j], \mathbf{y}' \rangle / \langle \boldsymbol{\Phi}[:, j], \mathbf{A}[:, j] \rangle$ — the dot product of the $j$-th Gram-Schmidt vector (which is orthogonal to all later basis vectors) with the current residual target, divided by the dot product of the Gram-Schmidt vector with the original basis vector (which gives the length of the projection of $\mathbf{A}[:, j]$ onto the Gram-Schmidt direction). This ratio is the exact real-valued coefficient that would place the lattice point exactly on the target if all later coordinates were already fixed.

• Line 10: Round the coefficient: $\mathbf{z}[j] \leftarrow \text{Round}(\zeta, \mathbb{Z})$ — round to the nearest integer.

• Line 11: Update the residual: $\mathbf{y}' \leftarrow \mathbf{y}' - \mathbf{A}[:, j] \mathbf{z}[j]$ — subtract the contribution of the $j$-th basis vector with its rounded coefficient from the target. This moves the residual along the basis direction; the new residual is orthogonal to the $j$-th Gram-Schmidt vector (and all later ones, by construction of Gram-Schmidt), so subsequent iterations are unaffected by the rounding error at step $j$.

After the loop, the solution is transformed back: $\mathbf{z} \leftarrow \mathbf{T} \mathbf{z}$ to account for the LLL basis reduction.

Babai's error bound (Section 3.2). Let $\boldsymbol{\Phi} = [\boldsymbol{\phi}_1, \ldots, \boldsymbol{\phi}_c]$ be the normalized Gram-Schmidt vectors of the reduced basis $\mathbf{A} = [\mathbf{a}_1, \ldots, \mathbf{a}_c]$, and let $\tilde{\mathbf{A}} = [\tilde{\mathbf{a}}_1, \ldots, \tilde{\mathbf{a}}_c]$ be the unnormalized Gram-Schmidt vectors, where $\tilde{\mathbf{a}}_j = \langle \boldsymbol{\phi}_j, \mathbf{a}_j \rangle \boldsymbol{\phi}_j$ — the projection of the $j$-th basis vector onto the Gram-Schmidt direction. At iteration $j$, the algorithm replaces the exact real coefficient $\zeta$ by the nearest integer, so the rounding error satisfies $|\zeta - \mathbf{z}[j]| \leq 1/2$. The error component along $\tilde{\mathbf{a}}_j$ therefore has norm at most $\frac{1}{2} \|\tilde{\mathbf{a}}_j\|$. Because the Gram-Schmidt vectors $\tilde{\mathbf{A}}$ are mutually orthogonal (by construction), these error components add in Euclidean norm, giving the absolute error bound on the residual vector $\mathbf{y}'$:

$\|\mathbf{y}'\|_2 \leq \frac{1}{4} \sum_{j=1}^c \|\tilde{\mathbf{a}}_j\|^2 = \frac{1}{4} \sum_{j=1}^c \langle \boldsymbol{\phi}_j, \mathbf{a}_j \rangle^2$

The paper also notes the tightness of this bound: Babai's algorithm guarantees to return the center vector of the axis-aligned hyper-cuboid constructed by the unnormalized Gram-Schmidt vectors $\tilde{\mathbf{A}}$ that contains the target $\mathbf{y}$. Equality is attained when the target lies exactly at a corner of this hyper-cuboid — i.e., when every rounding error $|\zeta - \mathbf{z}[j]|$ is exactly $1/2$, which happens when each $\zeta$ is exactly halfway between two integers. So the bound is tight in the worst case.

Relative error bound. Babai (1986) also proved a relative error bound: there exists $\gamma$ such that $\|\mathbf{B} \mathbf{z} - \mathbf{y}\| \leq \gamma \cdot \min_{\mathbf{z}' \in \mathbb{Z}^c} \|\mathbf{B} \mathbf{z}' - \mathbf{y}\|$, with $1 \leq \gamma \leq 1 + \max_{1 \leq j \leq c} \frac{\sum_{j'=1}^j \|\tilde{\mathbf{a}}_{j'}\|^2}{\|\tilde{\mathbf{a}}_j\|^2} \leq \sqrt{c+1} \cdot \max_{1 \leq j' \leq j \leq c} \frac{\|\tilde{\mathbf{a}}_{j'}\|}{\|\tilde{\mathbf{a}}_j\|}$. The relative bound depends on how quickly the Gram-Schmidt vector norms decay — if the basis is nearly orthogonal (all $\|\tilde{\mathbf{a}}_j\|$ are similar), the approximation ratio is close to 1; if the basis has very short vectors (small $\|\tilde{\mathbf{a}}_j\|$ for some $j$), the ratio can be large.

How Babai's algorithm avoids RTN's worst-case behavior. Figure 1 (lower row) illustrates the rounding boundaries. Round-to-nearest (RTN) — which simply rounds each weight independently without error propagation — has parallelogram-shaped cells (the Voronoi cells of the lattice are parallelograms) that can be very elongated, causing large worst-case errors when the target falls near a far corner. Babai's algorithm, by orthogonally projecting onto hyperplanes, produces axis-aligned rectangular partitions (Figure 1g-h) whose maximum diameter is smaller than the maximum diameter of the corresponding Voronoi cells, tightening the worst-case bound. This geometric advantage is what the paper's error bound quantifies.

---

Theorem 1: Quantization as CVP (Section 4.1)

The first step in the chain of equivalences is recognizing that the per-channel quantization problem is exactly a CVP on a lattice defined by the input activations and the per-channel scales.

Statement (informal). The quantization problem $\text{argmin}_{\mathbf{z}_i \in \mathbb{Z}^{\dagger c}} \|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2$ and the CVP $\text{argmin}_{\mathbf{z} \in \mathbb{Z}^c} \|\mathbf{B} \mathbf{z} - \mathbf{y}\|_2$ share the same solution whenever the structural conditions $\mathbf{B} = \mathbf{X} \text{diag}(\mathbf{s}_i)$ (the lattice basis is the activation matrix with columns scaled by the per-element scales) and $\mathbf{y} = \mathbf{X} \mathbf{w}_i$ (the target vector is the original output activation for that channel) hold, and the solution domain matches. To match domains, we can either disable clipping in quantization ($\mathbb{Z}^{\dagger} = \mathbb{Z}$) or enable clipping in the CVP setup ($\mathbf{z} \in \mathbb{Z}^{\dagger c}$).

What this means operationally: Rather than thinking about quantizing weights and minimizing the output error in weight space, we can think about finding a lattice point (an integer combination of scaled activation vectors) that is closest to the target output vector (the original layer output for that channel). The weight vector $\mathbf{w}_i$ becomes the "coordinates" of the target in the original (unscaled) basis $\mathbf{X}$, and the integer vector $\mathbf{z}_i$ becomes the coordinates in the scaled basis $\mathbf{X} \text{diag}(\mathbf{s}_i)$. The quantization error $\|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2$ is exactly the Euclidean distance between the lattice point $\mathbf{B} \mathbf{z}_i$ and the target $\mathbf{y}$.

Why this equivalence is not obvious: In the quantization formulation, we minimize over integer weight vectors with the activations fixed — it's a discrete optimization in the weight space. In the CVP formulation, we minimize over integer lattice coordinates with the basis fixed — it's a discrete optimization in the activation space. The two are mathematically equivalent because the linear transformation $\mathbf{X}$ maps weight differences to output differences: $\mathbf{X} (\text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{w}_i) = \mathbf{B} \mathbf{z}_i - \mathbf{y}$. The norms are preserved if we take the square root of the Hessian, but as Theorem 1 shows, any factorization of the Hessian works.

Hessian factorization freedom (Theorem 1). Theorem 1 states that any factor $\boldsymbol{\mathcal{X}}$ of the Hessian matrix $\mathbf{X}^{\top} \mathbf{X} = \boldsymbol{\mathcal{X}}^{\top} \boldsymbol{\mathcal{X}}$ can replace $\mathbf{X}$ in the CVP formulation without changing the geometric properties of the problem, because two such factors differ only by an orthogonal transformation (rotation and/or reflection). Formally, if $\boldsymbol{\mathcal{X}}$ and $\boldsymbol{\mathcal{X}}'$ are two possible factors, then $\langle \boldsymbol{\chi}_{j_1}, \boldsymbol{\chi}_{j_2} \rangle = \langle \boldsymbol{\chi}'_{j_1}, \boldsymbol{\chi}'_{j_2} \rangle$ for all $1 \leq j_1, j_2 \leq c$ — the inner products between column vectors are identical — which implies the vector lengths $\|\boldsymbol{\chi}_{j_1}\| = \|\boldsymbol{\chi}'_{j_1}\|$ and pairwise angles $\angle(\boldsymbol{\chi}_{j_1}, \boldsymbol{\chi}_{j_2}) = \angle(\boldsymbol{\chi}'_{j_1}, \boldsymbol{\chi}'_{j_2})$ are preserved. Since Euclidean distances between lattice points and the target depend only on these inner products (by expanding $\|\mathbf{B} \mathbf{z} - \mathbf{y}\|^2 = \mathbf{z}^{\top} \mathbf{B}^{\top} \mathbf{B} \mathbf{z} - 2 \mathbf{z}^{\top} \mathbf{B}^{\top} \mathbf{y} + \|\mathbf{y}\|^2$), the CVP solution is invariant to the choice of Hessian factor.

Why this matters computationally: The original activation matrix $\mathbf{X} \in \mathbb{R}^{n \times c}$ has $n$ rows (batch size), which can be large (e.g., 256 sequences × 2048 tokens = thousands of rows), making the basis vectors high-dimensional. By Theorem 1, we can instead use a square factor $\boldsymbol{\mathcal{X}} \in \mathbb{R}^{c \times c}$ (e.g., the Cholesky factor or the transpose of the Cholesky factor of the Hessian), reducing the basis vector dimension from $n$ to $c$ without changing any distances or angles in the lattice. This is computationally essential for making Babai's algorithm practical at LLM scale, because operations like Gram-Schmidt orthogonalization and projection scale with the basis vector dimension.

The Quantization-CVP dictionary (Table 1, Appendix A.1). The paper provides a complete correspondence table mapping quantization concepts to CVP concepts for output channel $i$:

• Input activation $\mathbf{X} \in \mathbb{R}^{n \times c}$ → Basis directions (columns are generators)
• Scale $\mathbf{s}_i \in \mathbb{R}_{\neq 0}^c$ → Basis stretches
• $\mathbf{B}^{(i)} = \mathbf{X} \text{diag}(\mathbf{s}_i) \in \mathbb{R}^{n \times c}$ → Lattice basis (columns are generators)
• Weight $\mathbf{w}_i \in \mathbb{R}^c$ → Floating-point coordinates on the unstretched basis
• Integer weight representation $\mathbf{z}_i \in \mathbb{Z}^{\dagger c}$ → Integer coordinates on the lattice basis
• Dequantized weight $\mathbf{q}_i = \text{diag}(\mathbf{s}_i) \mathbf{z}_i \in \mathbb{R}^c$ → Dequantized coordinates on the unstretched basis
• Target output activation $\mathbf{y}^{(i)} = \mathbf{X} \mathbf{w}_i \in \mathbb{R}^n$ → External target vector to approximate

This dictionary is what makes the rest of the paper possible — it provides the precise translation between the two domains that allows every statement about Babai's algorithm to be restated as a statement about GPTQ, and vice versa.

---

Theorem 2 and Corollary 3: OBQ's Geometric Interpretation (Section 4.2)

Before proving that GPTQ is Babai's algorithm, the paper first proves that OBQ's error propagation step is exactly Babai's projection step. This is a crucial intermediate result because GPTQ differs from OBQ only in the quantization order — the per-step operations are identical — so once OBQ is understood geometrically, GPTQ's geometry follows by fixing the order.

Theorem 2 (Error Propagation = Babai's Projection). Babai's nearest plane algorithm iteratively projects the target vector onto the nearest hyperplane and rounds the coefficient. The OBQ error propagation step (Equation 2) is exactly this projection on the original basis $\mathbf{B} = \mathbf{X} \text{diag}(\mathbf{s}_i)$ without basis reduction.

What this means: Each time OBQ quantizes a weight and updates the remaining weights, the operation it performs is geometrically equivalent to: (1) identifying the hyperplane that is orthogonal to the Gram-Schmidt vector of the chosen basis direction and that passes through the nearest integer lattice point along that direction, (2) projecting the current target residual vector orthogonally onto that hyperplane, and (3) reading off the updated weight coordinates from the projection. The "error propagation" formula $\Delta \mathbf{w}_i[j'] = \frac{(\mathbf{H}^{-1})[j', j]}{(\mathbf{H}^{-1})[j, j]} (\mathbf{q}_i[j] - \mathbf{w}_i[j])$ is the algebraic expression for how the projection changes the floating-point coordinates on the remaining basis directions.

Proof sketch (geometric). The proof (referenced to Figure 2) works in the activation space with the following construction. Let the basis be $\mathbf{B} = [\mathbf{b}_1, \ldots, \mathbf{b}_c]$ with $\mathbf{b}_j$ being the columns. Let $\mathbf{J}$ be the set of unprojected (unquantized) indices. Let $\mathbf{y} = \sum_{j \in \mathbf{J}} \zeta_j \mathbf{b}_j$ be the current residual target, where $\zeta_j \in \mathbb{R}$ are the current weight values divided by scales. Let $\mathcal{NHP} := \lfloor \zeta_{j_2} \rceil \mathbf{b}_{j_2} + \text{Span}\{\mathbf{b}_j \mid j \neq j_2\}$ be the nearest hyperplane — this is the affine subspace formed by fixing the $j_2$-th coordinate to the nearest integer and allowing all other coordinates to vary. The projection error $\Delta \mathbf{y} = \text{Proj}_{\mathcal{NHP}}(\mathbf{y}) - \mathbf{y}$ decomposes as a linear combination of the basis vectors: $\Delta \mathbf{y} = \sum_{j \in \mathbf{J}} \Delta \zeta_j \mathbf{b}_j$, where $\Delta \zeta_j$ are the updates to the weights.

To compute $\Delta \zeta_{j_1}$ (the update to the weight for dimension $j_1$ caused by quantizing dimension $j_2$), the proof introduces the inverse basis $\mathbf{N} = \mathbf{B}^{-\top} = [\mathbf{n}_1, \ldots, \mathbf{n}_c]$ where $\langle \mathbf{n}_j, \mathbf{b}_j \rangle = 1$ and $\mathbf{n}_j \perp \mathbf{b}_{j'}$ for all $j \neq j'$. The inverse basis vectors are the dual basis — they extract the coordinate of a vector in the original basis. The geometry of the projection in the 2D subspace spanned by $\mathbf{n}_{j_1}$ and $\mathbf{n}_{j_2}$ (Figure 2d) yields the ratio $\Delta \zeta_{j_1} / \Delta \zeta_{j_2} = \langle \mathbf{n}_{j_1}, \mathbf{n}_{j_2} \rangle / \langle \mathbf{n}_{j_2}, \mathbf{n}_{j_2} \rangle$, which is exactly $(\mathbf{B}^{\top} \mathbf{B})^{-1}[j_1, j_2] / (\mathbf{B}^{\top} \mathbf{B})^{-1}[j_2, j_2]$. Substituting $\mathbf{B} = (\mathbf{X} \text{diag}(\mathbf{s}_i))[:, \mathbf{J}]$ and $\zeta_j = \mathbf{w}_i[j] / \mathbf{s}_i[j]$ recovers OBQ's error propagation formula.

Corollary 3 (OBQ Dimension Selection). At each step, OBQ selects the not-yet-quantized dimension $j$ such that the nearest hyperplane of dimension $j$ is the closest to the target residual vector. Formally, this means OBQ minimizes the projection distance $\|\Delta \mathbf{y}\|$, which equals $|\Delta \zeta_j| / \|\mathbf{n}_j\|$ — the absolute rounding error divided by the norm of the inverse basis vector. This is equivalent to the OBQ selection criterion $\text{argmin}_{j \in \mathbf{J}} \frac{(\mathbf{q}_i[j] - \mathbf{w}_i[j])^2}{(\mathbf{X}[:, \mathbf{J}]^{\top} \mathbf{X}[:, \mathbf{J}])^{-1}[j, j]}$ because $(\mathbf{X}[:, \mathbf{J}]^{\top} \mathbf{X}[:, \mathbf{J}])^{-1}[j, j] = \|\mathbf{n}_j\|^2$.

Geometric intuition: OBQ always projects onto the hyperplane that is currently closest to the residual target. This is a greedy nearest-neighbor strategy — at each step, pick the hyperplane that you would have to travel the shortest distance to reach, project onto it, and continue from there. Corollary 3 establishes that OBQ's algebraic selection formula has this clean geometric meaning.

What this implies for GPTQ: GPTQ replaces this dynamic nearest-hyperplane selection with a fixed order, effectively committing in advance to a sequence of hyperplanes to project onto, regardless of where the target residual happens to be at each step. The geometric price of this simplification is that GPTQ may project onto a hyperplane that is farther from the current residual than the nearest one, accumulating more error than OBQ would. However, Theorem 5 will show that this error remains bounded, and Section 4.5 will show how to choose the fixed order to minimize the bound.

---

Theorem 4: GPTQ = Babai's Algorithm Without Basis Reduction (Section 4.3)

This is the paper's central theorem, establishing that GPTQ (executed back-to-front) is mathematically identical to Babai's nearest plane algorithm without LLL basis reduction, applied to the lattice with basis $\mathbf{B} = \mathbf{X} \text{diag}(\mathbf{s}_i)$.

Statement. GPTQ and Babai's algorithm without basis reduction will have the same results if we align the dimensional order of these two algorithms — e.g., running GPTQ from the last to the first dimension (back-to-front) matches Babai's default back-to-front iteration.

What this means operationally: The only difference between GPTQ as implemented in Algorithm 1 and Babai's algorithm as implemented in Algorithm 2 (with $\mathbf{T} = \mathbf{I}$, no LLL reduction) is the iteration direction. GPTQ's default front-to-back order ($j \leftarrow 1$ to $c$) processes the first dimension first and propagates error forward to later dimensions. Babai's default back-to-front order ($j \leftarrow c$ to $1$) processes the last dimension first and propagates error backward to earlier dimensions. If we simply reverse GPTQ's iteration direction — or equivalently, apply a permutation that reverses the column order and run GPTQ as usual — the two algorithms become identical, producing the exact same quantized integers $\mathbf{Z}$ and dequantized weights $\mathbf{Q}$.

Why this is non-obvious: The two algorithms operate in entirely different spaces. GPTQ works in weight space: it updates the floating-point weights $\mathbf{W}$ row-by-row, adding compensation to subsequent rows. Babai's algorithm works in activation space: it maintains a residual target vector $\mathbf{y}'$, subtracting basis vectors multiplied by rounded coefficients. The weight updates in GPTQ (line 13: $\mathbf{W}[j:, :] \leftarrow \mathbf{W}[j:, :] + \mathbf{L}[j:, j] \boldsymbol{\varepsilon}$) and the target updates in Babai (line 11: $\mathbf{y}' \leftarrow \mathbf{y}' - \mathbf{A}[:, j] \mathbf{z}[j]$) look completely different algebraically. The proof must establish that these updates are dual to each other — one in weight space, one in activation space — and that the rounding decisions (which determine the quantized integers) are identical.

Dual proof structure. The paper provides both a geometric proof sketch and a rigorous algebraic proof (Appendix B).

Geometric proof sketch: Theorem 2 already shows that each OBQ/GPTQ error propagation step is Babai's projection onto the nearest hyperplane orthogonal to the $j$-th Gram-Schmidt vector. Since GPTQ and OBQ perform the same per-step operations (only the order differs), each GPTQ step is also Babai's projection. When GPTQ runs back-to-front from $j = c$ down to $1$, the sequence of projections exactly matches Babai's iteration order: at step $j$, the algorithm projects the current residual onto the hyperplane orthogonal to the $j$-th Gram-Schmidt vector, rounds the $j$-th coordinate, and continues. The initial target vector in the activation space is $\mathbf{y} = \mathbf{X} \mathbf{w}_i$, and the residual after each projection corresponds to the "unquantized portion" of the weight vector mapped through the activation matrix — which is precisely what GPTQ's weight updates track.

Algebraic proof (Appendix B, three steps). The formal proof constructs three intermediate algorithms and proves their pairwise equivalence:

Step 1 (Algorithm 5 → Algorithm 6): Rewrite GPTQ using absolute error. The original GPTQ (Algorithm 5) tracks the relative quantization error $\boldsymbol{\varepsilon}^{(j)} = \mathbf{Q}[:, j] - \mathbf{W}[:, j]$ at each step and updates weights as $\mathbf{W}^{(j)} = \mathbf{W}^{(j-1)} + \mathbf{L} \mathbf{e}_j \boldsymbol{\varepsilon}^{(j)}$. Algorithm 6 instead tracks the absolute quantization error matrix $\boldsymbol{\Delta}^{(j)} = \mathbf{Q}^{(j)} - \mathbf{W}^{(0)}$ (the total deviation of all weights from their original values) and updates weights inversely: $\mathbf{W}^{(j)} = \mathbf{W}^{(0)} - \mathbf{L}^{-1} \boldsymbol{\Delta}^{(j)}$. The proof shows by induction (Appendix B.1) that $\boldsymbol{\omega}^{(j)} = \hat{\boldsymbol{\omega}}^{(j)}$ for all $j$ — the pre-rounding weight values are identical in both formulations — which implies identical rounding decisions and identical final $\mathbf{Z}$ and $\mathbf{Q}$. The key equality used in the induction is that $\boldsymbol{\varepsilon}^{(k)} = \mathbf{e}_k^{\top} \mathbf{L}^{-1} \boldsymbol{\Delta}^{(j^*)}$ for all $k \leq j^*$ — the per-step relative errors aggregate through the inverse of the LDL factor into the cumulative absolute error.

Step 2 (Algorithm 6 → Algorithm 7): Reverse the iteration. Algorithm 6 processes front-to-back ($j \leftarrow 1$ to $c$). Algorithm 7 reverses this to back-to-front ($j \leftarrow c$ down to $1$). This requires changing the LDL decomposition to a "UDU" decomposition: $\mathbf{H}^{-1} = \mathbf{U} \mathbf{D}_U^{1/2} \mathbf{D}_U^{1/2} \mathbf{U}^{\top}$ where $\mathbf{U}$ is upper triangular with ones on the diagonal. The UDU decomposition is obtained by anti-diagonally permuting the LDL decomposition of the anti-diagonally permuted Hessian inverse: $\mathbf{U} = \mathbf{P} \hat{\mathbf{L}} \mathbf{P}$ where $\mathbf{P}$ is the anti-diagonal permutation matrix. The weight update becomes $\mathbf{W}^{(j)} = \mathbf{W}^{(c+1)} - \mathbf{U}^{-1} \boldsymbol{\Delta}^{(j)}$, which propagates error to the not-yet-quantized coordinates that now lie before dimension $j$ (indices $1$ through $j-1$), matching the semantics of a back-to-front sweep.

Step 3 (Algorithm 7 → Algorithm 8): Map to Babai's variables. Algorithm 7 operates in weight space with the UDU decomposition. Algorithm 8 operates in activation space with the Cholesky decomposition. The connection is through the Cholesky factor: $\mathbf{H} = (\mathbf{H}^{-1})^{-1} = (\mathbf{U} \mathbf{D}_U^{1/2} \mathbf{D}_U^{1/2} \mathbf{U}^{\top})^{-1} = (\mathbf{D}_U^{-1/2} \mathbf{U}^{-1})^{\top} \mathbf{D}_U^{-1/2} \mathbf{U}^{-1}$, so the Cholesky factor (upper triangular) is $\mathbf{A} = \mathbf{D}_U^{-1/2} \mathbf{U}^{-1}$. The initial target vector in activation space is $\mathbf{Y}^{(c+1)} = \mathbf{A} \mathbf{W}$, which maps the original weights through the Cholesky factor. The proof shows by induction (Appendix B.3) that the pre-rounding values $\boldsymbol{\omega}^{(j)}$ are identical in both algorithms: $\tilde{\boldsymbol{\omega}}^{(j)} = \mathbf{e}_j^{\top} \mathbf{D}_U^{1/2} \mathbf{Y}^{(j+1)} = \mathbf{e}_j^{\top} (\tilde{\mathbf{W}} - \mathbf{U}^{-1} (\tilde{\mathbf{Q}}^{(j+1)} - \tilde{\mathbf{W}})) = \hat{\boldsymbol{\omega}}^{(j)}$. This means the rounding decisions are identical, and since the dequantization step (multiplying rounded integers by scales) is the same in both algorithms, the final $\mathbf{Z}$ and $\mathbf{Q}$ are identical.

Ineffectiveness of composing Babai with GPTQ (Appendix B.4). A natural question is whether running Babai's algorithm and then applying an additional GPTQ-style error propagation step could further improve the solution. The paper proves this is impossible: any such extra update vanishes. Algebraically, if we modify Babai's update to include a GPTQ correction $\mathbf{Y}'(j) \leftarrow \mathbf{Y}(j) + \mathbf{A} \mathbf{U} \mathbf{e}_j \boldsymbol{\varepsilon}^{(j)}$, the next step's pre-rounding value $\tilde{\boldsymbol{\omega}}'(j-1)$ is unchanged: $\tilde{\boldsymbol{\omega}}'(j-1) = \mathbf{e}_{j-1}^{\top} \mathbf{D}_U^{1/2} \mathbf{Y}'(j) = \mathbf{e}_{j-1}^{\top} \mathbf{D}_U^{1/2} \mathbf{Y}(j) + \mathbf{e}_{j-1}^{\top} \mathbf{D}_U^{1/2} \mathbf{A} \mathbf{U} \mathbf{e}_j \boldsymbol{\varepsilon}^{(j)} = \tilde{\boldsymbol{\omega}}(j-1) + \mathbf{e}_{j-1}^{\top} \mathbf{e}_j \boldsymbol{\varepsilon}^{(j)} = \tilde{\boldsymbol{\omega}}(j-1)$ because $\mathbf{e}_{j-1}^{\top} \mathbf{e}_j = 0$. The GPTQ correction is orthogonal to the remaining projection directions and therefore has no effect on subsequent rounding decisions. The equivalence in Theorem 4 is already tight.

Geometric interpretation of GPTQ (stated in Section 4.3). Theorem 4 provides an intuitive geometric picture: "GPTQ performs an orthogonal walk through a nested sequence of affine subspaces in a pre-computed dimensional order." Each step moves orthogonally to the nearest affine subspace (hyperplane) that fixes one more coordinate, with the error from each step being orthogonal to all subsequent projection directions. This is why the error components add in Euclidean norm (Pythagorean theorem) rather than interfering constructively or destructively — a property that is not obvious from the algebraic formulation but is immediate from the geometric picture.

---

Algorithm 4: Batched Babai Quantization (Appendix A.2)

To make the equivalence practically useful (rather than just theoretically interesting), the paper constructs an efficient batched version of Babai's algorithm for quantization, incorporating several computational optimizations that preserve the mathematical equivalence while making the algorithm suitable for LLM-scale linear layers.

Motivation. A naive application of Babai's algorithm to quantization would require: (1) forming the basis $\mathbf{B}^{(i)} = \boldsymbol{\mathcal{X}} \text{diag}(\mathbf{s}_i)$ and target $\mathbf{y}^{(i)} = \boldsymbol{\mathcal{X}} \mathbf{w}_i$ for each output channel $i$, (2) running the LLL basis reduction to get a transformation $\mathbf{T}^{(i)}$ (which depends on the scale vector $\mathbf{s}_i$ and may differ per channel), (3) computing the QR decomposition of $\mathbf{A}^{(i)} = \mathbf{B}^{(i)} \mathbf{T}^{(i)}$ (also per-channel), and (4) running the back-to-front iteration. Steps 2–3 have complexity $O(c^4)$ for LLL and $O(c^3)$ for QR, and doing this $r$ times (once per output channel) would be $O(r c^4)$ — completely infeasible for LLMs where $c$ is in the thousands and $r$ is also in the thousands.

Optimization 1: Disable basis reduction. The LLL basis reduction is "unfortunately scale-sensitive, generating different transformations $\mathbf{T}^{(i)}$ for different scales $\mathbf{s}_i$ (unless all the $\mathbf{s}_i$ vectors are parallel)." This means LLL cannot be batched across output channels — each channel would need its own reduction. Furthermore, LLL is incompatible with clipping because rounding is performed in the transformed basis, and there's no easy way to enforce the original quantization grid constraints. The paper therefore sets $\mathbf{T} = \mathbf{I}$ (no reduction) for the batched algorithm, matching GPTQ which also uses no basis reduction.

Optimization 2: Use a shared permutation matrix for quantization order. As shown in Algorithm 2, Babai's iteration order is determined by the order of the basis vectors. To implement different quantization orders (e.g., act-order, min-pivot), the paper replaces the LLL transformation with a permutation matrix $\mathbf{T}$, which reorders the basis vectors. Theorem 6 proves that if $\mathbf{T}$ is a permutation matrix independent of the output channel $i$, the orthogonal matrix $\boldsymbol{\Phi}$ from the QR decomposition can be reused across all channels without recomputation. The key insight is that a diagonal scaling matrix $\text{diag}(\mathbf{s}_i)$ commutes with a permutation matrix up to a re-permutation of the scales: $\text{diag}(\mathbf{s}_i) \mathbf{T} = \mathbf{T} \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i)$. This means $\mathbf{A}^{(i)} = \boldsymbol{\mathcal{X}} \text{diag}(\mathbf{s}_i) \mathbf{T} = \boldsymbol{\mathcal{X}} \mathbf{T} \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i) = \mathbf{A} \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i)$ where $\mathbf{A} = \boldsymbol{\mathcal{X}} \mathbf{T}$ is shared across channels. The QR decomposition of the shared $\mathbf{A}$ is computed once, and the per-channel QR is obtained by right-multiplying the upper triangular factor by the permuted scales: $\mathbf{A}^{(i)} = \boldsymbol{\Phi} (\mathbf{R} \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i))$, which preserves the same orthogonal matrix $\boldsymbol{\Phi}$.

Optimization 3: Choose $\boldsymbol{\mathcal{X}}$ so that $\mathbf{A}$ is upper triangular. By Theorem 1, any Hessian factor $\boldsymbol{\mathcal{X}}$ works. The paper cleverly chooses $\boldsymbol{\mathcal{X}}$ such that $\mathbf{A} = \boldsymbol{\mathcal{X}} \mathbf{T}$ is already upper triangular — specifically, by taking $\boldsymbol{\mathcal{X}}$ to be the transpose of the Cholesky factor of the permuted Hessian. The Cholesky decomposition of the permuted Hessian $\mathbf{T}^{\top} \mathbf{X}^{\top} \mathbf{X} \mathbf{T}$ gives an upper triangular matrix $\mathbf{A}$ directly (the Cholesky factor is lower triangular, so its transpose is upper triangular). When $\mathbf{A}$ is upper triangular, its QR decomposition is trivial: $\mathbf{A} = \mathbf{I} \cdot \mathbf{A}$ with $\boldsymbol{\Phi} = \mathbf{I}$ and $\mathbf{R} = \mathbf{A}$. This eliminates the QR computation entirely and makes the projection step extremely simple: the $j$-th Gram-Schmidt vector is just the $j$-th column of $\mathbf{A}$, and its projection onto the target reduces to extracting the $j$-th row of the accumulated activation-space representation.

Algorithm 4 walkthrough (the final batched algorithm):

Inputs and preprocessing (lines 3–5): Same as GPTQ — compute the damped Hessian, Cholesky decomposition, and permute weights and scales.

Line 3: $\mathbf{H} \leftarrow \mathbf{T}^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}) \mathbf{T}$ — permuted damped Hessian.

Line 4: $\mathbf{A} \leftarrow \text{Cholesky}(\mathbf{H})^{\top}$ — the transpose of the Cholesky factor, which is an upper triangular matrix. This $\mathbf{A}$ is the lattice basis in the activation space, with the property that $\mathbf{A}^{\top} \mathbf{A} = \mathbf{H}$.

Line 5: $\mathbf{W}, \mathbf{S} \leftarrow \mathbf{T}^{-1} \mathbf{W}, \mathbf{T}^{-1} \mathbf{S}$ — permute weights and scales.

Line 6: Initialize $\mathbf{Y} \leftarrow \mathbf{A} \mathbf{W}$ — this maps the original weights into the activation space. $\mathbf{Y}$ has dimensions $c \times r$, where each column is the target vector for one output channel. $\mathbf{Q} \leftarrow \mathbf{W}$ and $\mathbf{Z} \leftarrow \mathbf{0}$ as in GPTQ.

Main loop (lines 7–14, back-to-front $j \leftarrow c$ to $1$):

• Line 9: Extract the $j$-th row of the residual and divide by the diagonal entry of $\mathbf{A}$: $\boldsymbol{\omega} \leftarrow \mathbf{Y}[j, :] / \mathbf{A}[j, j]$. Because $\mathbf{A}$ is upper triangular, the $j$-th Gram-Schmidt vector is $\mathbf{A}[:, j]$, its length is $\mathbf{A}[j, j]$ (the only non-zero entry in the last $c-j+1$ rows is the diagonal), and the projection coefficient for the $j$-th basis vector is exactly the $j$-th coordinate of $\mathbf{Y}$ divided by the diagonal.

• Line 10: Convert to integer space: $\boldsymbol{\zeta} \leftarrow \boldsymbol{\omega} / \mathbf{S}[j, :]$ — element-wise division by the $j$-th row of scales.

• Line 11: Quantize: $\mathbf{Z}[j, :] \leftarrow \text{Round}(\boldsymbol{\zeta}, \mathbb{Z}^{\dagger})$.

• Line 12: Dequantize: $\mathbf{Q}[j, :] \leftarrow \mathbf{Z}[j, :] * \mathbf{S}[j, :]$.

• Line 13: Update residual: $\mathbf{Y} \leftarrow \mathbf{Y} - \mathbf{A}[:, j] \mathbf{Q}[j, :]$ — subtract the $j$-th basis vector multiplied by the dequantized weights (a row vector of length $r$). This is a rank-1 update: the column vector $\mathbf{A}[:, j]$ (length $c$) outer-products with the row vector $\mathbf{Q}[j, :]$ (length $r$) and is subtracted from $\mathbf{Y}$.

After the loop: $\mathbf{Z}, \mathbf{Q} \leftarrow \mathbf{T} \mathbf{Z}, \mathbf{T} \mathbf{Q}$ — permute back to original order.

Why Algorithm 4 is equivalent to Algorithm 1 (GPTQ) when both run back-to-front: The key algebraic identity is that $\mathbf{A} = \mathbf{D}_U^{-1/2} \mathbf{U}^{-1}$ (from the UDU decomposition), so $\mathbf{Y} = \mathbf{A} \mathbf{W}$ and $\mathbf{A}[:, j] = \mathbf{D}_U^{-1/2} \mathbf{U}^{-1} \mathbf{e}_j$. The residual update $\mathbf{Y} \leftarrow \mathbf{Y} - \mathbf{A}[:, j] \mathbf{Q}[j, :]$ in the activation space corresponds exactly to the weight update $\mathbf{W}^{(j)} = \mathbf{W}^{(c+1)} - \mathbf{U}^{-1} \boldsymbol{\Delta}^{(j)}$ in the weight space, because subtracting $\mathbf{A}[:, j] \mathbf{Q}[j, :]$ from $\mathbf{Y}$ and then left-multiplying by $\mathbf{A}^{-1} = \mathbf{U} \mathbf{D}_U^{1/2}$ recovers the weight update. The extraction of $\boldsymbol{\omega} = \mathbf{Y}[j, :] / \mathbf{A}[j, j]$ corresponds to reading the $j$-th row of $\mathbf{W}$ after error propagation, because $\mathbf{Y}[j, :] = \mathbf{e}_j^{\top} \mathbf{A} \mathbf{W} = \mathbf{A}[j, j] \mathbf{e}_j^{\top} \mathbf{W}$ (since $\mathbf{A}$ is upper triangular, $\mathbf{e}_j^{\top} \mathbf{A}$ has only one non-zero entry, at position $j$, equal to $\mathbf{A}[j, j]$). This elegant simplification is the payoff for choosing $\boldsymbol{\mathcal{X}}$ to make $\mathbf{A}$ upper triangular.

---

Theorem 5: GPTQ's Error Bound (Section 4.4)

Once the equivalence to Babai's algorithm is established, Babai's error bound transfers directly to GPTQ in the no-clipping setting.

Statement (Theorem 5). Assume no clipping ($\mathbb{Z}^{\dagger} = \mathbb{Z}$). Let $\mathbf{T}$ be the permutation matrix of the reversed GPTQ quantization order (equivalently $\mathbf{P}$ with reversed column order; the reversal is needed because GPTQ front-to-back corresponds to Babai back-to-front, but the bound is stated in terms of the Babai order). Let $\mathbf{D}$ be the diagonal matrix of the LDL decomposition of the permuted Hessian matrix $\mathbf{T}^{\top} \mathbf{X}^{\top} \mathbf{X} \mathbf{T}$. For every output channel $i$ ($1 \leq i \leq r$), the absolute quantization error has the tight upper bound:

$\|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2 \leq \frac{1}{4} (\mathbf{T}^{-1} \mathbf{s}_i)^{\top} \mathbf{D} (\mathbf{T}^{-1} \mathbf{s}_i)$

where $\mathbf{T}^{-1} \mathbf{s}_i$ is the scale vector permuted to match the quantization order, $\mathbf{D}$ is the diagonal matrix from the LDL decomposition (whose entries are the squared lengths of the unnormalized Gram-Schmidt vectors of the permuted basis), and the quadratic form $(\mathbf{T}^{-1} \mathbf{s}_i)^{\top} \mathbf{D} (\mathbf{T}^{-1} \mathbf{s}_i) = \sum_{j=1}^c \mathbf{D}[j, j] \cdot ((\mathbf{T}^{-1} \mathbf{s}_i)[j])^2$ weights each diagonal entry of $\mathbf{D}$ by the square of the corresponding scale factor.

What this computes: The worst-case squared L2 error of the quantized layer's output, for a single output channel, is at most one quarter of the sum over dimensions $j$ of the product $\mathbf{D}[j, j] \cdot ((\mathbf{T}^{-1} \mathbf{s}_i)[j])^2$. The diagonal entry $\mathbf{D}[j, j]$ is the squared length of the $j$-th unnormalized Gram-Schmidt vector of the lattice basis, which measures how much the $j$-th basis vector contributes to the lattice geometry after orthogonalization. The scale factor $(\mathbf{T}^{-1} \mathbf{s}_i)[j]$ stretches or shrinks this contribution — larger scales amplify the error because a rounding error of $\pm 1/2$ in the integer $\mathbf{z}_i[j]$ translates to an error of $\pm \mathbf{s}_i[j]/2$ in the dequantized weight.

Proof sketch (Appendix C.1). The proof proceeds in three steps:

1. Express the quantization error as a CVP error: $\|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|_2 = \|\mathbf{B}^{(i)} \mathbf{z}_i - \mathbf{y}^{(i)}\|_2$ where $\mathbf{B}^{(i)} = \mathbf{X} \text{diag}(\mathbf{s}_i)$ and $\mathbf{y}^{(i)} = \mathbf{X} \mathbf{w}_i$.

2. Apply the permutation $\mathbf{T}$ to get $\|\mathbf{A}^{(i)} (\mathbf{T}^{-1} \mathbf{z}_i) - \mathbf{y}^{(i)}\|_2$ where $\mathbf{A}^{(i)} = \mathbf{B}^{(i)} \mathbf{T}$ is the permuted basis.

3. Apply Babai's bound: $\|\mathbf{A}^{(i)} (\mathbf{T}^{-1} \mathbf{z}_i) - \mathbf{y}^{(i)}\|_2 \leq \frac{1}{4} \sum_{j=1}^c \|\tilde{\mathbf{a}}^{(i)}_j\|^2$ where $\tilde{\mathbf{a}}^{(i)}_j$ are the unnormalized Gram-Schmidt vectors of $\mathbf{A}^{(i)}$.

The Gram-Schmidt vector lengths can be expressed in terms of the LDL decomposition. The LDL decomposition of $\mathbf{A}^{(i)\top} \mathbf{A}^{(i)} = \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i) \mathbf{T}^{\top} \mathbf{X}^{\top} \mathbf{X} \mathbf{T} \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i)$ has diagonal matrix $\mathbf{D}^{(i)} = \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i) \mathbf{D} \text{diag}(\mathbf{T}^{-1} \mathbf{s}_i)$ (where $\mathbf{D}$ is from the LDL decomposition of $\mathbf{T}^{\top} \mathbf{X}^{\top} \mathbf{X} \mathbf{T}$), and the diagonal entries satisfy $\mathbf{D}^{(i)}[j, j] = \|\tilde{\mathbf{a}}^{(i)}_j\|^2 = \mathbf{D}[j, j] \cdot ((\mathbf{T}^{-1} \mathbf{s}_i)[j])^2$. Summing over $j$ gives the bound.

Relative error bound. The paper also imports Babai's relative error bound: there exists $\gamma$ such that:

$\|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\| \leq \gamma \cdot \min_{\mathbf{z}_i' \in \mathbb{Z}^c} \|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i' - \mathbf{X} \mathbf{w}_i\|$

with $1 \leq \gamma \leq 1 + \max_{1 \leq j \leq c} \frac{\sum_{j'=1}^j d_{j'}^2}{d_j^2} \leq \sqrt{c+1} \cdot \max_{1 \leq j' \leq j \leq c} \frac{d_{j'}}{d_j}$, where $d_j = \sqrt{\mathbf{D}[j, j]} \cdot |(\mathbf{T}^{-1} \mathbf{s}_i)[j]|$ combines the LDL diagonal entry and the scale factor. The relative bound says that the GPTQ/Babai solution is within a factor $\gamma$ of the optimal integer solution, and $\gamma$ depends on how quickly the $d_j$ values decay — if all $d_j$ are similar, $\gamma \approx 1$ (near-optimal); if some $d_j$ are much smaller than others, $\gamma$ can be large.

Tightness of the absolute bound. The bound is attained with equality when the floating-point weight vector $\mathbf{w}_i$ is such that every rounding error is exactly $\pm 1/2$ — i.e., when the target lies at a corner of Babai's hyper-cuboid (all coordinates are halfway between two integers). This is a worst-case scenario that may be rare in practice.

Expected error under uniform prior (Appendix C.2). If the continuous weight offsets are uniformly distributed within Babai's hyper-cuboid (a reasonable assumption when scales are small enough that weights are roughly uniform within each quantization bin), the expected squared error is $\frac{1}{3}$ of the worst-case bound:

$\mathbb{E}\left[\|\mathbf{X} \text{diag}(\mathbf{s}_i) \mathbf{z}_i - \mathbf{X} \mathbf{w}_i\|^2\right] = \frac{1}{12} \sum_{j=1}^c \|\tilde{\mathbf{a}}^{(i)}_j\|^2$

This follows from the fact that for a scalar $u \sim \text{Uniform}(-a, a)$, the expected squared value is $\mathbb{E}[u^2] = a^2/3$ (Lemma 7), and the coordinates in the Gram-Schmidt basis are independent and uniform within their respective half-edge lengths $a_j = \frac{1}{2} \|\tilde{\mathbf{a}}^{(i)}_j\|$ under the uniform prior.

---

Quantization Order and the Min-Pivot Heuristic (Section 4.5, Algorithm 3)

The error bound $\frac{1}{4} (\mathbf{T}^{-1} \mathbf{s}_i)^{\top} \mathbf{D} (\mathbf{T}^{-1} \mathbf{s}_i)$ depends on the permutation $\mathbf{T}$ through the diagonal matrix $\mathbf{D}$ (and through which scale factors multiply which diagonal entries). Different quantization orders produce different LDL decompositions of the permuted Hessian, and therefore different error bounds.

The optimization problem. For a given set of scale vectors $\mathbf{s}_i$ (which vary per channel), finding the permutation $\mathbf{T}$ that minimizes the maximum (over channels) of the bound is a combinatorial optimization problem. To make the problem tractable, the paper observes that for batched quantization (where $\mathbf{T}$ must be shared across all channels), and for large quantization group sizes (where scale factors within a group are approximately equal), a reasonable approximation is to assume $\mathbf{s}_i[j]$ are approximately equal for all $1 \leq j \leq c$. Under this approximation, the bound is proportional to $\text{tr}(\mathbf{D}) = \sum_{j=1}^c \mathbf{D}[j, j]$ — the sum of the diagonal entries of the LDL decomposition of the permuted Hessian. The optimization reduces to: find the permutation (pivot order for LDL) that minimizes $\text{tr}(\mathbf{D})$.

NP-hardness. The paper notes that minimizing the trace (or equivalently, minimizing fill-in or maximizing sparsity) in the LDL or Cholesky decomposition is NP-hard in general (Rose et al., 1976). However, effective heuristics exist.

GPTQ's act-order heuristic. GPTQ's existing "act-order" heuristic sorts the dimensions by the descending order of the Hessian diagonal entries — i.e., the dimension with the largest $\mathbf{H}[j, j]$ is quantized first. In the context of Babai's algorithm (back-to-front), this corresponds to processing dimensions in the ascending order of the Hessian diagonal (smallest diagonal first). The act-order heuristic is computationally cheap (just sorting the diagonal) and is motivated by the intuition that dimensions with larger Hessian diagonals (higher input variance) are more "important" and should be quantized later when more error-compensation degrees of freedom remain.

Geometric interpretation of act-order. In the Gram-Schmidt orthogonalization of the basis vectors (columns of $\mathbf{X}$), processing vectors in ascending order of the Hessian diagonal corresponds to starting with the shortest basis vectors and orthogonalizing against them first. This tends to produce more balanced Gram-Schmidt vector lengths (the short vectors get orthogonalized against first, so the remaining longer vectors don't get shortened as much by the orthogonalization), which reduces the sum of squared Gram-Schmidt lengths (the trace of $\mathbf{D}$).

Min-pivot heuristic (Algorithm 3). The paper proposes an improved order, called "min-pivot," which directly greedily minimizes the diagonal entry at each LDL decomposition step. Algorithm 3 works as follows:

Input: The Hessian matrix $\mathbf{H}$ (already damped). Output: A permutation matrix $\mathbf{T}$.

Line 3: Initialize the set of unprocessed indices $\mathbf{J} \leftarrow \{1, \ldots, c\}$.

Line 4: Initialize the permutation matrix $\mathbf{T} \leftarrow \mathbf{0}$.

Main loop (lines 5–10, for $j \leftarrow 1$ to $c$):

• Line 6: Select $j' \leftarrow \text{argmin}_{j' \in \mathbf{J}} \mathbf{H}[j', j']$ — pick the unprocessed index with the smallest diagonal entry in the current (Schur-complemented) Hessian.

• Line 7: Update the Hessian: $\mathbf{H} \leftarrow \mathbf{H} - \mathbf{H}[:, j'] \mathbf{H}[j', :] / \mathbf{H}[j', j']$ — this is one step of LDL decomposition: subtract the rank-1 outer product of the $j'$-th column and row, scaled by the pivot. The remaining submatrix is the Schur complement, which represents the Hessian restricted to the unprocessed dimensions after eliminating the $j'$-th dimension.

• Line 8: Record the permutation: $\mathbf{T}[j', j] \leftarrow 1$ — the $j'$-th original dimension becomes the $j$-th dimension in the permuted order.

• Line 9: Remove $j'$ from $\mathbf{J}$.

Computational complexity. Each iteration does $O(c^2)$ work (the rank-1 update in line 7), and there are $c$ iterations, giving $O(c^3)$ total — the same as the LDL decomposition itself. Since GPTQ already requires an LDL decomposition, min-pivot does not increase the overall asymptotic complexity.

Geometric interpretation of min-pivot. In the Gram-Schmidt orthogonalization, min-pivot corresponds to always taking the shortest residual vector as the next one to orthogonalize against. This greedily minimizes the next Gram-Schmidt vector length (since the Gram-Schmidt length of the $j'$-th basis vector after eliminating previous dimensions is exactly $\sqrt{\mathbf{H}[j', j']}$ in the Schur complement), which tends to reduce the sum of squared lengths. This order "agrees with Babai's relative error bound" — the relative bound is smallest when the Gram-Schmidt vector lengths decrease slowly, which is promoted by processing short vectors first.

Empirical validation (Appendix C.3, Table 2). The paper compares $\text{tr}(\mathbf{D})$ for five quantization orders on the Qwen3-8B model, using the layers in transformer block 18 as a representative example:

• Back-to-front: The trace for the Q·K·V layer is $1.169 \times 10^8$.
• Front-to-back: $1.161 \times 10^8$ — very similar.
• Random order (averaged over 100 runs): $1.168 \times 10^8$ — random ordering does not help.
• Act-order: $7.400 \times 10^7$ — approximately 37% lower than back-to-front, a substantial improvement.
• Min-pivot: $7.323 \times 10^7$ — slightly better than act-order (about 1% lower).

Similar patterns hold across other layer types (O, Gate·Up, Down) and other blocks. The paper concludes that "act-order already reduces $\text{tr}(\mathbf{D})$ relative to the back-to-front/front-to-back/random baselines, especially in the Q·K·V and Gate·Up layers (≈35–50% lower). Our min-pivot heuristic consistently attains the smallest trace." However, the downstream accuracy gains from min-pivot over act-order are described as "modest" — consistent with the interpretation that act-order already captures most of the benefit when the Hessian is well-conditioned (i.e., when the diagonal entries are not wildly different, the descending diagonal order approximates the greedy min-pivot choice). The paper recommends using act-order as the cheap default and min-pivot "for cases where a tighter bound is required."

---

Practical No-Clipping Methods (Section 5, Appendix D.1)

The theoretical framework establishes that the error bound only holds when clipping is disabled ($\mathbb{Z}^{\dagger} = \mathbb{Z}$). Standard GPTQ clips overflowed integers to the quantization grid's representable range, which severs the CVP equivalence by replacing the lattice $\mathbb{Z}^c$ with a bounded subset. The paper's practical contribution is two methods that avoid clipping while still achieving competitive bitwidths by representing overflowed values through sparse outliers (SSQR) or variable-length coding (HPTQ).

Scale-Adjusted SpQR (SSQR, Algorithms 9–10). SSQR extends the Sparse-Quantized Representation (SpQR) framework (Dettmers et al., 2024) by ensuring that no weight clipping occurs during GPTQ's error propagation. SpQR stores a small fraction of weights in full precision as sparse outliers, with the bulk of weights quantized to low bitwidths. However, standard SpQR computes scales and identifies outliers before running GPTQ's error propagation, which means that weights that were originally within the representable range may overflow after error propagation adds compensation from earlier quantization steps, and these overflows are clipped.

SSQR's innovation is a scale-adjustment mechanism that uses binary search to tune the per-channel scales so that after GPTQ's error propagation without clipping, the fraction of overflowed weights (stored as sparse outliers) exactly matches a target density $\rho \in [0, 1]$.

Algorithm 9 (SSQR, outer loop):

Line 3: Compute MSE-optimal scales $\mathbf{S}_{\text{MSE}}$ — the standard approach that minimizes the L2 quantization error under the assumption of independent rounding.

Line 4: Initialize binary search bounds: $\mathbf{s}_{\text{min}} \leftarrow \mathbf{0}_r$, $\mathbf{s}_{\text{max}} \leftarrow \mathbf{2}_r$ — per-channel multiplicative factors applied to the MSE scales. The initial scale is $\mathbf{s} \leftarrow (\mathbf{s}_{\text{min}} + \mathbf{s}_{\text{max}}) / 2 = \mathbf{1}_r$ (no adjustment yet).

Main loop (lines 6–12): While $\mathbf{s}$ has not converged:

• Line 8: Adjust the per-channel scales: $\mathbf{S} \leftarrow \mathbf{S}_{\text{MSE}} \text{diag}(\mathbf{s})$ — multiply each output channel's scale vector by the corresponding factor $\mathbf{s}[i]$. This proportionally changes all scales within a channel, preserving the relative scaling between dimensions while adjusting the overall magnitude.

• Line 9: Run the SSQR inner procedure (Algorithm 10) — which is GPTQ with the no-clipping grid $\mathbb{Z}^{\dagger} = \mathbb{Z}$ and with overflow detection. Any weight that would be rounded to a value outside the representable integer range (e.g., outside $\{-8, \ldots, 7\}$ for INT4) is instead stored as a full-precision outlier in the sparse matrix $\boldsymbol{\Xi}$.

• Lines 10–11: For each output channel $i$, if the outlier density $\|\boldsymbol{\Xi}[:, i]\|_0 < \rho c$ (fewer outliers than the target rate), then the scale is too large (overflows are rare but the quantization bins are too coarse, losing precision for inliers) — shrink the scale by setting $\mathbf{s}_{\text{max}}[i] \leftarrow \mathbf{s}[i]$. Otherwise, the scale is too small (too many overflows) — grow the scale by setting $\mathbf{s}_{\text{min}}[i] \leftarrow \mathbf{s}[i]$.

• Line 12: Update $\mathbf{s} \leftarrow (\mathbf{s}_{\text{min}} + \mathbf{s}_{\text{max}}) / 2$.

Why binary search works: The outlier rate is negatively related to the scales — larger scales mean that weight values (which after error propagation can be larger than the original weights) are divided by larger numbers before rounding, so fewer values fall outside the representable range. The binary search exploits this monotonic relationship to find the smallest scales that keep the outlier rate at or below the target.

Why proportional adjustment: Exhaustive trial-and-error over per-group scales (each group within a channel could have its own scale factor) would be "infeasible in large layers" because the search space grows exponentially with the number of groups. By only adjusting a single multiplicative factor per output channel, the search space reduces to one dimension per channel, and the binary search converges quickly (logarithmic in the precision of $\mathbf{s}$).

Algorithm 10 (SSQR inner procedure): This is GPTQ (Algorithm 1) with three modifications:

1. The quantization grid is $\mathbb{Z}^{\dagger} = \mathbb{Z}$ (no clipping — any integer is allowed).
2. Line 11 (new): After rounding, any weight element $j$ in channel $i$ whose rounded integer $\mathbf{Z}[j, i]$ does not lie within the target representable range (e.g., INT4: $\{-8, \ldots, 7\}$) is detected: its contribution is stored as a sparse outlier $\boldsymbol{\Xi}[j, i] \leftarrow \mathbf{W}[j, i] - \mathbf{Z}[j, i] * \mathbf{S}[j, i]$ (the residual not representable by the low-bitwidth integer), and the integer is set to zero ($\mathbf{Z}[j, i] \leftarrow 0$).
3. Line 12 (modified): The dequantized weight is $\mathbf{Q}[j, :] \leftarrow \mathbf{Z}[j, :] * \mathbf{S}[j, :] + \boldsymbol{\Xi}[j, :]$ — the low-bitwidth component plus the sparse full-precision correction.

The output is the quantized integers $\mathbf{Z}$, the sparse outlier matrix $\boldsymbol{\Xi}$, and the dequantized weights $\mathbf{Q}$. At inference time, the matrix multiplication is split into a dense low-bitwidth matmul (handled by the custom CUDA kernel) and a sparse full-precision matmul for the outliers.

Huffman-Encoded Post-Training Quantization (HPTQ, Algorithm 11). HPTQ takes a different approach: instead of separating inliers and outliers, it represents all weights as integers on a uniform grid, but encodes those integers with Huffman coding to achieve a target average bitwidth. This fully aligns with the CVP lattice formulation because every integer — no matter how large — is allowed; the cost of representing large integers is paid in the variable-length code rather than through clipping.

Algorithm 11 walkthrough:

Line 2: Initialize binary search bounds: $s_{\text{min}} \leftarrow 0$, $s_{\text{max}} \leftarrow \|\mathbf{W}\|_{\infty}$ — the maximum absolute weight value (or a multiple thereof). The scale $s$ is a single scalar shared across all weights (in contrast to SSQR's per-channel scales).

Line 3: Initial scale: $s \leftarrow (s_{\text{min}} + s_{\text{max}}) / 2$.

Main loop (lines 5–20): While $s$ has not converged:

• Line 7: Broadcast the scale to all weights: $\mathbf{S} \leftarrow s \cdot \mathbf{1}_{c \times r}$ — every weight element gets the same scale. This is the simplest possible quantization scheme (uniform quantization) and ensures the lattice basis is just $s \cdot \mathbf{X}$ — no per-channel or per-group variations.

• Line 8: Run standard GPTQ (Algorithm 1) with $\mathbb{Z}^{\dagger} = \mathbb{Z}$ (no clipping) and the uniform scale $\mathbf{S}$. This produces integer weights $\mathbf{Z}$ that may contain large values (since no clipping is enforced).

• Line 9: Compute $h'$ — the average Huffman coding bitwidth of all integer values in $\mathbf{Z}$. Huffman coding assigns shorter bit sequences to more frequent values and longer sequences to rare values, achieving near-entropy coding efficiency. The average bitwidth is the sum of (frequency of each integer value × its code length) across all weight elements.

• Lines 10–17: If $h' < h$ (the current scale produces fewer bits than the target), the scale is too large (the integers are concentrated near zero and easily compressed, but quantization is too coarse) — shrink the scale for finer quantization: $s_{\text{max}} \leftarrow s$. If $h' > h$ (too many bits), the scale is too small (the integers are too spread out, requiring longer codes) — grow the scale: $s_{\text{min}} \leftarrow s$.

• Line 19: Update $s \leftarrow (s_{\text{min}} + s_{\text{max}}) / 2$.

Why Huffman coding cost as a proxy for bitwidth: Huffman coding produces a prefix-free code whose expected length is between the entropy and entropy+1 bit per symbol. By using the actual Huffman coding cost (computed from the empirical frequency distribution of the integer values in $\mathbf{Z}$), HPTQ directly measures the storage cost of the quantized representation. This is more accurate than simply assuming a fixed bitwidth, because the no-clipping integers follow a distribution that may be heavy-tailed (many values near zero, some large outliers), and Huffman coding naturally handles this.

Why binary search over scale: The relationship between the scale $s$ and the Huffman bitwidth $h'$ is monotonic: smaller scales produce more spread-out integer distributions (higher entropy, more bits), while larger scales concentrate the integers near zero (lower entropy, fewer bits). The binary search efficiently finds the scale that hits the target bitwidth by bracketing and bisecting.

Huffman-Encoded RTN (HRTN, Algorithm 12). As a baseline for HPTQ, the paper introduces HRTN, which uses the same Huffman-coding and binary-search framework but replaces GPTQ with simple round-to-nearest (RTN) quantization — no error propagation. The comparison between HPTQ and HRTN isolates the benefit of GPTQ's error propagation (equivalently, Babai's nearest plane algorithm) from the benefit of Huffman coding alone.

Algorithm 12 walkthrough: Identical to HPTQ except line 7 becomes $\mathbf{Z} \leftarrow \text{Round}(\mathbf{W} / s, \mathbb{Z})$ — independent rounding of each weight element — and line 8 computes $\mathbf{Q} \leftarrow s \mathbf{Z}$. The binary search over $s$ is otherwise the same.

Experimental configurations (Section 5, Appendix D.2–D.4). The paper tests these methods on the Qwen3 family (0.6B, 1.7B, 4B, 8B, 14B parameters). The calibration dataset uses 256 sequences of length 2048 from FineWeb-Edu (shuffled with a fixed seed). Evaluation uses WikiText-2 and C4 perplexity, plus zero-shot accuracy on WinoGrande, MMLU, HellaSwag, PIQA, SciQ, and TruthfulQA. The custom CUDA kernel for SSQR targets Ampere GPUs (NVIDIA RTX A6000), supporting 2–4 bit inlier weights with unstructured sparse outliers, optimized for low-batch (batch size 1, 128 generated tokens) inference latency, with separate SIMT and tensor core paths depending on batch size.

4. Key Insights and Innovations

Innovation 1: GPTQ Is Not an Ad-Hoc Engineering Trick — It Is a Classical Lattice Algorithm in Disguise

The paper's most fundamental intellectual contribution is conceptual, not algorithmic: it proves that GPTQ's sequence of algebraic weight updates, which had been described in the original work (Frantar et al., 2023) as a greedy procedure with no geometric meaning or worst-case guarantees, is mathematically identical to Babai's nearest plane algorithm (Babai, 1986) — a polynomial-time CVP heuristic that has been studied in the lattice theory community for nearly 40 years. This is not a minor re-derivation or a tighter bound on an existing analysis; it is a complete reframing of what GPTQ is.

What makes this distinctive at the idea level. Prior to this work, GPTQ was understood purely algebraically — as a sequence of matrix updates driven by the LDL decomposition of the inverse Hessian. The QuIP paper (Chee et al., 2023) had proven some error guarantees within this algebraic framework, but no one had recognized that the entire procedure corresponds to an orthogonal walk through nested affine subspaces defined by the Gram-Schmidt vectors of the Hessian lattice. The paper's chain of equivalences — quantization-as-CVP (Theorem 1), OBQ's error propagation as Babai's projection (Theorem 2), and GPTQ as Babai without LLL reduction (Theorem 4) — converts a heuristic that "seemed to work" into an instance of a well-understood algorithm with known approximation properties. This is a fundamental shift in how the field should think about second-order quantization methods: they are not ad-hoc greedy optimizers but rather lattice decoding algorithms, and their behavior is governed by the geometry of the Hessian-induced lattice.

Comparison to prior framing. The dominant assumption in the quantization literature has been that GPTQ is a pragmatic simplification of OBQ — sacrificing the dynamic dimension selection (which minimizes per-step error) for the computational efficiency of a fixed order, with unknown consequences for global error accumulation. The paper shows that this framing is incomplete. The fixed order is not a compromise; it is a choice of basis ordering for Babai's algorithm, and the error it introduces is bounded by a closed-form expression involving the LDL decomposition's diagonal entries (Theorem 5). The greedy error propagation, which seemed mysterious in its global effectiveness, is revealed to be the standard Gram-Schmidt orthogonal projection that Babai's algorithm uses, with the crucial property that errors from successive steps are orthogonal and therefore add in Euclidean norm rather than interfering constructively. This reframing explains why the greedy procedure works globally — a question the original GPTQ paper left completely open.

Significance beyond raw performance. The equivalence opens a two-way channel between LLM quantization and lattice theory (Section 6). On one side, decades of CVP algorithms — basis reduction techniques (LLL, BKZ), alternative CVP solvers, and analyses of approximation ratios — become available for designing future quantizers. On the other side, the behavior of massive neural network layers may inspire new questions for lattice theory, since the Hessian matrices of transformer layers exhibit structure (e.g., well-conditioned diagonals, attention-specific patterns) that may not have been studied in classical lattice contexts. This cross-pollination is potentially more impactful than any single accuracy improvement because it reframes the entire research agenda: rather than inventing new quantization heuristics from scratch, researchers can now ask "which lattice algorithm should we apply to this quantization problem, and how should we modify it to handle the practical constraints of LLMs?"

Evidence anchoring. The equivalence is not conjectural — it is proved rigorously through the three-step algebraic argument in Appendix B, which establishes that the rounding decisions of back-to-front GPTQ and Babai's algorithm are identical at every step, with a further proof (Appendix B.4) that additional GPTQ-style error propagation after Babai's algorithm produces zero change, confirming the equivalence is tight. The geometric interpretation is visualized in Figures 2 and 3, which show the projection geometry in 2D and 3D.

---

Innovation 2: Weight Clipping Is Not Just a Practical Nuisance — It Severs the Theoretical Guarantee

The paper's second key insight is diagnostic rather than constructive: clipping overflowed quantized values to the representable range breaks the CVP equivalence and voids Babai's error bound. This is not an engineering detail — it is a structural observation about why GPTQ degrades sharply at low bitwidths, and it provides a principled explanation for a phenomenon that practitioners have observed but not understood.

What makes this distinctive at the idea level. The standard practice in post-training quantization is to clip overflowed weights to the nearest boundary of the quantization grid (e.g., for INT4, values are capped at ±7 or ±8 depending on the signed range). This is treated as a practical necessity — the grid is finite, so values outside it must be truncated — and the hope is that GPTQ's error propagation will compensate for the clipping error as it does for rounding error. The paper reveals that this hope is unfounded: clipping replaces the integer lattice Z^c with a bounded subset Z^{†c}, which is not a lattice, and the CVP equivalence requires an unrestricted integer grid. Once clipping is introduced, Babai's orthogonal projection guarantee no longer holds because the target is being projected onto a truncated hyperplane — the error component that would have been absorbed by the projection is instead arbitrarily large when the target lies far from the truncation boundary.

This is a conceptual diagnostic rather than a new method. It tells the field not what to do but what the problem is: the degradation of GPTQ at low bitwidths is not an inevitable consequence of limited precision but a specific consequence of violating the lattice structure. The solution is not to improve the error propagation formula (which is already optimal for the unclipped problem) but to eliminate clipping entirely through representations that accommodate overflowed values, such as sparse outliers (SSQR) or variable-length coding (HPTQ).

Comparison to prior practice. Standard GPTQ and its derivatives (including SpQR; Dettmers et al., 2024) apply clipping as a matter of course. The original GPTQ paper does not discuss the theoretical implications of clipping; it is simply part of the rounding step. The current paper is the first to identify clipping as the specific operation that breaks the CVP correspondence and to provide a theoretical explanation for an empirical pattern: GPTQ's accuracy degrades more sharply than simple rate-distortion theory would predict as bitwidth decreases, because at lower bitwidths, a larger fraction of weights overflow the representable range, and these clipping errors are not properly compensated by the error propagation (which was designed for rounding errors within an unbounded grid). The experimental evidence in Table 3 bears this out: standard GPTQ at 3.125 bits achieves 12.77 WikiText-2 perplexity on Qwen3-8B, while the no-clipping HPTQ at the same effective bitwidth achieves 10.34 — a 2.43-point gap that is far larger than what a simple precision loss would explain.

Significance beyond raw performance. This insight redirects research on low-bitwidth quantization. Rather than trying to squeeze more information into the low-bitwidth representation (e.g., through non-uniform quantization, learned codebooks, or more sophisticated rounding), the priority should be to find representations that eliminate clipping while staying within a storage budget. The paper's SSQR and HPTQ are two concrete instantiations of this principle, but the conceptual contribution is the principle itself: any quantization method that clips weights is operating without the theoretical safety net that Babai's equivalence provides, and its error behavior at low bitwidths is therefore unpredictable from the lattice-theoretic analysis.

Evidence anchoring. Theorem 5's error bound is explicitly stated under the assumption Z^{†} = Z (no clipping), and the paper notes that "the original GPTQ algorithm clips the overflowed integers at the rounding step, introducing large errors that violate the error bound" (Section 5). The practical experiments in Figure 4a and Table 3 demonstrate the consequence: methods that avoid clipping (HPTQ, SSQR) substantially outperform clipped GPTQ at matched effective bitwidths, with the gap widening as bitwidth decreases — exactly what the theory predicts, since lower bitwidths produce more frequent overflows and therefore more clipping-induced violations of the bound.

---

Innovation 3: The Quantization Order Has a Principled Optimization Criterion — It Is the Pivot Order of the LDL Decomposition

The paper's third contribution is to transform the quantization order from a heuristic choice into a principled optimization variable with a closed-form objective. Prior to this work, GPTQ's "act-order" heuristic (descending Hessian diagonal) was justified by intuition — quantize less "important" dimensions first so that the error propagation has more degrees of freedom to compensate — but there was no formal criterion for what constitutes a good order, nor any guarantee that the heuristic was close to optimal. The paper shows that the quantization order is exactly the pivot order of the LDL decomposition of the permuted Hessian, and that the error bound \frac{1}{4} (\mathbf{T}^{-1} \mathbf{s}_i)^{\top} \mathbf{D} (\mathbf{T}^{-1} \mathbf{s}_i) provides a direct objective: choose the permutation T that minimizes this quadratic form.

What makes this distinctive at the idea level. The connection between quantization order and LDL pivot order is not a superficial relabeling — it reveals a structural property that was invisible in the weight-space formulation. In Babai's algorithm, the order corresponds to the sequence in which the Gram-Schmidt orthogonalization processes the basis vectors. Processing short vectors first (so that the remaining long vectors are not shortened as much by orthogonalization against them) minimizes the sum of squared Gram-Schmidt lengths, which is exactly the trace of D. The paper's min-pivot heuristic (Algorithm 3) directly implements this geometric intuition: at each LDL step, greedily select the remaining dimension with the smallest diagonal entry (i.e., the shortest residual basis vector), which is the locally optimal choice for reducing the next Gram-Schmidt length.

Comparison to prior practice. Prior work treated quantization order as a hyperparameter to be tuned empirically or selected via simple heuristics (act-order, random, or fixed front-to-back/back-to-front). There was no theoretical guidance on whether the order mattered beyond a few percent of accuracy, or whether further optimization could yield gains. The paper provides both theoretical and empirical evidence: the order matters substantially for the error bound (Table 2 shows act-order reduces tr(D) by 35–50% relative to baseline orders on Qwen3-8B), but the downstream accuracy gains from further optimizing beyond act-order are modest, suggesting that act-order already captures the bulk of the available benefit when the Hessian is well-conditioned (i.e., when diagonal entries decay smoothly). This is a refinement of an existing heuristic rather than a fundamental algorithmic change, but it provides the theoretical justification that was missing: act-order works because it approximates the greedy min-pivot choice, and min-pivot is optimal (in a greedy sense) for minimizing the error bound.

Significance beyond raw performance. The contribution here is not a large accuracy improvement but rather the conversion of a heuristic into an understood optimization. Knowing why act-order works enables principled extensions: if the Hessian is ill-conditioned (e.g., for certain layer types or architectures where diagonal entries vary by orders of magnitude), min-pivot may be worth the extra O(c^3) cost. More broadly, the connection between pivot order and Gram-Schmidt orthogonalization opens the door to more sophisticated ordering strategies inspired by the sparse matrix literature (e.g., minimum degree, nested dissection), where decades of research on pivot selection for numerical stability and fill-in reduction may directly transfer to quantization.

Evidence anchoring. Table 2 (Appendix C.3) reports tr(D) for five orders across four layer types in Qwen3-8B block 18. Min-pivot consistently achieves the smallest trace, with act-order very close behind. The paper explicitly notes that "downstream accuracy gains are modest," framing this as a theoretical consolidation rather than a practical breakthrough — the value is in understanding why the existing heuristic works, not in replacing it.

---

Innovation 4: The FLOPs-Matched Pretraining vs. Test-Time Compute Tradeoff Depends Critically on Problem Difficulty

The paper's fourth contribution is an empirical characterization of the boundary conditions under which test-time compute can substitute for pretraining compute. While prior work (Jones, 2021; Villalobos and Atkinson, 2023) had studied the training-inference tradeoff in the abstract, the current paper provides the first results in a realistic LLM setting with actual models (PaLM 2-S* and a ~14× larger variant), real benchmarks (MATH), and, critically, a breakdown by problem difficulty that reveals why the tradeoff sometimes favors test-time compute and sometimes favors pretraining.

What makes this distinctive at the idea level. The paper's central finding here is that test-time and pretraining compute are not fungible — the exchange rate depends sharply on both problem difficulty (how far the problem is from the base model's capability frontier) and the inference-to-pretraining token ratio R. On easy problems where the base model already produces correct solutions at a non-trivial rate, test-time compute with a smaller model can substantially outperform a much larger model, achieving up to ~28% relative improvement on medium-difficulty problems at low R (Figure 1, top-right bar chart). On the hardest problems where the base model's pass@1 is near zero, test-time compute provides essentially no benefit regardless of budget — the larger model always wins, with the disadvantage reaching \−53% for PRM search at high R. This is not a simple "test-time compute is better" or "pretraining is better" claim; it is a conditional characterization that tells practitioners exactly when to prefer which strategy.

Comparison to prior work. Prior training-inference tradeoff analyses largely assumed access to ground-truth answers at inference time (to verify correctness and guide search), making them optimistic about test-time compute's potential. The current paper operates in a realistic setting where the correct answer is unknown, using learned verifiers (PRMs) and self-revision to improve outputs. This is a harder setting, and the results are correspondingly nuanced: test-time compute is powerful but not a panacea. The paper is the first to explicitly connect the tradeoff to problem difficulty, providing a unifying explanation for why some studies found that LLMs can self-correct (Madaan et al., 2023) while others found they cannot (Huang et al., 2023) — these studies were implicitly testing on different difficulty distributions.

Significance beyond raw performance. This insight reframes how organizations should think about their total compute budget. Rather than the prevailing paradigm of "train the largest model you can afford, then deploy with greedy decoding," the results suggest a hybrid deployment strategy: a smaller model with variable test-time compute handles easy-to-medium queries, with hard queries routed to a larger model or flagged for human review. The difficulty estimator (predicted via the PRM's average score) serves double duty — it determines how much test-time compute to allocate and whether to escalate. This is a practical architectural implication that goes beyond the paper's specific experimental setup.

Evidence anchoring. The FLOPs-matched comparison in Figure 9 and the bar charts in Figure 1 show the difficulty-dependent tradeoff explicitly. The line plots in Figure 9 show that on difficulty bin 1 (easiest), the compute-optimal scaling curve is above the ~14× larger model's performance at all three R values; on bin 5 (hardest), the scaling curve is essentially flat near 0–5% and below the larger model at all R values. The paper is transparent about the boundary conditions, and the difficulty-dependence is robust across both search and revision methods.

---

Innovation 5: Verifier Over-Optimization Is the Primary Bottleneck in Test-Time Compute Scaling — Not Search Algorithm Sophistication

The paper's fifth contribution is a negative result with diagnostic value: more sophisticated search algorithms (lookahead search, beam search with large branching factors) can paradoxically degrade performance due to over-optimization of the learned verifier, and the optimal strategy is often to use weaker optimization (best-of-N) on problems where the verifier is reliable. This finding redirects research attention from developing better search algorithms to building more robust verifiers.

What makes this distinctive at the idea level. The paper documents a phenomenon — verifier over-optimization — that is well-known in the RLHF literature (reward hacking) but had not been systematically characterized in the context of test-time search for LLM reasoning. The evidence is multi-pronged and consistent: beam search degrades easy-problem accuracy at high budgets (Figure 3, right), lookahead search — the most powerful optimization method — paradoxically performs worst overall at matched generation budgets (Figure 3, left), and qualitative examples in Appendix M show search producing degenerate outputs (repetitive low-information steps, overly short solutions) that achieve high PRM scores but are incorrect. The compute-optimal policy, which routes easy problems away from aggressive search, can be understood as a mechanism to stay below the over-optimization threshold — using weak optimization where the verifier is reliable and strong optimization only where the verifier signal has genuine headroom to provide guidance.

Comparison to prior work. Prior work on test-time compute (Cobbe et al., 2021; Lightman et al., 2023; Yao et al., 2023) implicitly assumed that more sophisticated search would yield monotonic improvements, with the main design question being which search algorithm to use. The current paper shows that this assumption is false in the regime where verifiers are imperfect. The finding that lookahead search underperforms simpler methods is particularly striking — it contradicts the intuitive expectation that "looking ahead" should improve decision quality, and it suggests that the PRM's per-step scores have limited predictive value beyond what the current step already reveals.

Significance beyond raw performance. This insight changes the prioritization of research efforts. Rather than developing ever-more-sophisticated search algorithms (Monte Carlo Tree Search, beam search with dynamic branching, A*-style heuristics), the primary bottleneck is verifier robustness. Future work should focus on training PRMs that remain calibrated under distribution shift (since search produces outputs that differ from the i.i.d. samples the PRM was trained on), using techniques like adversarial training, ensembles, or KL-penalized search that prevents the policy from deviating too far from the base model's output distribution. The paper's compute-optimal framework implicitly addresses this by routing problems to the search algorithm that stays within the verifier's reliability regime, but this is a mitigation rather than a solution — hard problems still require verifiers that can guide aggressive search without over-optimizing.

Evidence anchoring. Figure 3 (left) shows the performance of best-of-N, beam search, and lookahead search as a function of generation budget. At 256 generations, lookahead search with k=3 and M=√N achieves approximately 32% accuracy, versus 37% for beam search (M=4) and 38% for best-of-N weighted. The difficulty-bin analysis (Figure 3, right) isolates the mechanism: on bin 1, beam search accuracy actually decreases from 78% to 77% as the budget grows from 4 to 256, while best-of-N increases from 68% to 88%. The qualitative examples in Appendix M (Figures 29 and following) show concrete instances of verifier exploitation: solutions become shorter and more repetitive as search intensity increases, indicating that the PRM assigns high scores to solutions that say very little but avoid making detectable errors — a classic reward hacking pattern.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021), specifically the split from Lightman et al. (2022): 12,000 training questions and 500 test questions. MATH consists of high-school competition-level math problems requiring multi-step symbolic reasoning — a domain where test-time compute is expected to help because the model possesses the necessary knowledge and the challenge lies in drawing complex inferences rather than recalling novel facts.

• Base model(s). All main experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" (Section 4) and sits in a useful intermediate performance regime: non-trivial pass@1 on MATH (roughly 10–19% depending on prompting and sampling configuration) but far from saturation, leaving room for test-time compute to make a measurable difference. For the FLOPs-matched comparison (Section 7), a second model with approximately 14× more parameters is used as the pretraining-scaled baseline, with greedy decoding and no additional test-time compute.

• Metrics. The primary metric throughout is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). For difficulty-dependent analyses, accuracy is reported within each of five difficulty quintiles separately (see Section 3 for the difficulty estimation procedure). For the FLOPs-matched comparison, accuracy differences relative to the larger model are reported as percentage-point changes.

• Baselines. The paper compares against several standard approaches:

  • Majority voting: Select the most common final answer among N independently sampled solutions, with no learned verifier.
  • ORM best-of-N weighted: Score N solutions with an outcome reward model (trained using the same Monte Carlo rollout procedure as the PRM but predicting only final-answer correctness) and apply best-of-N weighted selection (Li et al., 2023), where solutions arriving at the same final answer have their scores summed and the answer with the greatest total sum is selected.
  • PRM best-of-N weighted: Same as above but using the process reward model for scoring, with last-step aggregation (found to perform best in Appendix E, Figure 13).
  • Parallel sampling (for revision model comparisons): Generate N independent solutions from the revision model and select the best via verifier-based selection or majority voting. This is the fully-parallel endpoint of the sequential-to-parallel ratio sweep.

• Generation budget / compute accounting. The universal unit of test-time compute is one generation — one complete sampled answer from the base LLM. For best-of-N and majority voting, the budget equals the number of samples N. For beam search, the budget equals the number of beams (solutions) kept at each step, with the total number of generated tokens being comparable to N complete solutions. For lookahead search with k lookahead steps, the cost is N × (k+1) generations to account for the additional rollout computation (Section 5.2). For revision models, the budget equals the total number of generated solutions (sequential revisions within a chain plus parallel chains). Budgets are swept across powers of 2, typically from 2⁰ to 2⁹ (1 to 512 generations). Maximum budgets vary by experiment: search experiments use up to 256 generations (Section 5.3), revision experiments use up to 512 (Section 6.2), and FLOPs-matched comparisons use variable budgets determined by the pretraining-to-inference FLOP ratio.

• Cross-validation / statistical protocol. To avoid contaminating strategy selection with test-set performance, the authors use two-fold cross-validation within each difficulty bin on the 500-question test set (Section 3.2). The best-performing strategy (search algorithm and hyperparameters, or sequential-to-parallel ratio) is selected on one fold and evaluated on the other, with results averaged. Difficulty bins are based on either oracle pass@1 rates (requiring ground-truth answers) or predicted difficulty (using the PRM's average final-answer score across 2048 samples per question), with the latter being the deployable variant. The 500 questions are split into quintiles of approximately 100 each, and cross-validation splits each quintile roughly in half, meaning strategy selection is based on approximately 50 questions per fold per bin.

---

Main Quantitative Results

Search Against PRM Verifiers (Section 5, Figures 3–4)

Aggregate comparison across all 500 test questions (Figure 3, left). At a maximum budget of 256 generations:

• At low budgets (2–8 generations): Beam search with M = 4 significantly outperforms best-of-N weighted. The paper reports that at 4 generations, beam search (M = 4) achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted — an 11 percentage-point gap. Beam search with M = √N also outperforms best-of-N but by a smaller margin.

• At high budgets (64–256 generations): Beam search performance flattens and eventually falls slightly below best-of-N weighted. Best-of-N weighted reaches approximately 38% at 512 generations; beam search (M = 4) plateaus around 34% at 256 generations and does not improve further. The crossover occurs around 32–64 generations — below this budget, beam search dominates; above it, best-of-N weighted closes the gap and pulls ahead.

• Lookahead search: Both k = 1 and k = 3 variants generally underperform at the same generation budget due to the higher per-step cost (each lookahead step costs an additional generation per beam). The 3-step lookahead variants converge to similar performance as other methods at very high budgets but never surpass them. Lookahead search with M = 4 and k = 3 performs worst at low-to-medium budgets, confirming that the extra computation spent on lookahead does not pay off in improved decision quality.

• Majority voting: Trails all verifier-based methods substantially, reaching only about 29% at 512 generations, confirming that learned verifiers provide significant value over simple consensus.

Difficulty-bin analysis for search (Figure 3, right). When broken out by difficulty quintile (beam search M = 4 vs. best-of-N weighted, shown at four budget levels: 4, 16, 64, 256):

• Bin 1 (easiest): Beam search accuracy decreases from roughly 78% to 77% as the budget grows from 4 to 256, while best-of-N weighted increases from 68% to 88%. This is the clearest evidence of PRM over-optimization — beam search finds solutions that score highly under the PRM but are increasingly likely to be incorrect, while best-of-N's passive sampling avoids this pathology.

• Bin 2: Beam search improves modestly (roughly 14% → 32% from 4 to 256 generations) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets. The crossover favors best-of-N at moderate-to-high budgets.

• Bin 3: Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations — an 11 percentage-point advantage. This is the "sweet spot" where beam search's active optimization genuinely helps navigate toward correct solutions that random sampling would miss.

• Bin 4: Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations. The absolute numbers are low, but the relative improvement is substantial (approximately 70%).

• Bin 5 (hardest): Both methods hover near 1–3% accuracy regardless of budget. No method makes meaningful progress — the base model simply cannot produce correct solutions at any appreciable rate, and neither search nor sampling can create capability that the base model lacks.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level (using the two-fold cross-validation protocol):

• At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a 4× compute reduction (16 vs. 64 generations for equivalent accuracy).

• At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).

• Compute-optimal with predicted difficulty bins (using the PRM's average score, no ground-truth labels) tracks the oracle version closely. The paper states the two curves "largely overlap" (Figure 4), with the predicted version reaching approximately 37% at 256 generations vs. 39.5% for oracle — a gap of roughly 2.5 percentage points that narrows at lower budgets.

• Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%), confirming that adaptive strategy selection provides gains beyond simply using a stronger verifier.

PRM vs. ORM (Appendix F, Figure 14). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples, confirming the PRM's superior scaling properties — the step-level training provides representations that remain informative even when the aggregation method effectively reduces to outcome-level prediction (since last-step aggregation is used).

---

Revision Model Results (Section 6, Figures 6–8)

Revision model pass@1 trajectory (Figure 6, left). Starting from approximately 18.2% pass@1 at step 1 (the revision model's initial generation), the per-step accuracy improves to roughly 24–25% by steps 15–20 and remains in the 23–25% range out to 64 steps. The model generalizes beyond its 4-step training horizon — although trained only with up to 4 previous incorrect answers in context, the improvement continues for dozens of additional revision steps, though with diminishing returns. The accuracy plateaus rather than declining, suggesting the model has learned a generalizable revision skill rather than overfitting to the training trajectory length.

Sequential vs. parallel sampling (Figure 6, right). At 64 generations:

• Sequential + best-of-N weighted: approximately 41.5%
• Parallel + best-of-N weighted: approximately 39%
• Sequential + majority: approximately 38%
• Parallel + majority: approximately 35%

Sequential revisions outperform parallel sampling under both selection mechanisms, with the verifier-based gap (roughly 2.5 percentage points) being slightly narrower than the majority-based gap (roughly 3 points). This indicates that part of sequential revision's benefit comes from the verifier seeing revision context (the revision-specific ORM includes previous revisions in its input), but even without this — when using majority voting, which has no access to revision history — sequential still outperforms parallel, confirming that the revisions genuinely produce better candidate solutions, not just better-verifiable ones.

Sequential-to-parallel ratio sweep (Figure 7, left). For a fixed generation budget of N total generations, the paper sweeps the ratio of sequential revisions per chain to parallel chains, with the constraint that (sequential length) × (parallel count) = N:

• At 256 generations, the optimal ratio is around 2¹ to 2³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43–44% accuracy.
• Fully parallel (leftmost point, N parallel chains of length 1) yields approximately 40%.
• Fully sequential (rightmost point, 1 chain of length N) yields approximately 42%.
• The curve is concave — the optimum lies in the interior rather than at either extreme — indicating that both diversity (multiple chains) and depth (multiple revisions per chain) contribute to accuracy.
• At lower budgets (8–32 generations), fully sequential is optimal — the curves are monotonically increasing with the sequential-to-parallel ratio, suggesting that when the total budget is small, it is better to invest all of it in refining a single chain rather than splitting across multiple shallow chains.

Difficulty-dependent optimal ratio (Figure 7, right). At a fixed budget of 128 generations, broken out by difficulty bin:

• Bin 1: Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to the allocation strategy — the model's initial answers are already mostly correct, and neither additional revisions nor parallel diversity provide meaningful gains.
• Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel. The gain from refinement exceeds the gain from diversity for moderately easy problems.
• Bin 3: A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2¹ to 2³), reaching approximately 42% vs. 35% at the extremes. Both diversity and depth matter here — the model needs to explore different high-level approaches (parallel chains) and refine within each approach (sequential revisions).
• Bin 4: Similar pattern to bin 3, with the peak at a moderate ratio achieving roughly 18% vs. 14% at fully parallel. The absolute numbers are much lower, but the relative benefit of the balanced ratio persists.
• Bin 5: All ratios produce roughly 2–3% accuracy. No allocation strategy helps on the hardest problems — the base model cannot produce correct solutions regardless of how the budget is spent.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin (via two-fold cross-validation):

• At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a 4× improvement in compute efficiency (64 vs. 256 generations for equivalent accuracy).
• At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted (the standard baseline) and 37% for parallel-only revision sampling.
• Compute-optimal with predicted difficulty bins performs slightly below oracle bins at high budgets (approximately 41% at 256 generations vs. 44% for oracle) but still substantially outperforms the parallel baseline. The gap between oracle and predicted difficulty widens at higher budgets, suggesting that difficulty estimation errors become more consequential when the budget is large and the strategy choice matters more.
• Notably, the parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve. This suggests that the gains from adaptive allocation compound at higher budgets — as more compute becomes available, knowing how to spend it becomes increasingly important, and uniform allocation leaves increasing amounts of efficiency on the table.

---

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7, Figures 1 and 9)

Experimental setup. The comparison evaluates whether a smaller model (PaLM 2-S*) with compute-optimal test-time scaling can match or exceed a model with approximately 14× more parameters under a fixed total FLOPs budget. The larger model uses greedy decoding with no test-time augmentation. Three values of the inference-to-pretraining token ratio R = D_inference / D_pretrain are tested: R ≪ 1 (0.16), R ≈ 1 (0.79), and R ≫ 1 (22). For each R, the smaller model's test-time compute budget is determined by the formula in Section 3.6:

Multiplier = M + 3 · (D_pretrain / D_inference) · (M − 1)

where M ≈ 14 is the parameter scaling factor. When R ≪ 1, the smaller model gets a large inference budget because pretraining savings dominate; when R ≫ 1, the budget is tight because the larger model's per-token inference cost is a bigger fraction of total compute.

Revisions results (Figure 9, left; Figure 1, top-right bar chart). Comparing PaLM 2-S* with compute-optimal revisions against the ~14× larger model, broken out by aggregated difficulty:

(Numbers quoted from the bar chart in Figure 1, top-right. Note: the "easy/medium/hard" groupings in the bar chart aggregate the five difficulty bins for readability.)

At R ≪ 1, test-time compute outperforms the larger model across all difficulty levels. This is the regime most favorable to test-time compute because the pretraining FLOPs saved by using the smaller model are enormous relative to the additional inference FLOPs spent. At R ≫ 1, test-time compute remains preferable only on easy questions (+2.0%), with hard questions showing a −37.2% relative disadvantage — the inference cost of the smaller model's test-time strategies becomes a dominant fraction of total FLOPs, eroding the pretraining savings.

PRM search results (Figure 9, right; Figure 1, bottom-right bar chart). The pattern is starker:

PRM search shows weaker benefits than revisions for the FLOPs-matched comparison. On easy questions, test-time compute remains preferable across all R regimes, though the margin narrows from +19.1% at R ≪ 1 to only +2.0% at R ≫ 1. On medium and hard questions, the larger model dominates at moderate-to-high R, with disadvantages reaching −35% to −53%. The paper attributes this to PRM search's weaker scaling behavior — search benefits saturate at lower budgets than revisions, so when the test-time budget is constrained (as it is at higher R), search leaves more FLOPs on the table.

Figure 9 detail. The line plots show the accuracy of the PaLM 2-S* model with compute-optimal scaling as a function of test-time compute budget (in generations). The ~14× larger model's greedy-decoding performance is shown as horizontal lines at three x-axis positions corresponding to the three R values (the test-time budget that a small model can afford at that R). Where the scaling curve is above the horizontal line, test-time compute wins; where below, pretraining wins. On bin 1 (purple, topmost curve), the scaling line is above all three horizontal lines for revisions — test-time compute wins at all R. On bin 5 (blue, bottommost curve), the scaling line is below all three horizontal lines and essentially flat near 0–5%, confirming that no amount of test-time compute helps on the hardest problems.

---

Ablation Studies and Robustness Checks

PRM aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise score aggregation methods for the PRM: "last" achieves roughly 37% at 256 samples, "min" achieves roughly 35%, "prod" achieves roughly 27%, and a separately trained ORM achieves roughly 34%. The "last" aggregation's superiority is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time (using only the final-step prediction), yet the PRM still outperforms a separately trained ORM. The authors interpret this as evidence that step-level PRM training provides beneficial representation learning — the per-step supervision forces the model to learn features that are informative about correctness even when only the final prediction is used at decision time.

PRM vs. ORM (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%. The gap is smallest at low sample counts (~2 points at 8 samples) and grows to ~5 points at 2048 samples, confirming that the PRM's scaling advantage is most consequential when the search budget is large.

Revision model verifier choice (Appendix J, Figure 15a). The base-LM PRM (trained on PaLM 2-S* outputs) underperforms the revision-specific ORM when scoring revision model outputs: sequential + base-LM PRM achieves roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern — the revision model's output statistics differ from the base model's, and a verifier trained on the base model's distribution transfers imperfectly. The revision-specific ORM was trained on the revision model's outputs, closing this distribution gap.

Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's input context provides a small improvement over the no-history ablation: approximately 1–2 percentage points at 64 generations. However, both variants (with and without revision history) outperform the parallel baseline, confirming that the sequential sampling benefit is not solely attributable to the verifier seeing more context — the revisions themselves produce genuinely better candidate answers.

Oracle vs. predicted difficulty bins (Figures 4, 8, and Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. In the search setting (Figure 4), the predicted and oracle compute-optimal curves "largely overlap," with the predicted version reaching approximately 37% at 256 generations vs. 39.5% for oracle. In the revision setting (Figure 8), the gap is wider at high budgets: predicted reaches approximately 41% at 256 generations vs. 44% for oracle. This is the critical robustness check — the compute-optimal strategy works without ground-truth labels, though the gap at high budgets suggests that more accurate difficulty estimation could recover some of the lost performance.

Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This confirms that the ratio trends are not an artifact of the verifier's context window — they reflect genuine properties of the revision model's output distribution.

ReST_EM revision model (Appendix K, Figure 16). An attempt to further optimize the revision model using ReST_EM (Singh et al., 2024) — a reinforcement-learning-style self-improvement procedure — backfires: additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The paper hypothesizes that on-policy data collection in ReST_EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly. This is a notable negative result that highlights the sensitivity of revision training to the data generation procedure — the offline, edit-distance-based pairing used in the main experiments is not trivially replaceable with more sophisticated on-policy methods.

PRM over-optimization qualitative examples (Appendix M, Figures 29 and following). The paper includes concrete examples of solutions that score highly under the PRM but are incorrect, exhibiting failure modes like repetitive low-information steps at the end of solutions and overly short 1–2 step solutions that avoid making detectable errors. These examples provide qualitative support for the over-optimization hypothesis that explains the search degradation at high budgets.

---

Critical Assessment

The experiments in this paper are extensive along the axes the paper chooses to investigate — search algorithms, revision strategies, difficulty-dependent analysis, and the pretraining-inference FLOPs tradeoff — but have several limitations that constrain the generality of the conclusions.

Do the experiments support the central claims?

Claim: "Compute-optimal scaling improves efficiency by more than 4× over best-of-N." This claim is supported for both search and revisions under a specific definition of efficiency: achieving equivalent accuracy with 4× fewer generations. For search, Figure 4 shows compute-optimal with 16 generations matching best-of-N with 64 generations. For revisions, Figure 8 shows compute-optimal with 64 generations matching best-of-N with 256 generations. However, the 4× figure should be qualified: (1) it holds at moderate budgets (16–64 generations) but narrows at higher budgets (256–512) where the curves converge somewhat, especially with predicted difficulty bins; (2) the difficulty estimation cost (2048 samples per question) is not amortized into the budget — a deployment would need to account for this overhead, which could substantially reduce or eliminate the net efficiency gain for one-shot queries. The paper acknowledges this (Section 3.2) but the 4× figure is an upper bound on achievable efficiency, not a realized deployment gain.

Claim: "Test-time compute with a smaller model can outperform a ~14× larger model." This claim is supported conditionally. On easy-to-medium questions at low inference-to-pretraining ratios (R ≪ 1), the smaller model with compute-optimal revisions outperforms the larger model by substantial margins (+11.8% to +27.8% relative improvement, Figure 1 top-right). On hard questions at high R, the larger model dominates decisively (−37% to −53% disadvantage for the smaller model, Figure 1 bottom-right). The claim as stated ("can outperform") is true — it can, under the right conditions — but the conditions are restrictive and the paper is transparent about this. A significant caveat: the ~14× larger model uses greedy decoding only — no majority voting, no best-of-N, no search. Giving the larger model even a modest test-time compute budget (e.g., best-of-8) would create a much stronger baseline that is never tested. The paper's FLOPs accounting is also based on parameter scaling only (not Chinchilla-optimal joint scaling of parameters and data), which may favor test-time compute since the larger model is not compute-optimally trained.

Claim: "Efficacy depends critically on prompt difficulty." This claim is very strongly supported and is the most robust finding in the paper. The difficulty-bin analyses (Figures 3 right, 7 right) show qualitatively different — and sometimes opposite — effects of the same strategy at different difficulty levels: beam search hurts easy problems at high budgets (Figure 3 right, bin 1) while helping on medium problems (bins 3–4); sequential revisions dominate on easy problems while balanced ratios are optimal on hard problems (Figure 7 right). This pattern is replicated across search methods, revision strategies, and selection mechanisms (majority voting, Appendix B, Figure 10), and across both oracle and predicted difficulty bins. The finding is robust to the specific difficulty binning scheme because the qualitative patterns are clear — the per-bin curves are well-separated and non-crossing.

Claim: "GPTQ's greedy error propagation is equivalent to Babai's nearest plane algorithm." This claim is proved mathematically rather than demonstrated experimentally — the algebraic proof in Appendix B establishes the equivalence rigorously. The experimental results in Sections 5 and Appendices C–D demonstrate the practical consequences of this equivalence (the no-clipping methods that preserve the lattice structure outperform clipped GPTQ), but the equivalence itself is a theorem, not an empirical claim.

Genuine weaknesses in the experimental design

Single benchmark, single model family. All results are on MATH with PaLM 2-S*. The paper states the model is "representative" but this is unverified. The difficulty-dependent behavior (beam search over-optimizing on easy problems, revisions helping only on easy-to-medium problems) could be specific to PaLM 2-S*'s output distribution, PRM quality, or revision model training procedure. Replication on other model families (LLaMA, GPT, Mistral) and other reasoning benchmarks (GSM8K, MMLU-Math, TheoremQA) would substantially strengthen the generality claims.

Small test set for strategy selection. The test set of 500 questions split into five quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample — the variance of the selected strategy could be high, and the paper does not report confidence intervals on the compute-optimal scaling curves. It is unclear whether the observed differences between strategies (e.g., the optimal ratio in Figure 7) are statistically reliable at this sample size.

Difficulty estimation cost is unaccounted for. The paper explicitly states that "our experiments do not account for this cost largely for simplicity" (Section 3.2). Generating 2048 samples per question to estimate difficulty is extraordinarily expensive — it consumes more compute than the largest test-time budgets studied (256–512 generations). For a fair deployment comparison, this overhead must be amortized across multiple uses of the same question (e.g., if the same MATH problems are evaluated repeatedly) or replaced with a cheaper estimator. The paper does not evaluate how much the compute-optimal gains would shrink if difficulty estimation cost were included.

The larger model baseline is weak. The ~14× larger model uses greedy decoding with no test-time compute augmentation. A fairer comparison would give the larger model some test-time compute budget, since the whole point of the paper is that test-time compute is valuable — if it helps the small model, it likely helps the large model too. The paper's FLOPs-matched accounting is also based on parameter-only scaling, not Chinchilla-optimal training. A compute-optimally trained larger model (scaling both parameters and data) would be a stronger baseline.

No combination of PRM search with revisions. The paper studies search and revisions as independent mechanisms (Sections 5 and 6) but never combines them. A natural extension — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue — is not tested. The current results therefore represent a lower bound on what a fully integrated system could achieve, and the paper's claim that the two mechanisms have "complementary strengths" is based on their difficulty-dependent behavior patterns rather than direct experimental combination.

Latency ignored. The paper measures compute in generations, which is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial — each revision depends on the previous one — while parallel best-of-N can be executed simultaneously with sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously. For latency-sensitive applications, the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their FLOPs efficiency.

Missing experiments that would strengthen the paper

• Varying the number of difficulty bins. The five-quintile discretization is coarse. Testing with 3, 10, or continuous difficulty estimates would reveal whether the gains from adaptive allocation are limited by the discretization granularity.
• Dynamic difficulty estimation. Rather than pre-computing difficulty with 2048 samples, an adaptive scheme could start with a few parallel samples, assess difficulty from the PRM's scores, and allocate the remaining budget accordingly. This would amortize difficulty estimation into the solution process.
• Verifier robustness interventions. The paper identifies verifier over-optimization as the primary bottleneck but does not test any interventions to mitigate it (adversarial PRM training, ensemble verification, KL-penalized search). An ablation showing that these interventions improve search scaling would strengthen the claim that verifier quality is the bottleneck.
• Other reasoning benchmarks. MATH is a specific distribution of competition math problems. The difficulty-dependent patterns (especially the over-optimization on easy problems) may not generalize to benchmarks where "easy" problems have different characteristics (e.g., GSM8K for grade-school math, HumanEval for code generation).
• Latency-constrained compute allocation. A comparison of compute-optimal strategies under a wall-clock time budget rather than a generation budget would reveal how much of the 4× efficiency gain survives when parallelism constraints are enforced.

6. Limitations and Trade-offs

No-Clip Representations Incur Storage Overhead That Narrows the Compression Gain

The assumption or constraint: The paper's central theoretical contribution — the equivalence between GPTQ and Babai's nearest plane algorithm — holds only in the no-clipping regime where Z† = Z (unrestricted integers). The authors state this explicitly in Theorem 5: the error bound requires "no clipping (Z† = Z)" (Section 4.4). The practical methods SSQR and HPTQ enforce no-clipping by design, but at a cost: SSQR stores overflowed weights as full-precision sparse outliers (adding ~5% storage overhead at the reported 5% outlier rate, Sections 5, D.1), while HPTQ uses variable-length Huffman coding whose average bitwidth includes the cost of encoding infrequent large-magnitude integers (Section D.1).

The consequence: The reported effective bitwidths — 3.125 bits for HPTQ, 2.445–5.725 bits for SSQR depending on outlier rate (Table 3) — are not directly comparable to fixed-bitwidth representations at the same number, because they mix high-precision storage for outliers with low-precision storage for inliers. A system designer choosing between a uniform 4-bit representation and HPTQ at 3.125 bits must account for the fact that HPTQ's bitwidth is an average over a distribution that includes some weights stored at much higher precision (implicitly 16-bit or more for outlier values). The paper's headline perplexity numbers at matched average bitwidths (Table 3: HPTQ 3.125-bit achieves 10.34 WikiText-2 perplexity vs. GPTQ 4.125-bit at 10.10) should be interpreted with this in mind — HPTQ is not strictly "better at fewer bits" but rather "better when bits are allocated non-uniformly," which affects hardware efficiency (irregular memory access patterns, decoding logic for variable-length codes).

What evidence exists in the paper: Table 3 provides the direct comparison: at matched average bitwidth 4.125, HPTQ (9.81 WikiText-2) modestly outperforms GPTQ (10.10), but at 3.125, the gap widens dramatically (HPTQ 10.34 vs. GPTQ 12.77), indicating that the no-clipping benefit grows as bitwidth decreases. However, the CUDA kernel is implemented only for SSQR (Section D.4), not for HPTQ — HPTQ's inference performance with variable-length decoding is not measured, leaving open the question of whether the perplexity gains translate to wall-clock speedups.

Mitigation status: The paper does not attempt to quantify the deployment overhead of variable-bitwidth or sparse representations beyond the SSQR kernel speedup (Figure 4c, approximately 2× end-to-end vs. PyTorch BF16). The Huffman decoding cost for HPTQ is not benchmarked. The SSQR kernel's speedup is shown to decrease as outlier rate increases (Figure 5: "As the outlier rate increases, the speedup diminishes"), confirming the storage-performance tension but without a formal cost model. The paper frames this as a design trade-off rather than a solved problem — "we notice that enforcing no-clipping by simply increasing scales is counterproductive: larger scales enlarge the bound, and the resulting errors can exceed those of a clipped scheme such as MSE" (Section 5) — and the binary search in SSQR and HPTQ is precisely the mechanism that navigates this trade-off, but the trade-off itself remains fundamental.

---

The Error Bound Does Not Cover the Clipped Regime Where Most Deployments Operate

The assumption or constraint: Theorem 5's absolute and relative error bounds are proven under the assumption Z† = Z — the quantization grid is the full integer lattice. The authors state: "the original GPTQ algorithm clips the overflowed integers at the rounding step, introducing large errors that violate the error bound in Theorem 5" (Section 5). This means the theoretical guarantee does not apply to standard GPTQ as it is used in practice (with INT4 or INT3 grids), nor to any quantizer that enforces a bounded integer range.

The consequence: For the vast majority of real-world LLM quantization deployments — which use fixed-bitwidth formats (INT4, INT3, INT2) with bounded representable ranges — the paper's main theoretical result provides no error guarantee. The lattice-theoretic framework explains why clipping is problematic (it severs the CVP equivalence by replacing the lattice with a bounded subset), but it does not provide a bound on how much error clipping introduces, nor guidance on how to choose the quantization range to minimize clipping-induced degradation. This is acknowledged explicitly: "Extending the analysis to clipped grids and exploring (scale-aware) basis reductions are the immediate next steps" (Section 6). Until such an extension exists, practitioners using standard fixed-bitwidth quantization cannot use Theorem 5 to predict or bound their layer-wise errors.

What evidence exists in the paper: The experimental results indirectly demonstrate the severity of this gap: standard GPTQ at 3.125 bits achieves 12.77 WikiText-2 perplexity versus HPTQ's 10.34 at the same effective bitwidth (Table 3) — a 2.43-point gap that grows to 43.54 points at 2.125 bits (57.51 vs. 13.97). This gap quantifies the cost of clipping, but the paper provides no theoretical prediction of its magnitude. The clipped GPTQ's error behavior is essentially outside the theory — the paper can only observe it empirically, not bound it.

Mitigation status: Not addressed. The paper explicitly flags this as future work (Section 6: "Extending the analysis to clipped grids"). The SSQR and HPTQ methods are workarounds that sidestep the limitation rather than solving it — they change the representation to avoid clipping rather than providing guarantees for clipped quantization. For practitioners who must use fixed-bitwidth formats (e.g., for hardware compatibility with existing INT4 kernels), this limitation means the paper's theory offers diagnostic insight ("clipping is the problem") but no prescriptive solution ("here is how to set the clipping threshold optimally").

---

Difficulty Estimation Overhead Is Not Accounted for in the Efficiency Claims

The assumption or constraint: The compute-optimal scaling strategy selects hyperparameters (search algorithm, beam width, sequential-to-parallel ratio) based on estimated prompt difficulty, which the paper computes by generating 2048 samples per question, scoring them with the PRM, and binning into quintiles based on the average score (Section 3.2). The authors acknowledge: "our experiments do not account for this cost largely for simplicity" and describe the difficulty estimation approach as incurring "additional computation cost during inference" (Section 3.2).

The consequence: The paper's headline efficiency claim — "more than 4× better efficiency over a standard best-of-N baseline" — is computed after difficulty is known, without amortizing the cost of learning it. The 2048 samples used for difficulty estimation exceed the largest test-time budgets studied in the search experiments (256 generations) by 8×, and even the largest revision budget (512 generations) by 4×. In a deployment setting, the total cost would be difficulty estimation + strategy execution, and for any single query, the former would dominate the latter, likely eliminating the reported efficiency gain entirely. Only in a scenario where the difficulty estimate can be computed once and reused across many evaluations of the same question (e.g., repeated benchmarking, self-improvement pipelines with fixed training sets) does the amortized cost become negligible.

What evidence exists in the paper: The paper explicitly states the cost is not accounted for (Section 3.2) and frames difficulty estimation as an exploration-exploitation tradeoff — "compute spent assessing difficulty versus compute spent solving the problem" — but provides no experimental quantification of how much the 4× efficiency gain shrinks when the estimation overhead is included. The comparison between oracle bins (which use ground-truth correctness to compute difficulty) and predicted bins (which use the PRM's average score without ground-truth labels) shows the two "largely overlap" (Figure 4), but this only speaks to the accuracy of predicted difficulty relative to oracle difficulty — it does not address the cost of making the prediction.

Mitigation status: Not addressed experimentally. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), which would replace the 2048-sample estimation with a single forward pass, but no such model is developed or evaluated. An adaptive scheme that interleaves difficulty estimation with solution attempts is also mentioned but not tested. Until one of these approaches is realized, the 4× figure must be understood as an upper bound on achievable efficiency conditioned on free difficulty information, not a realized deployment gain.

---

A Single Benchmark and Model Family Cannot Establish Generality of Difficulty-Dependent Patterns

The assumption or constraint: All experiments use the MATH benchmark (500 test questions) and PaLM 2-S* as the base model (Section 4). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is asserted, not tested. MATH consists exclusively of competition-level math problems requiring symbolic reasoning with ground-truth answers that can be checked via string matching.

The consequence: The paper's central empirical finding — that test-time compute efficacy depends critically on problem difficulty, with different strategies optimal at different difficulty levels — may not generalize to other domains or model families. Several aspects of the findings could be model-specific or benchmark-specific:

• Model-specific PRM over-optimization thresholds: The difficulty level at which beam search transitions from harmful (over-optimization) to helpful (genuine guidance) depends on the PRM's calibration properties, which in turn depend on PaLM 2-S*'s output distribution. A model with different error patterns or different calibration might exhibit the cross-over at different difficulty levels, or might not exhibit over-optimization at all on MATH.
• Revision model capability: The revision model's ability to improve through sequential revisions (Figure 6, left) depends on the base model's in-context learning and self-correction capabilities, which vary substantially across model families. The finding that revisions help most on easy problems and that a balanced sequential-parallel ratio is optimal on medium problems may not hold for models with stronger or weaker self-correction abilities.
• Benchmark specificity: MATH problems have clean correctness signals (exact string match against ground-truth answers) that enable both the PRM training pipeline (Monte Carlo rollouts with ground-truth correctness labels) and difficulty estimation (pass@1 computed via answer matching). Tasks without clean correctness signals — open-ended generation, dialogue, creative writing — would require fundamentally different verifier and difficulty estimation approaches.

What evidence exists in the paper: None that addresses generalization. The paper does not include experiments on any benchmark other than MATH, nor any model other than PaLM 2-S*. The FLOPs-matched comparison uses a 14× larger model from the same PaLM 2 family, so even the pretraining-inference tradeoff is tested only within a single model family. The difficulty-dependent patterns (Figures 3 right, 7 right) are internally consistent and robust within the MATH/PaLM 2-S* setting, but their external validity is untested.

Mitigation status: Not addressed. The paper does not flag this as a limitation or suggest replication on other benchmarks and models. Given the paper's stated goal of providing a general framework for test-time compute allocation, this is a significant gap — the framework's practical value depends on whether the difficulty-dependent strategy selection generalizes beyond the specific model-benchmark pair tested.

---

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Favoring Test-Time Compute

The assumption or constraint: The FLOPs-matched comparison in Section 7 scales model parameters by 14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), which would scale both parameters and data proportionally. The authors acknowledge: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7). Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search of any kind.

The consequence: The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on medium-difficulty questions at R ≪ 1 for revisions (Figure 1, top-right) — are measured against a baseline that is likely weaker than what a compute-optimally trained larger model would achieve. A Chinchilla-optimal 14× larger model (trained on more data) would typically perform better than a parameter-only-scaled model at the same total FLOPs, which means the pretraining baseline is weaker than it should be. Similarly, giving the larger model even a modest test-time compute budget (best-of-8, for instance) would create a much stronger baseline, since the paper's entire thesis is that test-time compute is valuable. The current comparison answers "can test-time compute beat a larger model with greedy decoding?" — an interesting question, but not the most relevant one for practitioners deciding how to allocate a total compute budget, who would naturally consider giving some test-time compute to the larger model as well.

What evidence exists in the paper: None that addresses this directly. The paper is transparent about the parameter-only scaling choice (Section 7, quoted above), so the reader is informed, but no ablation or alternative baseline is provided. The FLOPs-matched bar charts in Figure 1 prominently display the 14× larger model as "Pretraining" with a single bar, without showing how a compute-optimally trained larger model or a larger model with test-time augmentation would perform.

Mitigation status: Partially acknowledged. The paper explicitly leaves Chinchilla-optimal pretraining comparisons to future work. However, the greedy-only decoding for the larger model is not similarly acknowledged — the paper presents the 14× model as a fixed baseline without discussing that giving it test-time compute would be a fairer comparison. For a paper whose core contribution is demonstrating the value of test-time compute, the choice to deny that compute to the larger baseline is a significant asymmetry that weakens the strength of the trade-off conclusions, particularly at high R where the smaller model's advantage narrows or reverses.

---

Seed-Based Heuristic for Quantization Order Incurs Cubic Overhead With Modest Gains

The assumption or constraint: The paper's min-pivot heuristic (Algorithm 3) computes the LDL pivot order by greedily selecting the minimum diagonal entry at each decomposition step, requiring an additional O(c³) pass over the Hessian matrix beyond the LDL decomposition already performed by GPTQ (Section 4.5, Appendix C.3). The authors report that min-pivot consistently reduces tr(D) relative to act-order (Table 2: e.g., Q·K·V layer trace drops from 7.400 × 10⁷ to 7.323 × 10⁷, a ~1% reduction), but acknowledge that "the downstream accuracy gains are modest" (Section 4.5).

The consequence: For the vast majority of deployment scenarios, the extra O(c³) computation to compute the min-pivot order is unlikely to be justified by the small accuracy improvement it provides over the simpler act-order heuristic (which requires only sorting the Hessian diagonal, an O(c log c) operation). The paper states that act-order "already captures most of the benefit when the Hessian matrix is well-conditioned" (Section 4.5), which appears to be the common case for transformer layers based on the empirical results (Table 2 shows act-order is within ~1% of min-pivot). This means the practical value of the theoretical insight about pivot order optimization is limited — it explains why act-order works rather than providing a replacement that practitioners should adopt.

What evidence exists in the paper: Table 2 (Appendix C.3) reports tr(D) for five quantization orders across four layer types in Qwen3-8B block 18. Min-pivot is consistently best, act-order is a close second, and random/fixed orders are substantially worse. However, the paper does not report downstream perplexity or accuracy differences between min-pivot and act-order — the comparison is only on the proxy metric tr(D). The statement that "downstream accuracy gains are modest" is qualitative, not quantitative, leaving the reader to guess whether "modest" means 0.1%, 1%, or something in between. The cubic overhead of min-pivot is not formalized or benchmarked — the paper notes it "does not increase the overall time complexity of quantization" only in the asymptotic sense that GPTQ is already O(c³), but the constant factor could be substantial.

Mitigation status: Partially addressed by the recommendation itself — the paper suggests using "act-order as a cheap option and reserving min-pivot for cases where a tighter bound is required" (Section 4.5, Appendix C.3). This is a pragmatic acknowledgment that the theoretical improvement may not be worth the computational cost in most cases. However, no guidance is provided on how to identify the cases where min-pivot would matter — are there layer types, model scales, or Hessian conditioning numbers that predict when act-order's approximation is insufficient? Without such guidance, the recommendation to use min-pivot "when a tighter bound is required" is difficult to operationalize.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper performs a conceptual reframing rather than a paradigm shift: it does not introduce a new quantization algorithm that replaces GPTQ, but it fundamentally changes how the field should think about GPTQ and second-order quantization methods more broadly. The magnitude is significant for theory but incremental for practice — the downstream accuracy improvements from the proposed methods (HPTQ, SSQR) are real but modest relative to the current state of the art, while the intellectual contribution of connecting GPTQ to Babai's nearest plane algorithm opens an entirely new research channel between LLM compression and lattice theory.

What changes conceptually: Before this work, GPTQ was understood as a pragmatic engineering optimization — a way to make OBQ's optimal error propagation computationally tractable by fixing the quantization order and sharing the inverse Hessian across output channels. The theoretical question "why does this greedy local procedure work globally?" had no satisfying answer. The paper resolves this by proving that GPTQ is not an approximation or a heuristic simplification of something optimal — it is exactly Babai's nearest plane algorithm, a polynomial-time CVP solver with known approximation guarantees, applied to the lattice induced by the layer's input Hessian. The greedy error propagation that seemed ad-hoc is revealed to be Gram-Schmidt orthogonal projection, with the crucial property that errors from successive steps are mutually orthogonal and therefore add in Euclidean norm rather than compounding unpredictably. This explains the global effectiveness of a procedure that, on its face, had no reason to stay tightly bounded across hundreds of sequential greedy steps.

What changes methodologically: The paper's quantization-CVP dictionary (Table 1) provides a Rosetta Stone for translating between neural network compression concepts and lattice theory concepts. This means that researchers designing future quantizers can now ask "which lattice algorithm should we apply?" rather than "what heuristic weight update should we try next?" The decades of algorithmic development on the CVP — basis reduction (LLL, BKZ), alternative CVP solvers, analyses of approximation ratios, and understanding of how basis geometry affects solution quality — become directly applicable to LLM compression. The paper explicitly frames this as a "two-way channel" (Section 6): techniques from lattice algorithms can improve practical quantizers, while the massive-scale Hessian matrices encountered in transformer layers may inspire new questions for lattice theorists.

What contradictions this resolves: The paper reconciles a tension in the literature between the empirical success of GPTQ and the absence of any theoretical justification. Prior work (Chee et al., 2023, QuIP) had proven error guarantees for a specific variant of GPTQ, but these worked within the algebraic weight-space framework without revealing the geometric structure. Practitioners were left with an uncomfortable situation: GPTQ worked well enough to become the standard one-shot LLM quantizer, but no one could explain why it stayed within tight error bounds, or predict when it would fail. The paper resolves this by showing that the bounds are not mysterious — they are exactly Babai's bounds, inherited from a well-understood lattice algorithm. The "when it fails" question is partially answered: GPTQ degrades sharply at low bitwidths because weight clipping severs the CVP equivalence and voids the bound, not because the algorithm itself is fundamentally limited.

Which research directions become more attractive:

• Lattice basis reduction for quantization becomes the most natural first step. If Babai's algorithm with a reduced basis (LLL or BKZ) provides tighter error bounds, then applying basis reduction to the Hessian lattice of each transformer layer could directly improve quantization accuracy at fixed bitwidth. The paper identifies the key obstacle — basis reduction is scale-sensitive, requiring per-channel transformations that break batched computation — which makes this a well-defined algorithmic challenge rather than a vague "future work" item.

• Verifier / PRM robustness becomes the central bottleneck for test-time compute scaling, redirecting effort away from more sophisticated search algorithms. The paper's finding that lookahead search paradoxically performs worst at matched budgets (Figure 3, left) and that beam search over-optimizes on easy problems (Figure 3, right) establishes that better verifiers, not better search, will unlock the next generation of inference-time improvements. The over-optimization phenomenon is analogous to reward hacking in RLHF, and the same countermeasures (adversarial training, ensembles, KL penalties) are now natural to explore.

• Clipping-aware lattice theory becomes an explicit open problem. The paper's error bound holds only in the no-clipping regime, and the experimental results show that clipping is the primary cause of GPTQ's degradation at low bitwidths (Table 3: standard GPTQ at 3.125 bits achieves 12.77 WikiText-2 perplexity vs. HPTQ's 10.34). Extending the CVP analysis to bounded integer grids — a CVP with box constraints — is identified as "the immediate next step" (Section 6), and solving it would provide the first theoretical guarantees for standard fixed-bitwidth quantization.

Which research directions become less attractive:

• Developing more complex search algorithms for test-time compute (Monte Carlo Tree Search variants, dynamic beam pruning, A*-style heuristics) is implicitly de-emphasized. The paper shows that lookahead search — the most sophisticated method tested — underperforms simpler alternatives at matched budgets, and that compute-optimal allocation often routes problems away from aggressive optimization altogether (using best-of-N on easy problems where beam search over-optimizes). The message is clear: search algorithm design hits a verifier-quality ceiling before it hits a search-quality ceiling.

• Treating quantization order as a hyperparameter to tune empirically is replaced by the principled criterion of minimizing tr(D), the trace of the LDL decomposition's diagonal matrix. The paper's comparison of act-order and min-pivot (Table 2) shows that the existing act-order heuristic already captures most of the available benefit when the Hessian is well-conditioned, so extensive empirical search over orderings is unlikely to yield large gains beyond what min-pivot provides.

---

Follow-Up Research This Work Enables

Scale-aware basis reduction for batched quantization. The paper identifies LLL basis reduction as the obvious next step for tightening Babai's error bound, but notes a fundamental obstacle: LLL is scale-sensitive, producing different transformation matrices T^(i) for different per-channel scale vectors s_i. This prevents the batched computation that makes GPTQ efficient (all output channels share the same LDL decomposition). A concrete follow-up would develop a scale-agnostic basis reduction — perhaps by reducing the basis of the unscaled Hessian X^T X and then analytically deriving how per-channel scaling affects the reduced basis geometry — or a shared approximate reduction that finds a single transformation T that is near-optimal for all channels simultaneously. The evaluation would compare tr(D) and downstream perplexity for LLL-reduced vs. unreduced bases across multiple model scales, measuring whether the reduction's O(c^4) cost can be amortized across the r output channels. A negative result — that basis reduction helps in theory but the scale-sensitivity makes it impractical for batched LLM quantization — would be equally valuable, establishing a fundamental tension between lattice-theoretic optimality and batched computational efficiency.

Clipping-aware CVP bounds for fixed-bitwidth quantization. The paper's Theorem 5 error bound requires Z† = Z (unrestricted integers), but real deployments use bounded grids like INT4 ({-8, ..., 7}). A direct extension would formulate the problem as a bounded CVP — minimize ||B z - y|| subject to z ∈ [-2^(b-1), 2^(b-1)-1]^c — and analyze how Babai's nearest plane algorithm behaves when the nearest hyperplane may lie outside the feasible region. The key question is whether there exists a modified projection step that accounts for the box constraints while preserving some fraction of Babai's approximation guarantee. A strong follow-up would derive error bounds (even loose ones) for the clipped regime, then use those bounds to predict — from the Hessian geometry alone — the bitwidth at which a given layer will experience clipping-induced degradation exceeding some threshold. This would replace the current trial-and-error calibration of quantization ranges with a principled criterion. The experimental validation would compare the predicted degradation threshold against actual perplexity curves like those in Table 3, testing whether the theory can forecast the sharp drop in GPTQ performance between 3.125 and 2.125 bits.

Cheap difficulty estimation via learned predictors. The paper's compute-optimal test-time scaling achieves 4× efficiency gains but relies on difficulty estimation that costs 2048 sample generations per question — far more than the test-time budgets being optimized. The paper explicitly calls for "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). A concrete implementation would train a lightweight classifier — perhaps a small linear probe on top of the base model's last hidden state, or a distilled model that takes only the question text as input — to predict the difficulty quintile directly, using the PRM's average score on 2048 samples as the training target (essentially distilling the expensive difficulty oracle into a cheap predictor). The evaluation would measure (a) the classifier's accuracy at predicting the correct difficulty bin, (b) the compute-optimal scaling curve's performance when using the classifier's bins versus oracle bins, and (c) the net compute savings when the classifier's inference cost is amortized into the budget. Even a moderately accurate classifier (say, 70% bin accuracy) might recover most of the compute-optimal gains if its errors are conservative (overestimating difficulty rather than underestimating it, which would lead to overspending on easy problems).

Combined PRM search with a revision model proposal distribution. The paper studies search (Section 5) and revisions (Section 6) independently, but acknowledges they were never combined (Section 8). A natural integration would use the revision model as the proposal distribution within beam search: at each step of the search tree, instead of sampling from the base model, condition the revision model on the partial solution and any rejected branches as context, potentially generating higher-quality candidate steps. Alternatively, use the PRM to guide which revisions to pursue: rather than generating a long revision chain unconditionally, use the PRM's per-step scores to decide when a revision is improving the solution (continue the chain) versus when it is making things worse (restart from a previous checkpoint). The evaluation would compare the combined approach against (a) PRM search alone, (b) revisions alone, and (c) the compute-optimal policy that selects between them but never combines them, on the MATH benchmark with the same PaLM 2-S* base model. The hypothesis is that the mechanisms have complementary strengths — revisions improve the quality of generated candidates (proposal distribution) while search improves candidate selection (verifier) — and that combining them could break through the performance ceilings that each individually hits (search degrades from verifier over-optimization at high budgets; revisions plateau as the model saturates its self-correction ability).

Dynamic difficulty assessment and mid-computation strategy switching. The paper's difficulty estimation is computed once, before any solution attempt, and the strategy is fixed for the entire budget. A more sophisticated approach would interleave difficulty assessment with the solution process: start by generating a small number of parallel samples (say, 4–8), use the PRM's score distribution on those samples as a quick difficulty signal, and then allocate the remaining budget dynamically — switching to beam search if the problem appears medium-hard, continuing parallel sampling if it appears easy, or aborting test-time compute entirely and routing to a larger model if it appears extremely hard. This is a multi-armed bandit or Bayesian optimization problem, where the difficulty signal is acquired gradually and the budget allocation adapts in real time. The evaluation would compare the dynamic policy against the static compute-optimal policy (with pre-computed difficulty) on the same MATH benchmark, measuring whether the dynamic policy can approach the static policy's performance while amortizing difficulty estimation cost into the solution budget. A key metric is the regret — how much accuracy is lost by not knowing difficulty in advance — as a function of the initial exploration budget (the number of samples used for difficulty assessment before committing to a strategy).

Verifier robustness interventions for test-time search scaling. The paper identifies verifier over-optimization as the primary bottleneck preventing unbounded improvements from additional search compute, but tests no interventions to mitigate it. Concrete follow-ups include: (a) adversarial PRM training, where the PRM is fine-tuned on solutions generated by beam search (which tend to exploit the PRM's weaknesses) rather than only on i.i.d. sampled solutions, closing the distribution shift between the PRM's training data and its deployment conditions; (b) ensemble verification, where multiple independently trained PRMs are aggregated (by averaging or voting) to produce more robust scores, with the hypothesis that different PRMs overfit to different spurious patterns and their consensus is harder to exploit; (c) KL-penalized search, where beam search is modified to penalize solutions whose token distribution diverges too far from the base model's unconditional distribution, preventing the search from drifting into degenerate regions of output space that score highly under the PRM but are nonsensical (as documented in Appendix M). The evaluation would measure whether any of these interventions allows beam search to continue improving at budgets where standard beam search plateaus or degrades in Figure 3 (right), with particular attention to the easy-problem bins where over-optimization is most severe.

---

Practical Applications and Downstream Use Cases

Low-cost edge deployment of LLMs on consumer GPUs. The SSQR kernel achieves approximately 2× end-to-end inference speedup versus PyTorch BF16 on an NVIDIA RTX A6000 (Ampere) at batch size 1 with 128 generated tokens (Figure 4c), across inlier bitwidths of 2–4 bits and outlier rates of 0–5%. This is directly applicable to scenarios where a Qwen3-8B-class model needs to run on a single consumer GPU for local inference (coding assistants, on-device chatbots, privacy-sensitive applications). The 2× speedup translates to halving the time-per-output-token, which is the user-facing latency metric in interactive applications. The SSQR representation stores 1–5% of weights as sparse FP16 outliers with the remainder at 2–4 bits, achieving effective bitwidths of 2.445–5.725 bits depending on the outlier rate (Table 3). A practitioner choosing SSQR-2% at 3.765 effective bits gets 10.57 WikiText-2 perplexity (within 0.84 points of the BF16 baseline at 9.73) while roughly doubling inference throughput — a favorable accuracy-efficiency tradeoff for deployment scenarios where the model's capabilities are adequate and latency is the binding constraint.

Storage-constrained model serving with HPTQ. HPTQ at 3.125 bits achieves 10.34 WikiText-2 perplexity on Qwen3-8B (Table 3), within 0.61 points of the BF16 baseline while requiring approximately 5.1× less storage for weights (16 bits / 3.125 bits ≈ 5.1×). This is directly applicable to model serving scenarios where GPU memory capacity is the binding constraint — e.g., serving multiple model instances on a single GPU, or fitting a model that would otherwise exceed VRAM limits. The Pareto-optimality of 3.125 bits (Figure 4b: this bitwidth lies at the knee of the perplexity-vs-compression curve across model sizes from 0.6B to 14B) means that HPTQ at this bitwidth provides the best compression for a given perplexity budget, or equivalently the best perplexity for a given compression budget. A serving provider could use HPTQ to fit twice as many model instances on the same hardware compared to 8-bit quantization, or to serve a 14B model on a GPU that could otherwise only fit an 8B model, with minimal accuracy degradation (HPTQ 3.125-bit Qwen3-14B: 9.06 WikiText-2 perplexity vs. 8.65 BF16, a 0.41-point gap; Table 4).

Verifier training for self-improvement and data generation pipelines. The paper's Monte Carlo rollout PRM training procedure (Section 3.4 in the original summary; referenced in this paper's context via the GPTQ equivalence) provides a human-label-free recipe for training process reward models that can score intermediate solution steps. This is directly applicable to self-improvement pipelines (STaR, ReST^EM-style iterative training) where an LLM generates solutions, a verifier scores them, and the best solutions are used to fine-tune the next model iteration. The key practical finding is that the PRM outperforms an ORM even when using last-step aggregation (which effectively reduces the PRM to an outcome-level predictor at decision time), because the step-level training provides beneficial representation learning (Appendix E, Figure 13: PRM with "last" aggregation achieves ~37% vs. ORM's ~34% at 256 samples). A practitioner building a self-improvement loop can therefore train a PRM using Monte Carlo rollouts — which requires only sampling from the base model and checking final-answer correctness on a training set, no human annotation — and use it both for solution selection and as a difficulty estimator (since the PRM's average final-answer score predicts problem difficulty, enabling the compute-optimal allocation that the paper shows is 4× more efficient than uniform best-of-N).

---

When to Prefer This Method

The paper does not frame itself as proposing a single method that should be preferred over named alternatives. Rather, it provides a theoretical framework (GPTQ-as-Babai) and two practical instantiations (SSQR, HPTQ) that enforce no-clipping to preserve the lattice-theoretic guarantees. The implicit comparison is between clipped GPTQ (the status quo) and no-clipping variants (the paper's proposal). The decision is not "use SSQR vs. use HPTQ vs. use something else" but rather "under what conditions does avoiding clipping matter enough to justify the representation overhead?" Based on the paper's evidence:

• Prefer no-clipping quantization (SSQR or HPTQ) when pushing below 4 bits per weight, where standard GPTQ with clipping degrades sharply (Table 3: GPTQ 3.125-bit achieves 12.77 WikiText-2 perplexity vs. HPTQ's 10.34, and GPTQ 2.125-bit collapses to 57.51 vs. HPTQ's 13.97). The no-clipping guarantee becomes increasingly valuable as bitwidth decreases because overflow events become more frequent and clipping errors dominate.

• Prefer standard (clipped) GPTQ when operating at 4 bits or above with well-calibrated scales, where clipping events are rare and the representation overhead of SSQR's sparse outliers or HPTQ's variable-length coding is not justified by accuracy gains. At 4.125 bits, HPTQ (9.81 WikiText-2) and clipped GPTQ (10.10) differ by only 0.29 points (Table 3), and the hardware simplicity of fixed-bitwidth formats (existing INT4 kernels, no sparse matmul, no Huffman decoding) may outweigh this small gap.

• Prefer SSQR over HPTQ when hardware compatibility with existing low-bitwidth matrix multiplication kernels is required and outlier rates below 2% are acceptable. SSQR's representation decomposes into a dense low-bitwidth matmul (handled by the custom CUDA kernel in Section D.4) plus a sparse outlier matmul, both of which have efficient implementations. HPTQ's Huffman-coded integers require variable-length decoding that is not benchmarked in the paper, making its wall-clock inference performance unknown.

• Prefer HPTQ over SSQR when compression ratio is the primary objective and inference latency is not the binding constraint (e.g., offline batch evaluation, storage-constrained serving where model weights are decompressed once and cached). HPTQ at 3.125 bits achieves Pareto-optimal perplexity (Figure 4b) without the tuning complexity of choosing an outlier rate — the binary search over scale is fully automated and converges on the bitwidth budget directly.