Inference Optimization2023~100 minarXiv:2308.13137

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

score 9Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang +7↑ 19★ 890Aug 25, 2023cs.LGhttps://github.com/opengvlab/omniquant

🎯 Pitch

OmniQuant matches the accuracy of expensive quantization-aware training while needing only 128 samples and a few GPU hours—reducing LLaMA‑13B W2A16 perplexity from GPTQ’s 5.5e3 down to 13.21. It achieves this by learning clipping thresholds and mathematically equivalent transformations that shift activation outliers into weights, all within a differentiable PTQ framework.

1. Executive Summary

This paper introduces OmniQuant, a differentiable post-training quantization technique for large language models that achieves quantization-aware training-level accuracy while maintaining PTQ-level time and data efficiency. Evaluated on the LLaMA, LLaMA-2, OPT, and Falcon model families across perplexity benchmarks (WikiText2, PTB, C4) and zero-shot tasks, OmniQuant combines two learnable mechanisms — Learnable Weight Clipping (LWC), which optimizes per-channel clipping thresholds to reduce weight quantization error (e.g., adapting the fraction of outlier values clipped via learned scaling strengths γ and β), and Learnable Equivalent Transformation (LET), which shifts quantization difficulty from outlier-heavy activations to better-behaved weights through gradient-optimized channel-wise scaling and shifting (e.g., learning the migration strength that SmoothQuant hand-crafts). OmniQuant delivers consistent improvements over prior PTQ methods across weight-only (W2A16 through W4A16) and weight-activation (W4A4, W6A6) configurations — for instance, reducing WikiText2 perplexity on LLaMA-13B W2A16 from 5.5e3 (GPTQ) to 13.21, and improving W4A4 zero-shot accuracy on LLaMA-7B by +4.99% to +11.80% across model sizes — while requiring only 128 calibration samples and 1–16 hours on a single A100-40G GPU, establishing that gradient-based optimization of a small set of quantization parameters can match QAT performance only when the learned parameters are designed to operate within the reduced solution space of relative clipping strengths and mathematically equivalent transformations rather than through direct threshold or step-size learning.

2. Context and Motivation

The Core Problem: Low-Bit Quantization Remains Brittle Despite Its Importance

The paper addresses a specific and pressing gap in LLM deployment: existing post-training quantization (PTQ) methods perform poorly in extremely low-bit settings (W2A16, W4A4), precisely where the memory and compute savings are most needed. While quantization has emerged as a promising technique to reduce both the memory footprint and computational cost of LLMs — a GPT-3 model requires 350GB of memory just to load its FP16 parameters, demanding at least five A100-80G GPUs for inference — the practical benefits of quantization are critically dependent on how low the bit-width can go.

The stakes are clear: weight-only quantization (e.g., W4A16) reduces memory consumption but leaves activations in full precision, meaning matrix multiplications still operate on 16-bit values. Weight-activation quantization (e.g., W4A4) quantizes both operands, enabling low-bit matrix multiplication that can substantially accelerate inference. However, as Figure 1(b,c) demonstrates, the performance of existing PTQ methods collapses when pushed to these aggressive regimes. On LLaMA-13B W2A16, GPTQ produces a WikiText2 perplexity of 5,500 (compared to 5.09 for FP16), while AWQ reaches 280,000. These are not marginal degradations — they represent models that have essentially lost the ability to model language. The paper's central question, stated explicitly in the introduction, is:

"can we attain the performance of QAT, while maintaining the time and data efficiency of PTQ?"

This framing reveals the fundamental tension: Quantization-Aware Training (QAT) can achieve good accuracy at low bit-widths by training the full model with quantization simulated during optimization, but at prohibitive cost — LLM-QAT requires 100k training samples and hundreds of GPU hours. PTQ methods are cheap (GPTQ quantizes LLaMA-13B in an hour with 128 samples) but produce unusable models at low bit-widths. The gap between these two regimes is what OmniQuant aims to close.

Why This Matters: The Deployment Reality

The importance of this problem extends beyond academic benchmarks. There are several practical dimensions:

Memory constraints on consumer hardware. An FP16 LLaMA-13B requires approximately 24.3GB just for weight storage (Table 3). With W4A16g128 quantization, this drops to 7.0GB — enabling deployment on a single consumer GPU with 8-12GB of VRAM. With W2A16g128, it drops further to 4.0GB. However, these memory savings are meaningless if the quantized model's outputs are incoherent. The paper's results on W2A16g128 (Table 1) — where OmniQuant achieves 7.93 perplexity on LLaMA-13B versus 15.60 for GPTQ and 2.8e5 for AWQ — demonstrate that useful 2-bit inference is possible, but only with a method that properly handles the quantization error.

Throughput gains are equally dependent on low-bit success. Table 3 shows that W4A16g128 quantization nearly doubles inference speed on LLaMA-7B (134.2 tokens/sec vs. 69.2 for FP16). But moving from W4A16 to W3A16 or W2A16 does not necessarily improve speed if the quantization method introduces overhead (e.g., mixed-precision storage, sparse representations). The paper emphasizes that OmniQuant produces pure uniform integer quantized models with no extra operations — the clipping thresholds and equivalent transformation factors are fused into the quantized weights — making it directly compatible with deployment frameworks like MLC-LLM. This architectural cleanliness is a practical advantage over methods like SpQR or SqueezeLLM, which employ mixed-precision or non-uniform quantization that complicates hardware implementation.

The scaling behavior with model size amplifies the need. As models grow from 7B to 70B to 180B parameters, the absolute memory savings from aggressive quantization scale proportionally. The paper tests Falcon-180B (Table A18), where W3A16g512 quantization reduces the model from requiring 5×A100-80G GPUs (335GB) to fitting on a single A100-80G (65GB). At this scale, the difference between a working and non-working quantization method determines whether the model can be deployed at all on a given hardware budget.

Where Prior Approaches Fall Short

The paper identifies two broad families of existing quantization methods and characterizes their specific failure modes:

Training-free PTQ methods rely on hand-crafted heuristics that fail at low bit-widths. The dominant approaches for LLM quantization — GPTQ, AWQ, SmoothQuant, Outlier Suppression+ — all share a common limitation: they determine quantization parameters (clipping thresholds, migration strengths, scaling factors) through pre-defined rules or grid search rather than through gradient-based optimization. The paper is explicit about this critique:

"they usually exhibit significant performance degradation when confronted with low-bit quantization... This performance shortfall in low-bit quantization can be attributed to the fact that these methods primarily rely on handcrafted quantization parameters such as migration strength and scaling parameters, which often leads to lower performance."

The failure manifests differently across methods but stems from the same root cause. SmoothQuant uses a fixed migration strength α to shift quantization difficulty from activations to weights via channel-wise scaling. AWQ performs grid search over scaling factors using a heuristic proxy based on activation magnitudes. Outlier Suppression+ adds channel-wise shifting with grid-searched scaling but pre-defined shifting parameters. These hand-crafted approaches cannot adapt to the complex interactions between weight and activation distributions that emerge at extremely low bit-widths — interactions that become critical precisely because the quantization error budget is so small.

Quantization-Aware Training is accurate but impractical for LLMs. QAT methods like LLM-QAT simulate quantization during training and optimize the full model weights accordingly. This produces strong results — LLM-QAT achieves 46.43% average zero-shot accuracy on LLaMA-7B W4A4 (Table 2) — but the training cost is enormous: 100k samples and hundreds of GPU hours. The paper positions this as a non-starter for most practitioners:

"it is not practical due to the high training cost because the whole model is trained with the awareness of the quantization process."

The key insight is that QAT optimizes too much. It tunes all model weights, treating quantization as an integral part of training. OmniQuant's central hypothesis is that most of this optimization is unnecessary — that a small set of strategically chosen learnable parameters, optimized on a tiny calibration set, can achieve comparable results.

Weight-only and weight-activation quantization have been treated as separate problems. The paper observes that existing methods specialize in one regime or the other. GPTQ and AWQ focus on weight-only quantization, using techniques like optimal brain compression or activation-aware scaling. SmoothQuant and Outlier Suppression+ address weight-activation quantization by managing activation outliers. No prior PTQ method provides a unified framework that works well for both — a gap that OmniQuant fills by design, since LWC handles weight quantization difficulty and LET handles activation outliers, and both are optimized jointly under the same block-wise error minimization objective.

The specific challenge of activation outliers is only partially solved. The paper identifies activation outliers — channels where activation magnitudes are systematically much larger than others — as "the main difficulty" in weight-activation quantization. SmoothQuant addresses this by migrating the quantization difficulty from activations to weights using channel-wise scaling with a pre-defined strength. But as Figure A2 shows visually, SmoothQuant reduces the outlier-to-regular channel ratio from 70× to about 2×, leaving a residual disparity that still causes quantization error. Outlier Suppression+ improves on this with additional channel-wise shifting, but the shifting parameters are pre-defined rather than learned. The paper's LET addresses this gap by making both scaling and shifting learnable through gradient descent, achieving nearly uniform activation magnitudes across channels (Figure A2c).

How This Paper Positions Itself

OmniQuant positions itself at the intersection of two methodological lineages — PTQ efficiency and QAT accuracy — by introducing a third category: differentiable PTQ. The paper is explicit that prior PTQ methods with gradient optimization exist (AdaRound, BRECQ), but they optimize over the entire weight space and are therefore intractable for billion-parameter models:

"Previous PTQ methods with gradient optimization, such as AdaRound, BRECQ cannot be applied in models with billions of parameters because they are hard to optimize due to the huge solution space."

OmniQuant's key design choice is to freeze the original full-precision weights and introduce only a "restrained set of learnable parameters" — the clipping strengths (γ, β) in LWC and the scaling/shifting factors (s, δ) and attention scaling (sₐ) in LET. This dramatically reduces the optimization problem from billions of parameters to a few thousand, making block-wise gradient descent feasible on a single GPU with minimal data.

The paper positions this as a fundamentally different optimization regime than QAT. QAT tunes all weights to accommodate quantization, which requires large-scale training data to avoid overfitting and maintain generalization. OmniQuant tunes only the quantization parameters themselves — how the weights are clipped, how the activations are transformed — which is a much more constrained optimization that converges quickly on small calibration sets (as few as 16 samples, per Table A11).

This design echoes the philosophy of parameter-efficient fine-tuning (e.g., LoRA, adapters) but applied to the quantization problem: rather than retraining the model, learn a small set of transformations that make the frozen model amenable to compression. The paper explicitly notes complementarity with QLoRA and INT2.1, which fine-tune quantized models after quantization — OmniQuant improves the quantization process itself, meaning the two approaches could be combined.

The unified framework claim is substantiated by the architecture. Because LWC handles weight quantization difficulty and LET handles activation outliers through mathematically equivalent transformations, the same framework applies to both weight-only and weight-activation quantization. The only difference is that weight-only quantization may disable LET when it provides negligible benefit (as observed for LLaMA in Table A3). This contrasts with prior work, where SmoothQuant/AWQ/Outlier Suppression+ each target a specific quantization regime with specialized mechanisms.

The block-wise optimization strategy is positioned as enabling the practicality of the approach. By quantizing one transformer block at a time — minimizing the L2 distance between the full-precision block's output and the quantized block's output on the calibration data — OmniQuant avoids the memory explosion that would come from backpropagating through the entire model. Each block's optimization is independent: the calibration data flows through all preceding (already-quantized) blocks, the current block's quantization parameters are optimized to match the full-precision block's output, and then the block is frozen and the process moves to the next block. This sequential, block-wise approach is what enables quantizing a 70B model on a single A100-40G GPU.

3. Technical Approach

3.1 Reader Orientation

OmniQuant is a differentiable post-training quantization system that converts a full-precision LLM into a low-bit quantized version by learning a small set of quantization parameters rather than by hand-crafting them or retraining the entire model. It solves the problem that existing PTQ methods produce unusably inaccurate models at extremely low bit-widths (W2A16, W4A4) because their hand-crafted quantization parameters cannot adapt to the complex interactions between weight and activation distributions that emerge when the error budget is severely constrained — OmniQuant addresses this by freezing the original model weights and instead learning optimal clipping thresholds and mathematically equivalent activation transformations through block-wise gradient descent on a tiny calibration set.

3.2 Big-Picture Architecture (Diagram in Words)

OmniQuant's system has five major components, organized in a sequential block-wise pipeline:

Full-precision LLM — the frozen pretrained model whose weights are never updated during quantization. It provides the reference outputs that the quantized model must match.
Learnable Weight Clipping (LWC) — a module that determines per-channel optimal clipping thresholds for weights by learning two scalar parameters per channel (γ for upper bound, β for lower bound) that scale the min-max range, reducing the quantization error of extreme weight values without directly learning thresholds.
Learnable Equivalent Transformation (LET) — a module that applies channel-wise scaling (s) and shifting (δ) to activations before linear layers, and channel-wise scaling (sₐ) to query/key matrices before attention, shifting quantization difficulty from outlier-heavy activations to better-behaved weights through mathematically equivalent transformations.
Block-wise Quantization Error Minimization — an optimization loop that processes one transformer block at a time, feeding calibration data through already-quantized preceding blocks, optimizing the current block's LWC and LET parameters to minimize the L2 distance between its full-precision output and its quantized output, then freezing the block and advancing.
Quantized Model Assembly — after all blocks are processed, the learned LWC clipping thresholds are fused into the quantized weights, and the LET scaling/shifting factors are absorbed into preceding normalization/linear layers, producing a pure uniform integer quantized model with zero additional operations or parameters at inference time.

Information flows as follows: calibration data enters the first transformer block → LET transforms the activation distribution → LWC determines weight clipping thresholds → quantization is applied → the block's output is compared against the full-precision block's output → gradients flow back to update LWC and LET parameters → the block is frozen → the process repeats for the next block using the quantized block's output as input.

3.3 Roadmap for the Deep Dive

First, the block-wise optimization objective (Equation 1), which defines what "good quantization" means mathematically and establishes the framework that makes both LWC and LET jointly optimizable.
Second, the Learnable Weight Clipping mechanism (LWC), because it handles the fundamental weight quantization challenge independently and is used as the quantizer in both weight-only and weight-activation settings, and its design choice of learning relative clipping strengths rather than absolute thresholds is what enables stable optimization under LET's distribution-shifting effects.
Third, the Learnable Equivalent Transformation (LET), covering both the linear layer transformation (Equation 3) and the attention operation transformation (Equation 5), because LET handles the activation outlier problem — the primary obstacle in weight-activation quantization — and its mathematical equivalence property is what makes the framework deployable without inference overhead.
Fourth, the integration of LWC and LET, covering how they interact synergistically (LET shifts difficulty to weights, LWC makes those shifted weights easier to quantize), how the learnable parameters are initialized, and the practical training configuration (optimizer, learning rates, calibration data, epochs, batch size).
Fifth, the block-wise algorithm and deployment process, explaining the sequential quantization procedure, how learned parameters get fused into the model, and what the final quantized model looks like at inference time.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a method paper whose core idea is that a small set of strategically chosen learnable quantization parameters — relative clipping strengths and equivalent transformation factors — optimized through block-wise gradient descent on minimal calibration data can achieve QAT-level accuracy at extreme low-bit settings while maintaining PTQ-level computational and data efficiency.

Block-Wise Quantization Error Minimization Framework

The optimization objective that unifies LWC and LET is a block-wise reconstruction loss, formalized in Equation 1:

$\arg\min_{\Theta_1, \Theta_2} ||\mathcal{F}(\mathbf{W}, \mathbf{X}) - \mathcal{F}\left(Q_w(\mathbf{W}; \Theta_1, \Theta_2), Q_a(\mathbf{X}, \Theta_2)\right)||$

where $\mathcal{F}$ is the mapping function for a single transformer block (the composition of attention and feed-forward layers that transforms input activations to output activations), $\mathbf{W}$ represents the full-precision weights of that block, $\mathbf{X}$ represents the full-precision input activations to that block, $Q_w(\cdot)$ is the weight quantizer (which uses LWC with parameters $\Theta_1$ to determine clipping thresholds), $Q_a(\cdot)$ is the activation quantizer (a vanilla MinMax quantizer operating on LET-transformed activations), $\Theta_1 = \{\gamma, \beta\}$ are the learnable weight clipping parameters, and $\Theta_2 = \{\delta, \mathbf{s}, \mathbf{s}_a\}$ are the learnable equivalent transformation parameters (channel-wise shifting $\delta$ , channel-wise scaling $\mathbf{s}$ , and attention scaling $\mathbf{s}_a$ ).

What it computes: the L2 norm of the difference between the full-precision block's output and the quantized block's output, when both are fed the same input activations $\mathbf{X}$ . The full-precision block computes $\mathcal{F}(\mathbf{W}, \mathbf{X})$ using the original FP16 weights and activations. The quantized block first applies LET to transform the activations using $\Theta_2$ , then quantizes the transformed activations with a MinMax quantizer, and quantizes the LET-transformed weights using LWC with parameters $\Theta_1$ , then computes the block output. The optimization searches for $\Theta_1$ and $\Theta_2$ that make these two outputs as close as possible in L2 distance, averaged over the calibration dataset.

Why this form: block-wise reconstruction is chosen over end-to-end reconstruction for two practical reasons. First, optimizing the entire model jointly would require backpropagating through all layers, which is memory-prohibitive for billion-parameter models — block-wise optimization processes one block at a time, with the input $\mathbf{X}$ being the output of the already-quantized previous block, making the optimization problem per-block small enough to fit on a single GPU. Second, block-wise optimization decouples the quantization of each block, meaning errors do not compound across blocks during optimization — each block is optimized to match its full-precision counterpart's output distribution given the (potentially imperfect) input from the previous quantized block, which empirically leads to better final performance than trying to optimize all blocks jointly.

The paper notes that prior PTQ methods with gradient optimization (AdaRound, BRECQ) could not scale to LLMs because they optimize over the entire weight space — millions to billions of parameters. OmniQuant's key innovation is that it freezes $\mathbf{W}$ and only optimizes $\Theta_1$ and $\Theta_2$ , which together comprise only a few thousand parameters per block (two clipping strengths per weight channel, two transformation parameters per activation channel). This reduces the optimization from a massively overparameterized problem to a tightly constrained one that converges rapidly on minimal data.

Block-wise processing is sequential. The calibration data flows through all blocks that have already been quantized. When processing block $i$ , the input $\mathbf{X}$ is the output of block $i-1$ after quantization. The full-precision reference $\mathcal{F}(\mathbf{W}, \mathbf{X})$ is computed using the same possibly-degraded input, meaning the optimization naturally compensates for errors introduced by earlier blocks. This is critical: if the full-precision reference used pristine inputs while the quantized block received degraded inputs, the optimization would be solving the wrong problem.

Learnable Weight Clipping (LWC)

The weight quantizer $Q_w$ in OmniQuant is based on uniform asymmetric quantization (integer grid with a zero-point offset), with the key addition that the dynamic range — the minimum and maximum values that map to the quantized grid — is not fixed to the actual weight min/max but is instead learned through two scalar parameters per channel.

The LWC formulation is given in Equation 2:

$\mathbf{W}_q = \text{clamp}\left(\left\lfloor\frac{\mathbf{W}}{h}\right\rceil + z, 0, 2^N - 1\right), \quad \text{where } h = \frac{\gamma \max(\mathbf{W}) - \beta \min(\mathbf{W})}{2^N - 1}, \quad z = -\left\lfloor\frac{\beta \min(\mathbf{W})}{h}\right\rceil$

where $\mathbf{W}_q$ is the quantized weight (an integer tensor), $\mathbf{W}$ is the full-precision weight, $\lfloor\cdot\rceil$ denotes the round-to-nearest operation, $\text{clamp}(\cdot, 0, 2^N-1)$ constrains values to the valid N-bit unsigned integer range $[0, 2^N-1]$ , $N$ is the target bit-width (e.g., 4 for INT4 quantization), $h$ is the normalization factor (the step size — the real-valued distance between adjacent quantized levels), $z$ is the zero-point (the integer value that corresponds to real-valued zero), $\gamma \in [0, 1]$ is the learnable upper clipping strength, and $\beta \in [0, 1]$ is the learnable lower clipping strength. Both $\gamma$ and $\beta$ are parameterized through the sigmoid function $\sigma(t) = 1/(1 + e^{-t})$ , where $t$ is the actual learnable parameter, ensuring the strengths are naturally bounded to $[0, 1]$ without explicit constraint handling.

What it computes: the standard uniform quantization procedure — scale full-precision values by $1/h$ , round to integers, shift by the zero-point $z$ , clamp to the valid range — but with $\gamma$ and $\beta$ controlling how much of the weight distribution's tails are included in the quantized range. When $\gamma = 1$ and $\beta = 1$ , $h$ becomes $(\max(\mathbf{W}) - \min(\mathbf{W}))/(2^N-1)$ , which is exactly the MinMax quantization scheme used in prior work (SmoothQuant, GPTQ) — the entire weight range is covered, but outlier values stretch the step size $h$ , reducing precision for the bulk of the weights. When $\gamma < 1$ or $\beta < 1$ , the effective range $[\beta\min(\mathbf{W}), \gamma\max(\mathbf{W})]$ is narrower than the full weight range, meaning extreme values get clipped (saturated to the min/max quantized level) but the step size $h$ is smaller, giving higher precision for the majority of weight values within the learned range.

Why this form — learning clipping strengths rather than direct thresholds: the paper identifies this as the critical design choice that makes LWC work. Prior clipping-based methods like PACT and LSQ learn the clipping threshold or step size directly as an unconstrained parameter. This works in QAT where weights are being updated to adapt to the clipping, but in OmniQuant's setting — where weights are frozen and the equivalent transformation LET is simultaneously shifting weight distributions — the absolute optimal threshold changes every iteration as LET modifies the weights. Direct threshold learning (PACT/LSQ) fails to converge under this distribution shift, as demonstrated in Table A14 (PACT and LSQ achieve worse perplexity than basic MinMax on W4A4 quantization). LWC's relative formulation — scaling the existing min/max rather than learning new bounds — constrains the solution space: the clipping threshold is always somewhere between the actual min/max and a narrower interior range, parameterized by two bounded scalars. This makes the optimization insensitive to the absolute scale changes introduced by LET because $\gamma$ and $\beta$ are dimensionless ratios, not absolute values.

Figure A5 provides empirical evidence: when LET shifts the weight distribution during training, the weight range (max minus min) fluctuates dramatically. PACT and LSQ, which learn absolute thresholds, fail to track these changes. LWC's $\gamma$ and $\beta$ , being relative, naturally adapt — if the range expands by 2×, the effective clipping threshold automatically scales by 2× without any parameter update.

The asymmetric design (separate $\gamma$ and $\beta$ ) matters because weight distributions are often asymmetric, especially after LET applies channel-wise scaling that may amplify some channels more than others. Learning separate upper and lower clipping strengths allows the quantizer to, for example, clip large positive outlier values while preserving negative values that may carry important information, or vice versa. The paper visualizes the learned clipping distributions in Figure A1, showing that different quantization configurations learn qualitatively different clipping patterns: per-channel W3A16 exhibits a roughly normal distribution of clipping strengths (about half of outlier channels get clipped), while group-wise W2A16g128 shows a long-tailed distribution (most groups use minimal clipping, but a few aggressive clippers handle extreme outliers).

Integration with the quantizer. The weight quantizer $Q_w$ in Equation 1 is specifically the MinMax quantizer with LWC, meaning it uses the learned $h$ and $z$ from Equation 2 as its scale and zero-point. For weight-activation quantization, the weights $\mathbf{W}$ that enter $Q_w$ are not the original weights but the LET-transformed weights $\tilde{\mathbf{W}} = \mathbf{s} \odot \mathbf{W}$ (from Equation 3), so LWC clips the already-transformed weights. This is the synergy: LET makes weights potentially harder to quantize (by pushing activation outliers into the weight domain), and LWC compensates by learning to clip the resulting weight outliers optimally.

At inference time, LWC introduces zero overhead. The learned $\gamma$ and $\beta$ determine $h$ and $z$ , which are used to quantize the weights once. After quantization, $\gamma$ and $\beta$ are discarded — only the quantized integer weights $\mathbf{W}_q$ , the step size $h$ , and the zero-point $z$ are stored. Dequantization during inference is the standard uniform dequantization: $\mathbf{W}_{\text{dequant}} = h \cdot (\mathbf{W}_q - z)$ , which is a simple integer-to-float scaling operation.

Learnable Equivalent Transformation (LET) for Linear Layers

The activation quantizer $Q_a$ in OmniQuant is a vanilla MinMax quantizer (no learned clipping — activations use the full observed range). The key to making activation quantization work at low bit-widths is not modifying the quantizer but modifying the activations themselves before quantization, using a mathematically equivalent transformation that shifts quantization difficulty from activations to weights.

The core insight is that a linear layer $\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{B}$ has many mathematically equivalent parameterizations. If you divide each activation channel by a scalar $s_i$ and multiply the corresponding weight row by the same $s_i$ , the output is unchanged. Similarly, if you subtract a constant $\delta_i$ from each activation channel and compensate by adding $\delta_i \mathbf{W}_{i,:}$ to the bias, the output is unchanged. LET exploits this equivalence to find a parameterization where the activations are well-behaved (no outlier channels with disproportionately large magnitudes) and the weights — even though they've absorbed the outliers — can be effectively quantized thanks to LWC.

The equivalent transformation for a linear layer is formalized in Equation 3:

$\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{B} = \underbrace{[(\mathbf{X} - \boldsymbol{\delta}) \oslash \mathbf{s}]}_{\tilde{\mathbf{X}}} \cdot \underbrace{[\mathbf{s} \odot \mathbf{W}]}_{\tilde{\mathbf{W}}} + \underbrace{[\mathbf{B} + \boldsymbol{\delta}\mathbf{W}]}_{\tilde{\mathbf{B}}}$

where $\mathbf{X} \in \mathbb{R}^{T \times C_{\text{in}}}$ is the input activation (T tokens, $C_{\text{in}}$ input channels), $\mathbf{W} \in \mathbb{R}^{C_{\text{in}} \times C_{\text{out}}}$ is the weight matrix, $\mathbf{B} \in \mathbb{R}^{1 \times C_{\text{out}}}$ is the bias vector, $\mathbf{s} \in \mathbb{R}^{1 \times C_{\text{in}}}$ is a channel-wise scaling vector (one scale factor per input channel), $\boldsymbol{\delta} \in \mathbb{R}^{1 \times C_{\text{in}}}$ is a channel-wise shifting vector (one shift per input channel), $\oslash$ denotes element-wise division (each activation channel divided by its corresponding $s_i$ ), and $\odot$ denotes element-wise multiplication (each weight row multiplied by its corresponding $s_i$ ).

What it computes: a reparameterization of the linear layer that produces exactly the same output $\mathbf{Y}$ as the original layer for any input $\mathbf{X}$ . The transformed activation $\tilde{\mathbf{X}}$ has each channel $i$ shifted by $-\delta_i$ and scaled by $1/s_i$ — if $\delta_i$ centers the channel and $s_i$ is proportional to the channel's magnitude, the transformed activation has channels with roughly uniform scale. The transformed weight $\tilde{\mathbf{W}}$ absorbs the inverse transformation — each row $i$ is multiplied by $s_i$ , so channels that were large in activation become large in weight. The bias becomes $\tilde{\mathbf{B}} = \mathbf{B} + \boldsymbol{\delta}\mathbf{W}$ , which accounts for the activation shift (since $(\mathbf{X} - \boldsymbol{\delta})\mathbf{W} = \mathbf{X}\mathbf{W} - \boldsymbol{\delta}\mathbf{W}$ , the extra $-\boldsymbol{\delta}\mathbf{W}$ must be added back to the bias).

Why this form: the shifting operation $\boldsymbol{\delta}$ handles asymmetric activation distributions — many activation channels have non-zero mean due to LayerNorm's centering effect. The scaling operation $\mathbf{s}$ handles the magnitude disparity — outlier channels have much larger magnitudes than regular channels. Together, they can normalize activation channels to have similar ranges, which is essential for per-token activation quantization (where all channels share the same quantizer scale). The mathematical equivalence guarantees that the transformation introduces no approximation error — the only errors come from the subsequent quantization of $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{W}}$ , which the optimization in Equation 1 directly minimizes.

Comparison to SmoothQuant and Outlier Suppression+: SmoothQuant uses only scaling ( $\boldsymbol{\delta} = 0$ , $\mathbf{s}$ hand-crafted via a fixed migration strength $\alpha$ ). Outlier Suppression+ uses both scaling and shifting, but the scaling is grid-searched and the shifting is pre-defined (computed from activation statistics). LET makes both $\mathbf{s}$ and $\boldsymbol{\delta}$ fully learnable through gradient descent, which the paper shows is critical — Table A4 demonstrates that removing the shifting component degrades W4A4 perplexity on LLaMA-13B from 10.87 to 11.47, and Table A6 shows that initializing $\mathbf{s}$ with SmoothQuant's hand-crafted values (rather than all ones) improves final accuracy by about 1.3 points on average zero-shot tasks. The gradient-based optimization finds transformations that grid search cannot, because the interaction between scaling and shifting is high-dimensional — optimal $\mathbf{s}$ depends on $\boldsymbol{\delta}$ and vice versa, and both depend on the weight clipping in LWC.

Where LET is applied: the paper applies LET to all linear layers in the transformer block except the second linear layer of the FFN (the one after the non-linear activation). As shown in Figure 3, the LET pairs are [LayerNorm before attention, (Q projection, K projection, V projection)], [V projection, output projection], [Q, K in the attention score computation], and [LayerNorm before FFN, first FFN linear layer]. The exclusion of the second FFN layer is explained as: "the high sparsity of features after the non-linear layer leads to unstable gradients when applying learnable equivalent transformations." Table A5 confirms that removing the LET from the [ln1, (q_proj, k_proj, v_proj)] pair causes the largest degradation (average accuracy drops from 52.65% to 46.79% on LLaMA-7B W4A4), consistent with LayerNorm outputs being the primary source of activation outliers.

Fusion at inference — no overhead. After optimization, the scaling factors $\mathbf{s}$ in $\tilde{\mathbf{W}}$ can be absorbed back into the original weight matrix $\mathbf{W}$ by element-wise multiplication: $\mathbf{W}_{\text{fused}} = \mathbf{s} \odot \mathbf{W}$ . The activation transformation $(\mathbf{X} - \boldsymbol{\delta}) \oslash \mathbf{s}$ can be absorbed into the preceding operation — if the preceding operation is a LayerNorm, the scaling $1/\mathbf{s}$ and shifting $-\boldsymbol{\delta}$ modify the LayerNorm's affine parameters; if it's a linear layer, they fuse into that layer's weight and bias. The bias compensation $\boldsymbol{\delta}\mathbf{W}$ is added to the original bias $\mathbf{B}$ . After fusion, the inference computation is identical to the original model — a standard linear layer with quantized weights and a standard MinMax activation quantizer — with no extra multiplications, additions, or memory accesses. This is a crucial practical advantage: the learned transformations improve quantization quality but leave no trace in the deployed model.

Learnable Equivalent Transformation (LET) for Attention Operations

Beyond linear layers, the self-attention mechanism involves a matrix multiplication between queries and keys that is also quantized in the weight-activation setting, and the paper extends LET to this operation. Additionally, KV cache memory — storing the key and value tensors for all previous tokens in autoregressive generation — becomes a significant bottleneck for long sequences, making key/query quantization practically important.

The equivalent transformation for the attention score computation is formalized in Equation 5:

$\mathbf{P} = \text{Softmax}(\mathbf{Q}\mathbf{K}^T) = \text{Softmax}\left(\underbrace{(\mathbf{Q} \oslash \mathbf{s}_a)}_{\tilde{\mathbf{Q}}} \cdot \underbrace{(\mathbf{s}_a \odot \mathbf{K}^T)}_{\tilde{\mathbf{K}}^T}\right)$

where $\mathbf{P}$ is the attention probability matrix, $\mathbf{Q} \in \mathbb{R}^{T \times d}$ is the query matrix, $\mathbf{K} \in \mathbb{R}^{T \times d}$ is the key matrix, $\mathbf{s}_a \in \mathbb{R}^{1 \times d}$ is a channel-wise scaling vector for the attention head dimension, and the transformations $\tilde{\mathbf{Q}} = \mathbf{Q} \oslash \mathbf{s}_a$ and $\tilde{\mathbf{K}}^T = \mathbf{s}_a \odot \mathbf{K}^T$ are inverses of each other, leaving the product $\mathbf{Q}\mathbf{K}^T$ unchanged.

What it computes: a channel-wise rescaling of the query and key matrices that preserves their dot product. Each dimension $i$ of the query vectors is divided by $s_{a,i}$ while the corresponding dimension of the key vectors is multiplied by $s_{a,i}$ . The element-wise product inside the matrix multiplication cancels out, so $\tilde{\mathbf{Q}}\tilde{\mathbf{K}}^T = \mathbf{Q}\mathbf{K}^T$ exactly. This reparameterization allows the quantizer to operate on rescaled $\tilde{\mathbf{Q}}$ and $\tilde{\mathbf{K}}$ where outlier dimensions have been suppressed in the query and amplified in the key (or vice versa — the optimization learns which direction minimizes quantization error).

Why this form: the attention score computation is a matrix multiplication, not a linear layer, so the standard LET from Equation 3 does not directly apply — there is no weight matrix to absorb the inverse transformation. Instead, the paper exploits the symmetry of the dot product: scaling one operand up and the other down by the same factor preserves the result. The learnable $\mathbf{s}_a$ allows the optimization to redistribute magnitude between $\mathbf{Q}$ and $\mathbf{K}$ to make both easier to quantize. This is particularly relevant because the outlier pattern in queries and keys may differ — some attention heads may have outlier dimensions in queries but not in keys, and $\mathbf{s}_a$ can shift magnitude to the better-behaved operand.

Fusion: the scaling $\oslash \mathbf{s}_a$ on $\mathbf{Q}$ is absorbed into the query projection weight matrix $\mathbf{W}_Q$ by scaling its rows: $\mathbf{W}_{Q,\text{fused}} = \mathbf{W}_Q \oslash \mathbf{s}_a$ (with appropriate broadcasting). The scaling $\odot \mathbf{s}_a$ on $\mathbf{K}^T$ is absorbed into the key projection weight matrix $\mathbf{W}_K$ by scaling its rows: $\mathbf{W}_{K,\text{fused}} = \mathbf{W}_K \odot \mathbf{s}_a$ . After fusion, the attention computation uses the standard quantized matrix multiplications with no extra operations.

The value matrix $\mathbf{V}$ is not explicitly transformed in Equation 5. The paper explains: "the explicit transformation of $\mathbf{V}$ is omitted as its distribution has already been channel-wise altered by the inverse transformation associated with the output projection linear layer." The V projection is part of the [v_proj, out_proj] LET pair (visible in Figure 3) — the output projection's inverse transformation effectively propagates to V through the attention-weighted sum, making a separate V transformation redundant.

The impact of attention LET is measurable but modest. Table A4 shows that removing the attention transformation increases W4A4 perplexity on LLaMA-13B from 10.87 to 11.34 — a smaller degradation than removing activation shifting (to 11.47). The paper attributes this to the fact that "the majority of outliers exist in the output of the normalization layer while being less prevalent in the Q/K/V matrix" — LayerNorm outputs have the most severe channel-wise magnitude disparities, which the linear layer LET primarily addresses.

Integration of LWC and LET: The Synergistic Optimization

LWC and LET are not independent modules applied sequentially — they are jointly optimized under the block-wise objective in Equation 1, and their interaction is what enables OmniQuant's performance at extreme low-bit settings.

The synergy mechanism: LET transforms activations to be quantization-friendly by scaling down outlier channels, but this mathematically requires scaling up the corresponding weight channels (Equation 3: $\tilde{\mathbf{W}} = \mathbf{s} \odot \mathbf{W}$ ). Weight channels that receive large scaling factors $s_i$ become harder to quantize because their effective range expands — this is the "migration of quantization difficulty from activations to weights" that SmoothQuant identified. LWC addresses this by learning to clip these now-amplified weight channels — $\gamma$ and $\beta$ can shrink the effective range to discard the newly created outliers. Without LWC, LET would improve activation quantization at the cost of severely degrading weight quantization. Without LET, LWC would only handle weight outliers but leave activation outliers unaddressed. Together, they form a complete solution: LET makes activations quantizable, LWC makes the LET-transformed weights quantizable.

Evidence for synergy: Table A3 quantifies this interaction. On LLaMA-13B W4A4, the full OmniQuant (LWC+LET) achieves 10.87 perplexity. Removing LWC (+LET only) degrades to 20.75 — LET pushes weight outliers that LWC is needed to handle. Removing LET (+LWC only) causes collapse to 5.4e3 — without LET, activation outliers remain, and LWC alone cannot fix activation quantization. Removing both (vanilla MinMax) gives 1.8e3. The gap between LWC+LET (10.87) and the sum of individual improvements shows synergy, not mere additivity. Table A2 further shows that replacing LET with SmoothQuant (hand-crafted scaling) and LWC with grid-searched weight clipping reduces W4A4 accuracy from 52.65% to 49.59%, confirming that the joint gradient-based optimization finds solutions inaccessible to sequential or heuristic combinations.

Simultaneous vs. iterative training: the paper explored training LWC and LET iteratively (alternating updates) rather than simultaneously (joint gradient updates each step). Table A8 shows that simultaneous training achieves 52.65% average accuracy on LLaMA-7B W4A4, while iterative training by epochs achieves 52.06% and by iterations achieves only 50.91%. Even doubling the training epochs for iterative training (40 epochs vs. 20) only reaches 52.50%, still below simultaneous training at 20 epochs. This demonstrates that LWC and LET reinforce each other during optimization — a gradient step that improves LET immediately affects the optimal LWC parameters, and joint updates allow both to adapt in lockstep.

Initialization and Optimization Configuration

LET initialization: the channel-wise scaling $\mathbf{s}$ is initialized using SmoothQuant's migration strength, which computes $s_i = \max(|\mathbf{X}_{:,i}|)^\alpha / \max(|\mathbf{W}_{i,:}|)^{1-\alpha}$ with $\alpha = 0.5$ for most layers, designed to equalize the maximum magnitudes of activation channels and weight rows. The channel-wise shifting $\boldsymbol{\delta}$ is initialized using Outlier Suppression+'s method, which sets $\delta_i = \text{mean}(\mathbf{X}_{:,i})$ to center each activation channel. The attention scaling $\mathbf{s}_a$ is initialized to all ones (no transformation). Table A6 confirms that these careful initializations matter: initializing $\mathbf{s}$ to all ones instead of SmoothQuant values reduces average zero-shot accuracy by about 1.3 points, and initializing $\boldsymbol{\delta}$ to zeros instead of channel means causes a smaller but measurable 0.43 point drop.

LWC initialization: $\gamma$ and $\beta$ are initialized to 0.5 (learnable parameter $t = 0$ for the sigmoid, giving $\sigma(0) = 0.5$ ), meaning the initial clipping uses roughly half the full weight range for both upper and lower bounds. This is a conservative starting point that clips outliers aggressively and then learns to expand or contract the range based on the calibration data.

Optimizer and hyperparameters: the paper uses AdamW with zero weight decay. The learning rate for LWC parameters ( $\Theta_1$ ) is $5 \times 10^{-3}$ , and for LET parameters ( $\Theta_2$ ) is $1 \times 10^{-2}$ . The higher learning rate for LET reflects that the transformation parameters have a broader impact — they affect both activation and weight quantization pathways — and need to move further from their initialization. The calibration dataset consists of 128 randomly selected 2048-token segments from WikiText2, following the convention established by GPTQ and AWQ. Training uses a batch size of 1 (one 2048-token sequence per gradient step) over 20 epochs for all configurations except W2A16 quantization, which uses 40 epochs (Table A9 shows that W2A16 perplexity continues improving from 17.46 at 20 epochs to 14.77 at 80 epochs, with diminishing returns after 40 epochs). The entire process runs on a single Nvidia A100 GPU.

Data efficiency: Table A11 demonstrates that OmniQuant converges with as few as 16 calibration samples — W3A16 WikiText2 perplexity is 6.47 with 16, 32, 64, 128, or 256 samples, showing no sensitivity to calibration set size above the bare minimum. W4A4 does show improvement from 11.56 at 16 samples to 11.23 at 128 samples, but the marginal gain beyond 64 samples is minimal. This robustness to calibration data volume is a consequence of the small parameter count — with only a few thousand learnable parameters, even 16 sequences of 2048 tokens provide millions of gradient updates.

Cross-dataset robustness: Table A10 tests calibration on different datasets (WikiText2, C4, Pile) and finds the variance in perplexity across calibration sources is extremely small — between 0.0006 and 0.17 — indicating that the learned transformations generalize across data distributions rather than overfitting to the calibration set's specific patterns.

Which components are activated for which quantization modes: for weight-activation quantization (W6A6, W4A4), both LWC and LET are enabled, with LET applied to all positions shown in Figure 3 (except the second FFN linear layer). For weight-only quantization (W2A16, W3A16, W4A16), LWC is always enabled, but LET is enabled only for OPT models and disabled for LLaMA models. Table A3 explains this: on LLaMA-13B W3A16, LET improves perplexity from 5.68 (LWC only) to 5.65 (LWC+LET) — a negligible 0.03 gain — while on OPT-13B W3A16, it improves from 11.29 to 10.87 — a meaningful 0.42 gain. The difference is attributed to LLaMA having "few weight outliers" compared to OPT, making the activation-to-weight migration from LET less beneficial. Disabling LET for LLaMA weight-only quantization reduces training time (fewer parameters to optimize) without sacrificing quality.

The Block-Wise Quantization Algorithm and Final Deployment

The full OmniQuant procedure is detailed in Algorithm 1 (Appendix A1) and proceeds sequentially through the model:

Phase 1 — Initialization: The calibration data $\mathbf{X}$ is passed through the full-precision model once to collect the inputs $\mathbf{X}_{\text{fp}, i}$ and outputs of each transformer block. The quantized model's input $\mathbf{X}_q$ is initialized to the same calibration data. For each block $B_i$ in order:

Phase 2 — Learnable parameter setup: The LWC parameters $\Theta_1 = \{\gamma, \beta\}$ are instantiated per weight tensor (each linear layer's weight matrix gets its own $\gamma$ and $\beta$ vectors, one scalar per output channel for per-channel quantization, or per group for group-wise quantization). The LET parameters $\Theta_2 = \{\mathbf{s}, \boldsymbol{\delta}, \mathbf{s}_a\}$ are instantiated for the LET pairs in that block. Both are initialized as described above.

Phase 3 — Optimization loop: For each epoch and each batch of calibration data:

Apply LET to create the transformed block $B'_i$ : weights are scaled via Equation 3 ( $\tilde{\mathbf{W}} = \mathbf{s} \odot \mathbf{W}$ ), biases are adjusted ( $\tilde{\mathbf{B}} = \mathbf{B} + \boldsymbol{\delta}\mathbf{W}$ ), and attention Q/K are scaled via Equation 5.
Apply quantization with LWC to $B'_i$ : weights are quantized using Equation 2 with the learned $\gamma$ and $\beta$ , activations are quantized with a vanilla MinMax quantizer (computing min/max from the current batch of $\tilde{\mathbf{X}}$ ).
Compute the quantized block's output $\mathbf{x}'_q = B'_i(\mathbf{x}_q)$ using the current batch of calibration data $\mathbf{x}_q$ (which is the output of the previous, already-quantized block).
Compute the L2 loss $\mathcal{L} = ||\mathbf{x}_{\text{fp}} - \mathbf{x}'_q||_2$ , where $\mathbf{x}_{\text{fp}}$ is the full-precision block's output on the same input.
Backpropagate to compute gradients w.r.t. $\Theta_1$ and $\Theta_2$ only — the original weights $\mathbf{W}$ are frozen and receive no gradient.
Update $\Theta_1$ and $\Theta_2$ using AdamW.

Phase 4 — Finalize block: After training converges (20 or 40 epochs), the optimal LET and LWC parameters are applied once more to create the final quantized block. The LET transformations are permanently fused: $\mathbf{W}_{\text{fused}} = \mathbf{s} \odot \mathbf{W}$ and $\mathbf{B}_{\text{fused}} = \mathbf{B} + \boldsymbol{\delta}\mathbf{W}$ for each linear layer, and $\mathbf{W}_{Q,\text{fused}} = \mathbf{W}_Q \oslash \mathbf{s}_a$ , $\mathbf{W}_{K,\text{fused}} = \mathbf{W}_K \odot \mathbf{s}_a$ for attention. The fused weights are then quantized using LWC with the learned $\gamma$ and $\beta$ . The activation transformations $(\mathbf{X} - \boldsymbol{\delta}) \oslash \mathbf{s}$ are fused into the preceding operation (LayerNorm or linear layer). The block is now a standard quantized transformer block with no extra parameters or operations. The quantized block's output on the calibration data becomes $\mathbf{x}_q$ for the next block.

Phase 5 — Repeat: The process advances to the next transformer block, using the quantized previous block's output as input.

What the deployed model looks like: After all blocks are processed, the model consists of standard transformer blocks where each linear layer has quantized integer weights (e.g., INT4 stored as packed 4-bit integers), a per-channel or per-group step size $h$ and zero-point $z$ , and quantized activations using per-token MinMax quantization during inference. There are no additional clipping modules, scaling layers, or transformation operations — LWC and LET have been completely absorbed. This is why OmniQuant is compatible with off-the-shelf deployment frameworks like MLC-LLM: the quantized model uses standard uniform integer quantization primitives that MLC-LLM already supports for CUDA inference.

The Softmax output is not quantized — the paper keeps it at full precision (16-bit) due to its "long-tail distribution making it unsuitable for uniform quantization." Table A7 explores quantizing Softmax outputs: 8-bit Softmax causes negligible degradation (52.65% → 51.93% average accuracy), 6-bit is acceptable (51.70%), but 4-bit causes a significant drop to 48.52%. The paper opts to keep Softmax at 16-bit as a conservative choice, noting that future work (such as log2 quantization from RepQ-ViT) could address this.

Training time and scalability: Table A12 reports training times on a single A100-80G: LLaMA-7B weight-activation quantization takes 1.6 hours, LLaMA-13B takes 3.3 hours, LLaMA-30B takes 7.3 hours, and LLaMA-65B takes 14.4 hours. Weight-only quantization is faster (1.1h, 2.2h, 4.5h, 8.9h respectively) because LET is disabled for LLaMA, reducing the number of learnable parameters. These times scale roughly linearly with model size, consistent with the block-wise approach where each block's optimization cost is proportional to its parameter count. The paper notes this is "approximately 5× greater than GPTQ" but "markedly faster than QAT methods, which demand hundreds of GPU hours."

4. Key Insights and Innovations

Innovation 1: Differentiable PTQ as a Third Path Between Hand-Crafted PTQ and Full QAT

The paper's most fundamental contribution is conceptual: identifying that the dominant dichotomy in LLM quantization — training-free PTQ (fast but inaccurate at low bit-widths) vs. quantization-aware training (accurate but prohibitively expensive) — is a false one enforced by the parameterization choices of existing methods, not by any inherent limitation of post-training optimization. OmniQuant introduces a third category: differentiable PTQ that learns only the quantization parameters themselves while keeping model weights frozen.

Prior PTQ methods with gradient optimization existed (AdaRound, BRECQ), but they optimized over the weight space — millions to billions of parameters — making them intractable for LLMs. The field consequently split into two camps: either use no gradient-based optimization at all (GPTQ's closed-form solutions, AWQ's grid search, SmoothQuant's hand-crafted migration strengths) or commit to full QAT with its massive data and compute requirements (LLM-QAT with 100k samples and hundreds of GPU hours). The underlying assumption — unstated but universally adopted — was that gradient-based optimization in PTQ necessarily means optimizing the weights.

OmniQuant breaks this assumption by asking: what if we optimize only the knobs that control the quantization process, not the weights themselves? The answer requires identifying which knobs matter — the weight clipping range and the activation-channel equalization — and designing them to be learnable through gradient descent while keeping the optimization landscape stable. This reframing is significant beyond the specific techniques: it establishes that the PTQ-vs-QAT tradeoff is a function of parameterization, not of whether gradients are used. Future work on quantization can now explore other learnable quantization parameterizations that sit in this middle ground, rather than accepting the binary choice between cheap heuristics and expensive retraining.

The evidence for this reframing is in the FLOPs comparison: OmniQuant achieves 52.65% average zero-shot accuracy on LLaMA-7B W4A4 (Table 2), exceeding LLM-QAT's 46.43%, while using 128 samples and 1.6 hours on one GPU versus 100k samples and hundreds of GPU hours. The fact that a handful of learned parameters (a few thousand per block) can outperform full model retraining is the empirical validation that the PTQ-vs-QAT dichotomy was about which parameters get optimized, not whether optimization happens.

Innovation 2: Relative Parameterization as the Enabler of Joint Weight-Activation Optimization

The paper's second innovation is a design principle for learnable quantization parameters: relative parameterization decouples optimization from absolute scale, enabling stable joint learning under distribution shift. This is embodied in LWC's choice to learn clipping strengths (γ, β) — multiplicative scalars on the existing min/max — rather than absolute clipping thresholds (as PACT, LSQ do), and is what makes the LWC-LET synergy possible.

The problem this solves is subtle and specific to the weight-activation setting. LET shifts activation outliers into the weight domain by scaling weight channels up (Equation 3: W̃ = s ⊙ W). This means the weight distribution changes every optimization step as s updates. An absolute threshold learner (PACT, LSQ) would need to chase a moving target — the optimal clipping point in absolute terms is different every iteration because the weight magnitudes themselves are changing. LWC's relative formulation (γ·max(W), β·min(W)) automatically tracks the distribution: if LET doubles a weight channel's magnitude, the effective clipping threshold doubles without any parameter update. The optimization only needs to learn what fraction of the distribution to clip, which is invariant under the scaling transformations that LET applies.

This insight is why the paper rejects direct adoption of PACT or LSQ — not because those methods are wrong in general, but because they are incompatible with joint weight-activation optimization under equivalent transformations. Table A14 and Figure A5 provide the evidence: PACT/LSQ work acceptably for weight-only quantization (where the distribution is static) but fail under LET's distribution shifting (where LWC maintains stability). The relative parameterization is thus not merely an implementation detail but the key conceptual move that makes the LWC-LET synergy viable.

This principle generalizes: whenever quantization parameters must be jointly optimized with transformations that alter the target distribution, relative parameterizations that factor out the absolute scale will be more stable than absolute ones. The field's prior default — learning thresholds or step sizes directly — was inherited from QAT contexts where weights adapt to the quantizer, not the reverse. OmniQuant inverts this relationship.

Innovation 3: Equivalent Transformation as a Learnable, Attention-Aware, Multi-Operation Framework

Prior work on equivalent transformations for LLM quantization (SmoothQuant, AWQ, Outlier Suppression+) treated the operation as a linear layer pre-processing step with hand-crafted or grid-searched parameters. OmniQuant re-conceptualizes equivalent transformation along three axes that collectively expand its reach and effectiveness:

First, making it learnable: converting scaling and shifting from heuristically determined scalars to gradient-optimized parameters. This is not merely "grid search but with gradients" — the learning enables joint optimization across all layers and transformation types simultaneously. Grid search over even two parameters per channel across 32 transformer blocks is combinatorially infeasible; gradient descent handles this high-dimensional space naturally. Table A4's comparison of omitting shifting (perplexity 10.87 → 11.47) or attention transformation (→ 11.34) shows that the gradient discovers configurations where these components interact beneficially — configurations that sequential or heuristic search cannot find because the optimal scaling at one layer depends on the shifting applied at that layer and the transformations at adjacent layers.

Second, extending to attention operations: prior work applied equivalent transformations only to linear layers (the Q/K/V projections and the FFN). OmniQuant adds the attention score computation itself (Equation 5), exploiting the symmetry of the dot product to rebalance magnitude between queries and keys. This is a conceptual extension: the equivalent transformation principle — that scaling one operand and inversely scaling the other preserves the output — applies to any bilinear operation, not just weight-activation products. The attention extension is modest in impact (Table A4 shows a 0.47 perplexity improvement) but significant as a proof of concept that the principle generalizes.

Third, applying across both weight-only and weight-activation regimes: prior methods specialized — SmoothQuant and Outlier Suppression+ were designed for weight-activation quantization, while AWQ adapted scaling for weight-only. OmniQuant's LET works for both by design (activated for OPT weight-only where it helps, disabled for LLaMA weight-only where it doesn't), demonstrating that the equivalent transformation concept is not inherently tied to activation quantization — it can also improve weight-only quantization when the model architecture creates weight outliers through activation-weight interactions that equivalent transformations can equalize.

The practical significance of this reframing is that future work on quantization can treat equivalent transformations as a generic, learnable pre-processing step applicable to any operation with mathematical invariants (bilinear forms, softmax attention, potentially even non-linear activations with appropriate reparameterization), rather than as a specialized fix for activation outliers in linear layers.

Innovation 4: Comprehensive Scaling Laws for Low-Bit LLM Quantization — and the 3-bit Viability Finding

While the paper does not frame it as a theoretical contribution, the sheer breadth of its experimental results constitutes an empirical contribution: the first systematic demonstration that uniform 3-bit and 2-bit quantization of LLMs is viable across model families, scales, and evaluation dimensions, provided the quantization parameters are properly optimized. Prior to this work, the dominant narrative — supported by figures like GPTQ's 5,500 perplexity on LLaMA-13B W2A16 and AWQ's 280,000 — was that extreme low-bit quantization was fundamentally too lossy for LLMs except with special representations (mixed-precision, non-uniform, sparse-quantized).

OmniQuant demonstrates that the problem was not the bit-width itself but the optimization of the quantization parameters. With properly learned clipping and equivalent transformations, W2A16g128 achieves 7.93 WikiText2 perplexity on LLaMA-13B (Table 1), close to the 5.09 FP16 baseline and dramatically better than GPTQ's 15.60. W2A16g64 reaches 7.34 — only 2.25 points above FP16 at 1/8th the weight precision. Even W2A16 (per-channel, no grouping) achieves 13.21, demonstrating that the degradation is not inherent to 2-bit representation.

The scaling law visualization in Figure A4 makes this point clearly: OmniQuant's 3-bit quantization achieves a comparable perplexity-vs-model-bits tradeoff to 4-bit quantization under prior methods. This has direct practical implications — a 3-bit model can serve more users on the same hardware budget as a 4-bit model, or fit a 13B model where previously only a 7B model would fit, without additional perplexity cost.

This finding changes the conversation around low-bit quantization from "can we make it work at all?" to "how do we optimize the tradeoff between bit-width, group size, and calibration?" The paper does not claim that 2-bit is always sufficient — performance degrades measurably — but shows that the degradation is smooth and controllable rather than catastrophic, which opens the design space for practitioners to choose their operating point based on hardware constraints rather than binary feasibility.

The instruction-tuned model results (Figure 4) add a critical generalization dimension: OmniQuant's W3A16g128 quantization of LLaMA-2-7b-chat achieves a 50% win rate against AWQ's quantization on the Vicuna benchmark (statistically tied) and 80.3% win rate against RTN. This demonstrates that the technique transfers to chatbot-quality evaluation, not just perplexity benchmarks, and that the learned parameters do not overfit to the calibration data distribution at the expense of conversational capability.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation datasets are WikiText2 (Merity et al., 2016), PTB (Marcus et al., 1994), and C4 (Raffel et al., 2020) for perplexity measurements, with 128 randomly selected 2048-token segments from WikiText2 serving as the calibration set for training. Zero-shot accuracy is evaluated on PIQA, ARC (easy and challenge), BoolQ, HellaSwag, and Winogrande, all implemented through lm-eval-harness (Gao et al., 2021). The Vicuna benchmark (Chiang et al., 2023) with 80 questions is used for instruction-tuned model evaluation, with GPT-4 as the judge following the protocol from Zheng et al. (2023). MMLU results are reported in Table A16 for LLaMA-7B only.
Base model(s). The paper tests on OPT (125M–66B; Zhang et al., 2022), LLaMA-1 (7B–65B; Touvron et al., 2023a), LLaMA-2 (7B–70B; Touvron et al., 2023b), LLaMA-2-chat (7B–13B; the instruction-tuned variant of LLaMA-2), and Falcon-180B (Penedo et al., 2023). This spans three orders of magnitude in parameter count (125M to 180B) across four model families, chosen to demonstrate generalizability beyond a single architecture. The main paper highlights LLaMA results, with comprehensive OPT and Falcon results in Appendix A8.
Metrics. For language generation, perplexity (PPL) is reported on WikiText2, PTB, and C4 — lower is better, computed as the exponential of the cross-entropy loss following the GPTQ evaluation settings. For zero-shot tasks, accuracy (%) is the primary metric, with the average across all six tasks reported to summarize overall capability preservation. For instruction-tuned models, win rate (%) against baselines is computed from GPT-4 pairwise comparisons, with ties excluded from win rate calculation. Memory (GB) and inference speed (tokens/second) are measured on real hardware (NVIDIA A100-80G) via MLC-LLM deployment.
Baselines. For weight-only quantization: RTN (vanilla round-to-nearest), GPTQ (Frantar et al., 2022), and AWQ (Lin et al., 2023). For weight-activation quantization: SmoothQuant (Xiao et al., 2023), Outlier Suppression+ (Wei et al., 2023), RPTQ (Yuan et al., 2023), and LLM-QAT (Liu et al., 2023b) — the latter being a QAT method. The paper reproduces SmoothQuant and Outlier Suppression+ with per-channel weight quantization and per-token activation quantization for fair comparison. RPTQ results are reported both from the original paper (which kept LayerNorm and SoftMax outputs at 8-bit) and reproduced with the paper's setting (all activations quantized except SoftMax at full precision, denoted RPTQ*).
Generation budget / compute accounting. OmniQuant's cost is measured in training time (GPU hours on a single A100) and calibration data volume (number of 2048-token sequences). This is compared against GPTQ (training-free, ~1 hour for 13B models) and LLM-QAT (100k samples, hundreds of GPU hours). Memory footprint is reported as "Weights Memory" (WM, the storage for quantized weights) and "Running Memory" (RM, the total GPU memory during inference including activations), with inference speed measured in tokens/second for 512-token generation.
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the reference example's compute-optimal scaling analysis). Training is performed once on the 128-sample WikiText2 calibration set, and evaluation is on standard test splits of each benchmark. The Vicuna benchmark uses double-blind pairwise comparison (each pair evaluated in both orders, totaling 160 trials per comparison) to negate position bias in GPT-4 judgments. Ablation studies on calibration dataset robustness (Tables A10, A11) test sensitivity to data distribution and volume but do not constitute formal cross-validation.

Main Quantitative Results

Weight-Only Quantization Results

The headline result for weight-only quantization is that OmniQuant consistently outperforms GPTQ and AWQ across all model families (LLaMA-1, LLaMA-2, OPT), all bit-widths (W2A16 through W4A16), and both per-channel and group-wise configurations, with the performance gap widening dramatically as bit-width decreases. Table 1 presents the LLaMA family results on WikiText2 perplexity.

W4A16 and W4A16g128: At 4-bit weight-only quantization, the gap between methods is modest because this regime is relatively easy. For LLaMA-13B W4A16 (per-channel), OmniQuant achieves 5.21 PPL versus 5.40 for GPTQ and 5.34 for AWQ — a 0.19 improvement over GPTQ (Table 1). At W4A16g128 (group-wise with 128-group size), the differences narrow further: OmniQuant achieves 5.17 versus 5.20 for GPTQ and 5.20 for AWQ on LLaMA-13B — only a 0.03 improvement. Across all LLaMA-1 and LLaMA-2 models at W4A16g128, OmniQuant's advantage over GPTQ ranges from 0.02 to 0.04 PPL. This is a small but consistent improvement at 4-bit, where even GPTQ performs reasonably well.

W3A16 and W3A16g128: At 3-bit, OmniQuant's advantage grows. On LLaMA-13B W3A16 (per-channel), OmniQuant achieves 5.68 PPL compared to GPTQ's 6.76 and AWQ's 7.45 — improvements of 1.08 and 1.77 respectively (Table 1). The more practically relevant W3A16g128 shows: OmniQuant 5.44 vs. GPTQ 5.62 vs. AWQ 5.51 on LLaMA-13B. On LLaMA-7B W3A16g128, OmniQuant achieves 6.15 versus GPTQ's 6.55 and AWQ's 6.46. The pattern holds across all model sizes: for LLaMA-30B W3A16g128, OmniQuant (4.56) vs. GPTQ (4.80) vs. AWQ (4.63); for LLaMA-65B, OmniQuant (3.94) vs. GPTQ (4.17) vs. AWQ (3.99). Notably, AWQ sometimes outperforms GPTQ at 3-bit (e.g., LLaMA-13B: 5.51 vs. 5.62) but is consistently beaten by OmniQuant.

W2A16 and its group-wise variants — the critical stress test: This is where prior methods collapse and OmniQuant's advantage becomes qualitative rather than quantitative. On LLaMA-13B W2A16 (per-channel, no grouping), GPTQ produces 5,500 PPL (effectively a broken model) while OmniQuant achieves 13.21 — still degraded from FP16's 5.09 but coherent and usable. RTN yields 68,000. At W2A16g128, GPTQ achieves 15.60 and AWQ catastrophically fails at 280,000 — OmniQuant achieves 7.93 PPL (Table 1). At W2A16g64 (finer granularity), OmniQuant reaches 7.34 while GPTQ achieves 10.06. The scaling across model sizes is consistent: LLaMA-65B W2A16g128: OmniQuant 5.95 vs. GPTQ 9.51; LLaMA-2-70B W2A16g128: OmniQuant 6.55 vs. GPTQ failing (NaN). The paper notes that for LLaMA-2-70B W2A16 and W2A16g64, GPTQ reports NaN, indicating numerical instability during the Hessian-based weight compensation that OmniQuant avoids entirely.

OPT family results (Tables A20, A21, A22): The pattern replicates on OPT models. On OPT-13B W2A16g128: OmniQuant achieves 12.94 WikiText2 PPL vs. 21.36 for GPTQ and 14.32 for AWQ (Table A20). On OPT-30B W2A16g128: OmniQuant 11.39 vs. GPTQ 12.71 vs. AWQ 12.31. On OPT-66B W2A16g64 (Table A20): OmniQuant achieves 10.59 — remarkably close to the FP16 baseline of 9.34 — while GPTQ collapses to 356.01 and AWQ manages 12.74. This demonstrates that the benefits are not LLaMA-specific but stem from the architecture-agnostic mechanism of learning clipping thresholds and equivalent transformations.

Falcon-180B (Table A18): OmniQuant's W3A16g512 quantization of Falcon-180B reduces the model from requiring 5 A100-80G GPUs (335GB memory) to fitting on a single A100-80G (65GB). WikiText2 PPL increases from 3.29 (FP16) to 3.71, while RTN degrades to 5.33. On zero-shot tasks, OmniQuant achieves 79.40% average accuracy compared to 80.58% for FP16 — a 1.18 percentage point drop — while RTN drops to 77.97%. This validates the approach at the largest scale tested (180B parameters).

C4 perplexity in weight-only quantization (Table A19): The C4 results mirror WikiText2 patterns. LLaMA-13B W2A16g128: OmniQuant 10.36 vs. GPTQ 15.29. LLaMA-13B W3A16g128: OmniQuant 7.05 vs. GPTQ 7.10 vs. AWQ 7.07 — the gap narrows at higher bit-widths. The consistency across WikiText2 and C4 (different data distributions) suggests the quantized models preserve general language modeling capability rather than overfitting to calibration data characteristics.

Weight-Activation Quantization Results

The headline for weight-activation quantization is that OmniQuant substantially outperforms prior PTQ methods at W6A6 and W4A4, and at W4A4 even surpasses the QAT method LLM-QAT while using orders of magnitude less compute. Results are primarily in Table 2 (zero-shot accuracy) and Tables A23, A24 (perplexity).

W6A6 quantization: At 6-bit weight and activation, the degradation from FP16 is modest for all methods, but OmniQuant consistently achieves the best results. On LLaMA-7B W6A6, OmniQuant achieves 63.17% average zero-shot accuracy versus SmoothQuant's 62.81% and Outlier Suppression+'s 61.13% (Table 2). The FP16 baseline is 64.09%, so OmniQuant loses only 0.92 percentage points. The perplexity results in Table A23 confirm: LLaMA-7B W6A6 WikiText2 PPL is 5.96 for OmniQuant vs. 6.03 for SmoothQuant vs. 5.68 for FP16. On larger models, the gap to FP16 narrows: LLaMA-65B W6A6 achieves 70.28% accuracy (OmniQuant) vs. 71.04% (FP16) — a 0.76 point drop — while SmoothQuant reaches 69.80% and Outlier Suppression+ 68.76%.

W4A4 quantization — the critical weight-activation stress test: This is where prior PTQ methods exhibit major degradation and OmniQuant's advantage becomes pronounced. On LLaMA-7B W4A4, OmniQuant achieves 52.65% average zero-shot accuracy, compared to SmoothQuant's 38.41% and Outlier Suppression+'s 48.43% — improvements of 14.24 and 4.22 percentage points respectively (Table 2). The gap to FP16 is 11.44 points (64.09% → 52.65%), which is substantial but far smaller than the 25.68-point gap for SmoothQuant. On LLaMA-13B W4A4: OmniQuant 54.37% vs. SmoothQuant 49.36% vs. Outlier Suppression+ 49.86%. On LLaMA-30B W4A4: OmniQuant 56.63% vs. SmoothQuant 44.83% vs. Outlier Suppression+ 52.62% — OmniQuant is 11.80 points above SmoothQuant. On LLaMA-65B W4A4: OmniQuant 59.22% vs. SmoothQuant 47.71% vs. Outlier Suppression+ 52.52%.

Comparison with QAT (LLM-QAT): On LLaMA-7B W4A4, LLM-QAT achieves 41.27% average accuracy, and with SmoothQuant initialization (LLM-QAT+SQ) reaches 46.43%. OmniQuant achieves 52.65% — a 6.22 percentage point improvement over the best QAT variant — while using 128 samples and 1.6 GPU hours versus LLM-QAT's 100k samples and hundreds of GPU hours (Table 2). This is the paper's most striking result, as it suggests that learning quantization parameters is not just more efficient than QAT but can actually produce better models, possibly because QAT's full weight fine-tuning on limited data overfits or destabilizes the pretrained representations.

Perplexity results for W4A4 (Table A23): The pattern in perplexity mirrors accuracy. LLaMA-7B W4A4 WikiText2 PPL: OmniQuant 11.26 vs. SmoothQuant 25.25. LLaMA-13B: OmniQuant 10.87 vs. SmoothQuant 40.05. LLaMA-30B: 10.33 vs. 192.40. LLaMA-65B: 9.17 vs. 275.53. SmoothQuant's perplexity explodes at larger model sizes (from 25.25 at 7B to 275.53 at 65B), while OmniQuant's remains relatively stable (11.26 to 9.17 — actually improving with model size, tracking the FP16 trend from 5.68 to 3.53). This suggests that SmoothQuant's hand-crafted migration strength becomes increasingly mis-calibrated at scale, while OmniQuant's learned transformations adapt to each model's specific activation patterns.

OPT weight-activation results (Table A25): On OPT models, OmniQuant is compared against SmoothQuant and RPTQ. At W6A6, all methods perform similarly: OPT-13B WikiText2 PPL is 10.21 (OmniQuant) vs. 10.56 (SmoothQuant) vs. 11.00 (RPTQ reported) vs. 10.25 (RPTQ* reproduced at comparable settings). At W4A4, OmniQuant achieves 11.65 on OPT-13B, compared to SmoothQuant's 7.4e3 and RPTQ*'s 16.45 — a massive gap. On OPT-66B W4A4, OmniQuant achieves 10.29 PPL, which is only 0.95 above the FP16 baseline of 9.34, while RPTQ reaches 12.23 and SmoothQuant collapses to 2.2e5. This demonstrates that OmniQuant's advantage persists across model families for weight-activation quantization.

The per-task accuracy breakdown in Table 2 reveals where quantization hurts most. On LLaMA-7B W4A4, HellaSwag drops from 73.00% (FP16) to 56.44% (OmniQuant) — a 16.56-point loss — while PIQA drops only from 77.47% to 66.15% (11.32 points) and BoolQ from 73.08% to 63.51% (9.57 points). This task-specific degradation pattern is consistent across methods (SmoothQuant shows similar relative drops) and suggests that commonsense reasoning tasks requiring multi-hop inference (HellaSwag) are more sensitive to quantization noise than knowledge-focused tasks (BoolQ, PIQA). The paper does not analyze this pattern explicitly, but the data is available for such analysis.

Instruction-Tuned Model Results

Figure 4 presents the Vicuna-bench results comparing W3A16g128 quantization methods on LLaMA-2-chat models, evaluated by GPT-4 pairwise comparison. On LLaMA-2-7b-chat, OmniQuant achieves a 50.0% win rate against AWQ (statistically tied), and an 80.3% win rate against RTN — meaning when GPT-4 is asked to compare two chatbot responses and pick the better one, it prefers OmniQuant's output over RTN's in 80.3% of non-tied comparisons. AWQ achieves a 69.4% win rate against RTN on the same model. On LLaMA-2-13b-chat, OmniQuant's advantage becomes clearer: it achieves a 54.4% win rate against RTN and 56.2% against AWQ, while AWQ actually underperforms RTN with a 46.6% win rate (i.e., RTN wins more often against AWQ than vice versa). This is notable because it shows AWQ's performance does not monotonically improve with model size in the instruction-tuned setting, while OmniQuant's does — consistent with the hypothesis that learned parameters adapt to model-specific characteristics that hand-crafted methods miss.

Real-Device Deployment Results

Table 3 reports memory and speed measurements on an NVIDIA A100-80G using MLC-LLM. For LLaMA-7B: FP16 requires 12.6GB weight memory and 14.4GB running memory, achieving 69.2 tokens/second. W4A16g128 reduces weight memory to 3.8GB and running memory to 5.7GB, with inference speed nearly doubling to 134.2 tokens/second. W3A16g128: 3.2GB weight, 5.1GB running, but speed drops to 83.4 tokens/second — faster than FP16 but slower than W4A16. W2A16g128: 2.2GB weight, 4.1GB running, 83.9 tokens/second. The paper explicitly notes that MLC-LLM's support for INT3/INT2 is "currently suboptimal, particularly for INT3" — the speed regression from W4A16 (134.2) to W3A16 (83.4) is not inherent to 3-bit computation but reflects the current state of kernel optimization. For LLaMA-13B: W4A16g128 achieves 91.3 tokens/second (vs. 52.5 FP16) with 7.0GB weight memory. For LLaMA-30B: W4A16g128 achieves 43.6 tokens/second (vs. 23.9 FP16) with 16.7GB weight memory — fitting on a single GPU where FP16 would require multiple. LLaMA-65B: FP16 runs out of memory (OOM), while W4A16g128 runs at 24.3 tokens/second with 33.0GB weight memory. Only weight-only quantization deployment is tested because W4A4 and W6A6 "lack out-of-the-box hardware support" — the paper cannot demonstrate activation-quantization speedups on real hardware.

Ablation Studies and Robustness Checks

Effect of each component (LWC and LET): Table A3 reports ablations on LLaMA-13B and OPT-13B. On LLaMA-13B W4A4, the full method (LWC+LET) achieves 10.87 WikiText2 PPL. Removing LWC (+LET only) degrades to 20.75 — LET pushes weight outliers that LWC is needed to handle. Removing LET (+LWC only) causes collapse to 5.4e3 — without LET, activation outliers remain unaddressed and LWC cannot compensate. Removing both (vanilla MinMax) gives 1.8e3. On OPT-13B, the pattern is similar but the LET contribution for weight-only quantization (W3A16) is more pronounced: LWC+LET achieves 10.87, removing LET degrades to 11.29 — a 0.42 PPL increase versus only 0.03 on LLaMA-13B (5.65 vs. 5.68 LWC-only). This is why LET is disabled for LLaMA weight-only quantization: the benefit is negligible relative to the training time cost.

Design choices of LET (Table A4): Removing the channel-wise shifting component (−shifting) degrades LLaMA-13B W4A4 from 10.87 to 11.47 — a 0.60 PPL increase. Removing the attention transformation (−attention) degrades to 11.34 — a 0.47 PPL increase. Both contributions are positive but the shifting component has a larger impact, consistent with the observation that activation channels have non-zero means that cause asymmetric quantization error. On OPT-13B, the relative importance is reversed: removing shifting degrades from 11.65 to 13.64 (1.99 PPL increase), while removing attention transformation degrades to 11.79 (0.14 PPL increase) — OPT activations apparently have more severe mean shifts than LLaMA's.

Impact of LET on each position (Table A5): On LLaMA-7B W4A4, the full method achieves 52.65% average accuracy. Removing LET from the [ln1, (q_proj, k_proj, v_proj)] pair causes the largest drop to 46.79% — a 5.86 percentage point decrease, confirming that LayerNorm outputs are the primary source of activation outliers. Removing [v_proj, out_proj] drops to 51.68%, removing [Q, K] attention transformation drops to 51.47%, and removing [ln2, fc1] drops to 51.04%. All four LET pairs contribute positively, but the first LayerNorm-to-QKV pathway dominates, consistent with the architecture's information flow where outlier channels originate at normalization layers and propagate into the attention projections.

Impact of LET initialization (Table A6): On LLaMA-7B W4A4, initializing scaling s as all ones (instead of SmoothQuant values) reduces average accuracy from 52.65% to 51.37% — a 1.28 point drop. Initializing shifting δ as all zeros (instead of channel means) reduces accuracy to 52.22% — a 0.43 point drop. Scaling initialization matters more because scaling handles the magnitude disparity across channels, which is the dominant quantization challenge; shifting handles the mean adjustment, which is a smaller effect. However, the fact that even with poor initialization (all ones, all zeros) OmniQuant still substantially outperforms SmoothQuant (38.41%) demonstrates that the gradient-based optimization can recover from suboptimal starting points.

Training time and convergence (Table A9): On LLaMA-7B, most quantization configurations converge within 20 epochs. W4A16: PPL improves from 6.29 (epoch 0, initialization only) to 5.87 (epoch 10) to 5.85 (epoch 20) — diminishing returns after 10 epochs. W3A16: 24.04 → 6.51 → 6.49. W4A4: 33.93 → 12.04 → 11.26 — continued improvement through 20 epochs. W2A16 is the outlier: 1.1e5 → 27.49 (epoch 10) → 17.46 (epoch 20) → 14.77 (epoch 80) — 2-bit quantization requires extended optimization to find good clipping thresholds, with meaningful improvements continuing through 80 epochs. The paper settles on 20 epochs for all configurations except W2A16 at 40 epochs, trading off training time versus marginal gains.

Calibration dataset robustness (Table A10 and A11): Table A10 tests calibration on WikiText2, C4, and Pile. For LLaMA-7B W3A16, WikiText2 PPL varies from 6.47 (calibrated on WikiText2) to 6.67 (C4) to 6.69 (Pile), with a variance of only 0.009. For W4A4, the range is 11.23 (WikiText2) to 12.17 (C4) to 12.04 (Pile), with variance 0.17. The cross-dataset robustness is strong — calibrating on C4 and evaluating on WikiText2 yields 6.67 (vs. 6.47 when calibrating on WikiText2), only a 0.20 PPL penalty. Table A11 tests calibration data volume: W3A16 achieves 6.47 PPL with 16, 32, or 128 samples — the method saturates immediately. W4A4 improves from 11.56 (16 samples) to 11.48 (32) to 11.40 (64) to 11.23 (128) — diminishing but positive returns up to 128 samples. This robustness to both distribution and volume of calibration data is a consequence of the small parameter count — with only thousands of learnable parameters, overfitting is unlikely even with limited data.

Softmax quantization (Table A7): The paper keeps Softmax at 16-bit by default but explores quantization: 8-bit Softmax yields 51.93% average accuracy (vs. 52.65% at 16-bit), 6-bit yields 51.70%, and 4-bit yields 48.52%. The degradation at 8-bit and 6-bit is modest, suggesting that block-wise calibration can compensate for Softmax quantization error at moderate bit-widths, but the long-tailed distribution becomes problematic at 4-bit.

Simultaneous vs. iterative training of LWC and LET (Table A8): On LLaMA-7B W4A4, simultaneous training achieves 52.65% average accuracy. Iterative training by iterations (alternating LWC then LET each step) achieves only 50.91% — a 1.74 point drop. Iterative training by epochs (train one for a full epoch, then the other) achieves 52.06%. Doubling the epochs for iterative training to 40 reaches 52.50%, still below simultaneous 20-epoch training at 52.65%. This demonstrates that LWC and LET reinforce each other during optimization — joint gradient steps find better solutions than alternating optimization because the optimal clipping threshold depends on the current equivalent transformation and vice versa.

LWC compared to other clipping methods (Table A14 and Figure A5): The paper replaces LWC with PACT (direct threshold learning) and LSQ (direct step-size learning) in the same optimization pipeline. For LLaMA-7B W3A16 (weight-only), PACT achieves 6.95 PPL and LSQ achieves 6.63, compared to LWC's 6.47 — both improve over MinMax (25.73) but underperform LWC. For W4A4 (weight-activation), the gap widens: PACT achieves 18.25, LSQ 15.03, and LWC 11.26. Figure A5 visualizes why: during W4A4 training, LET shifts weight distributions dramatically (the weight range of a specific output channel fluctuates), and while LWC's learned clipping strengths track these changes automatically, PACT and LSQ's absolute thresholds fail to adapt, leading to oscillatory optimization. This ablation directly validates the design rationale behind the relative parameterization.

Combination variants (Table A2): Testing alternative combinations on LLaMA-7B W4A4: LET alone (no LWC) achieves 16.97 PPL and 48.83% accuracy. LET + grid-searched weight clipping achieves 15.82 PPL and 49.59% accuracy — better than LET alone but worse than joint learning. SmoothQuant + LWC (replacing LET with SmoothQuant's hand-crafted scaling) achieves 15.80 PPL and 50.15% accuracy — interestingly better than LET + grid-searched clipping, suggesting that learning the clipping (even with suboptimal transformations) is more valuable than having optimal transformations with crude clipping. The full LET + LWC achieves 12.87 PPL and 52.65% accuracy — the joint learning of both components outperforms all hybrid combinations by a substantial margin.

Comparisons with mixed-precision and non-uniform methods (Table A15): OmniQuant is uniform integer quantization, so the paper compares against SpQR (mixed-precision) and SqueezeLLM (non-uniform + sparse). On LLaMA-7B at ~4.16 average bits (OmniQuant's W4A16g128), SqueezeLLM achieves 5.79 WikiText2 PPL (4.07 bits) and 5.77 (4.27 bits), while OmniQuant achieves 5.77 (4.16 bits) — comparable performance. At ~3.15 average bits (OmniQuant's W3A16g128), SqueezeLLM achieves 6.20 (3.05 bits) and 6.13 (3.24 bits), while OmniQuant achieves 6.15 (3.15 bits). OmniQuant is slightly worse than SqueezeLLM at matched bit-widths but the differences are small (0.02–0.05 PPL), and OmniQuant's uniform integer format provides simplicity for hardware deployment — a practical tradeoff that the paper acknowledges explicitly.

MMLU results (Table A16): On LLaMA-7B MMLU (FP16: 38.41%), OmniQuant achieves 37.50% at W4A16g128 (0.91 point drop), 35.60% at W3A16g128 (2.81 point drop), 26.03% at W2A16g128 (12.38 point drop), and 26.93% at W4A4 (11.48 point drop). GPTQ achieves 35.39%, 30.53%, 23.83% respectively — OmniQuant consistently outperforms. AWQ achieves 37.71% at W4A16g128 (slightly better than OmniQuant's 37.50%) but drops to 35.43% at W3A16g128 and 22.58% at W2A16g128. This shows AWQ excels at 4-bit but degrades faster than OmniQuant at lower bit-widths — consistent with the pattern that hand-crafted scaling works adequately for moderate compression but cannot adapt to extreme regimes.

Asymmetric bit quantization (Table A17): The paper tests configurations where weights and activations use different bit-widths on LLaMA-7B. W4A8: 62.40% average accuracy (vs. 64.09% FP16) — a modest 1.69 point drop. W4A6: 61.64% — an additional 0.76 point drop. W8A4: 53.46% — a substantial 10.63 point drop. This asymmetry reveals that activation quantization is more challenging than weight quantization: keeping activations at 8-bit while quantizing weights to 4-bit (W4A8) preserves most of the accuracy, but quantizing activations to 4-bit while keeping weights at 8-bit (W8A4) causes significant degradation. This aligns with the paper's framing — activation outliers are the primary obstacle — and suggests that future hardware supporting mixed-precision matrix multiplication could benefit from asymmetric quantization schemes.

Perplexity-vs-model-size scaling laws (Figure A6): The paper presents a performance overview showing the tradeoff between model size and perplexity for quantized models. OmniQuant's W3A16g128 quantization of a given model size achieves perplexity comparable to the next-size-up model at W4A16g128 — for instance, LLaMA-13B at 3-bit achieves similar perplexity to LLaMA-7B at 4-bit, but with 0.33× the memory. The paper claims this means OmniQuant "delivers a reduction of 0.81 in perplexity for an equivalent model size and achieves the same perplexity with only 0.33× of the model size." Figure A4 further shows the bit-level scaling laws for perplexity: OmniQuant's 3-bit curve lies close to prior methods' 4-bit curve, demonstrating that improved quantization effectively buys back roughly one bit-width of precision.

Critical Assessment

Do the experiments support the central claim: "OmniQuant achieves QAT-level performance with PTQ-level efficiency"?

This claim has two parts — quality and cost — and the evidence is strong for quality but less complete for cost.

On quality: The comparison against LLM-QAT (Table 2) directly supports the claim for LLaMA-7B W4A4: OmniQuant achieves 52.65% average accuracy versus LLM-QAT's best variant at 46.43%. This is a clean win — OmniQuant is both better and cheaper. However, this is a single data point. The paper does not compare against QAT for other bit-widths (W6A6, W2A16), other model sizes (13B, 30B, 65B), or other model families (OPT). LLM-QAT may perform differently at other scales or configurations, and without those comparisons, the claim that OmniQuant "achieves QAT performance" is only demonstrated for one specific setting. The paper also does not compare against QAT for weight-only quantization, where the advantage of learning quantization parameters over heuristics is smaller (the W4A16g128 gap between OmniQuant and GPTQ is only 0.02–0.04 PPL). A QAT method at W4A16 might achieve near-lossless compression that PTQ cannot match, and OmniQuant might not close that gap. The claim should therefore be understood as: on the specific configurations tested, OmniQuant matches or exceeds the one QAT baseline available.

On cost: The paper claims PTQ-level efficiency, measuring 1–16 hours on a single A100 with 128 samples. This is indeed far cheaper than QAT (100k samples, hundreds of GPU hours). However, it is approximately 5× more expensive than GPTQ (the paper acknowledges this in Appendix A4), and the paper does not provide a detailed runtime comparison against AWQ or SmoothQuant. For a practitioner choosing between OmniQuant and GPTQ at W4A16g128, the 5× training time increase buys only ~0.03 PPL improvement — a marginal return that may not justify the cost. The value proposition of OmniQuant is strongest at low bit-widths (W2A16, W4A4) where GPTQ/AWQ fail entirely, making the cost comparison irrelevant (there is no cheaper working alternative). The paper would be strengthened by a table showing cost-vs-quality tradeoffs across bit-widths explicitly, so practitioners can decide when OmniQuant's extra training time is justified.

Do the experiments support the claim that "relative parameterization enables stable joint optimization"?

The evidence for this claim is strong and multi-faceted. Table A14 and Figure A5 provide direct causal evidence: replacing LWC's relative formulation with absolute formulations (PACT, LSQ) causes optimization instability under LET's distribution shifting, while LWC remains stable. The W4A4 results in Table A14 (LWC: 11.26 vs. LSQ: 15.03 vs. PACT: 18.25) show that the effect is large. Table A8 shows that simultaneous joint training outperforms iterative training, confirming that the two components mutually reinforce each other during optimization. The fact that OmniQuant converges on as few as 16 calibration samples (Table A11) and generalizes across calibration datasets (Table A10) provides indirect evidence that the optimization landscape is well-behaved — an unstable parameterization would likely show more sensitivity to data quantity and distribution.

However, the paper does not provide a systematic evaluation of why relative parameterization works beyond the PACT/LSQ comparison. There is no ablation testing different relative formulations (e.g., learning clipping as a percentile of the distribution rather than as a fraction of min/max), no measurement of gradient variance under different parameterizations, and no analysis of whether the benefit is specific to the weight-activation setting (where LET shifts distributions) or also applies to weight-only (where distributions are static). The claim that relative parameterization is the enabler is supported by the PACT/LSQ failure, but the paper does not rule out alternative explanations (e.g., better initialization, different effective learning rates, different optimization dynamics unrelated to the relative-vs-absolute distinction).

Do the perplexity results on WikiText2 and C4 constitute sufficient evaluation of language modeling quality?

Perplexity is the standard metric in the quantization literature, and the paper follows the evaluation protocol established by GPTQ and AWQ. The consistency of results across WikiText2, PTB, and C4 (Tables A19, A21, A22, A24) suggests that improvements are not dataset-specific. However, perplexity is known to be an imperfect proxy for downstream task performance — a model can have good perplexity but poor generation quality (e.g., repetitive outputs, factual errors). The paper partially addresses this with the zero-shot task evaluations (Table 2) and the Vicuna benchmark (Figure 4), which show that OmniQuant's improvements transfer to accuracy-based and preference-based evaluation. But the zero-shot evaluation is limited to 6 tasks, the Vicuna evaluation is limited to 80 questions on two model sizes, and there is no evaluation of open-ended generation quality (fluency, coherence, diversity). A stronger evaluation would include generation benchmarks like AlpacaEval or MT-Bench, especially for the instruction-tuned models where quality of natural language output matters more than multiple-choice accuracy.

Do the real-device speed measurements (Table 3) demonstrate practical acceleration from low-bit quantization?

The speed measurements reveal an important practical limitation that the paper acknowledges but does not fully resolve. W4A16g128 achieves excellent speedups (1.8–1.9× over FP16 for 7B–30B). However, W3A16g128 is slower than W4A16g128 on LLaMA-7B (83.4 vs. 134.2 tokens/second), and W2A16g128 is only marginally faster at 83.9. This means the memory savings of going from 4-bit to 2-bit (3.8GB → 2.2GB) come with a speed regression relative to 4-bit. The paper attributes this to suboptimal kernel support in MLC-LLM for non-4-bit integer formats. This is an important caveat: OmniQuant can produce accurate 2-bit and 3-bit models, but the inference speed benefits depend on hardware and kernel support that may not exist yet. The paper's deployment results are therefore better characterized as demonstrating memory reduction rather than speed improvement for the lower bit-widths. Furthermore, the paper does not deploy weight-activation quantization on real hardware at all ("W4A4 and W6A6 quantization methods lack out-of-the-box hardware support"), so all speed and memory measurements are for weight-only quantization only. The practical benefits of OmniQuant's strongest results (W4A4 zero-shot accuracy) remain theoretical until hardware catches up.

Are the comparisons against baselines fair and complete?

Several baseline comparisons merit scrutiny:

GPTQ comparison at W2A16: The paper reports GPTQ as producing NaN on LLaMA-2-70B W2A16g128 (Table 1). This appears to be a failure of GPTQ's Hessian-based weight compensation, but the paper does not investigate whether GPTQ can be stabilized (e.g., by adjusting the damping parameter or using a different calibration set). The comparison is fair in the sense that OmniQuant works out of the box where GPTQ fails, but the failure may be fixable with hyperparameter tuning rather than algorithmic superiority.

AWQ comparison at low bit-widths: AWQ produces perplexities of 280,000+ on several W2A16 configurations (Table 1). AWQ was designed primarily for 4-bit quantization with group-wise scaling, and its grid search over a heuristic proxy may be fundamentally unsuited to 2-bit. The paper's comparison is valid — OmniQuant works where AWQ does not — but AWQ's authors might argue this is outside the method's intended operating range.

LLM-QAT comparison only at W4A4, one model size: As noted above, the QAT comparison is limited to a single configuration. LLM-QAT may perform better at higher bit-widths (where OmniQuant's advantage over PTQ is smaller) or at larger model sizes (where QAT's full weight tuning might better preserve performance). The paper's claim of surpassing QAT would be stronger with results across the full quantization grid.

Missing baseline: OmniQuant without LET but with more calibration data or epochs. The paper shows that LET is essential for weight-activation quantization (Table A3: removing LET causes collapse to 5.4e3 on W4A4). But could this gap be closed by simply training LWC-only for more epochs or with more data? The paper's convergence analysis (Table A9) shows W4A4 with LWC+LET saturates after 20 epochs, but the collapse without LET might be recoverable with more optimization — the paper does not test this, making the LET contribution potentially overstated if extended training could compensate.

Are there experiments that would have strengthened the paper but were not run?

End-to-end latency benchmarks for weight-activation quantization: The paper demonstrates that W4A4 models can be created with good accuracy but cannot deploy them to show speedups. This is a hardware limitation, not a methodological gap, but it means the practical value proposition of OmniQuant's best results is unvalidated.

Scaling to larger calibration datasets: The paper shows that 16–128 samples suffice. But does performance improve with more data, even marginally? The W4A4 calibration study (Table A11) shows improvement from 11.56 to 11.23 when going from 16 to 128 samples, with unknown behavior beyond 128. A sweep to 512 or 1024 samples would establish whether the saturation is real or whether OmniQuant plateaus prematurely relative to QAT's massive data appetite.

Ablation of the block-wise sequential optimization: The paper claims block-wise optimization prevents error accumulation, but there is no comparison against optimizing all blocks jointly (even if only for a small model like OPT-125M). The relative contribution of block-wise vs. joint optimization to final performance is therefore unquantified.

Evaluation on generative tasks beyond multiple-choice: A qualitative comparison of generated text from FP16 vs. OmniQuant models (e.g., completions of WikiText2 prompts) would complement the quantitative perplexity and accuracy metrics. The Vicuna benchmark provides some generation quality signal through GPT-4 judgments, but only for instruction-following, not for open-ended text generation.

Cross-family generalization of learned parameters: The paper calibrates each model separately. An interesting experiment would be to calibrate on LLaMA-7B and apply the learned clipping strengths and transformation factors to LLaMA-13B (with appropriate dimensionality mapping). This would test whether OmniQuant learns generic quantization strategies or model-specific ones, and whether calibration cost could be amortized across model sizes.

6. Limitations and Trade-offs

6.1 The Learnable Equivalent Transformation Is Disabled for LLaMA Weight-Only Quantization, Limiting Its Generality as a Unified Framework

The assumption or constraint. OmniQuant is presented as a unified framework that handles both weight-only and weight-activation quantization through the same LWC + LET pipeline. However, the paper reveals in Section 4.1 and Table A3 that LET provides negligible benefit for LLaMA weight-only quantization — a 0.03 perplexity improvement on LLaMA-13B W3A16 (5.68 → 5.65) — and is therefore disabled in practice. The paper states:

"for weight-only, both [LWC and LET] are used for OPT, but only the clipping is for LLaMA, as Table A3 shows negligible benefits from the equivalent transformation for LLaMA."

The consequence. The claimed "versatile quantization framework for both weight-only and weight-activation quantization" (Section 1) is not equally versatile across model families. For LLaMA weight-only quantization, OmniQuant effectively reduces to LWC alone — which is learnable weight clipping operating on untransformed weights. This means the full LWC+LET optimization pipeline is only essential for two cases: (a) weight-activation quantization on any model family, and (b) weight-only quantization on OPT-like architectures with pronounced weight outliers. For LLaMA weight-only — arguably the most common deployment scenario given that W4A16 weight-only quantization has mature hardware support while W4A4 lacks it (Section 4.5) — OmniQuant's advantage over GPTQ/AWQ comes solely from LWC's learned clipping thresholds, which at 4-bit yields only a 0.02–0.04 perplexity improvement (Table 1, W4A16g128). A practitioner deploying LLaMA-7B W4A16g128 must decide whether a 5× training time increase over GPTQ (Appendix A4) is justified for a ~0.03 PPL gain.

What evidence exists in the paper. Table A3 directly shows the LET contribution breakdown: on LLaMA-13B W3A16, the gap between LWC-only (5.68) and LWC+LET (5.65) is 0.03 PPL, versus a 1.29 PPL gap on OPT-13B W3A16 (LWC-only: 11.29, LWC+LET: 10.87). Table 1 shows that OmniQuant's W4A16g128 advantage over GPTQ is 0.02–0.04 PPL across LLaMA models. The paper does not report an ablation isolating LWC's contribution at W4A16g128 specifically, but the pattern from Table A3 suggests that with LET disabled and only LWC active, the gap over existing methods at 4-bit weight-only is minimal.

Mitigation status. The paper acknowledges this asymmetry implicitly by disabling LET for LLaMA weight-only, but does not frame it as a limitation of the unified framework claim. The architecture-specific nature of the benefit is presented as a practical optimization ("for faster training") rather than as evidence that the framework's generality is partial. There is no investigation into why LLaMA weights respond differently to equivalent transformation, and no suggestion for making LET beneficial for LLaMA weight-only. A practitioner cannot know in advance whether their model family will benefit from the full OmniQuant or only the LWC subset without running the ablation themselves.

6.2 Weight-Activation Quantization Speedups Cannot Be Demonstrated on Real Hardware

The assumption or constraint. The paper's strongest results are in weight-activation quantization — W4A4 zero-shot accuracy improvements of 4–12 percentage points over prior PTQ methods (Table 2) — which should theoretically provide both memory savings (from weight compression) and computational speedups (from low-bit matrix multiplication). However, the paper explicitly states in Section 4.5:

"we only explore the deployment of weight-only quantization in this study due to that W4A4 and W6A6 quantization methods lack out-of-the-box hardware support."

The consequence. All real-device speed and memory measurements (Table 3) are for weight-only quantization only. The paper cannot demonstrate that a W4A4 model — its flagship result — actually runs faster than an FP16 or W4A16 model on any real GPU. This is a critical gap between algorithmic achievement and practical deployment: OmniQuant proves that accurate W4A4 models can be created, but not that they can be accelerated. The speed regression observed when moving from W4A16g128 to W3A16g128 on LLaMA-7B (134.2 → 83.4 tokens/second, Table 3) underscores that lower bit-widths do not automatically translate to faster inference — kernel support is the bottleneck. If W4A4 kernels were similarly suboptimal, the actual speedup might be negative despite the theoretical 4× reduction in matrix multiplication cost. This is a deployment-facing limitation: the paper sells memory reduction and speed improvement as dual benefits, but only the former is verified on hardware.

Furthermore, the memory measurements in Table 3 include only weight storage ("Weights Memory"). The paper does not report the memory savings from activation quantization — since activations are quantized per-token at runtime, they require less SRAM/register space during computation, which should reduce the "Running Memory" gap. Without W4A4 hardware deployment, the activation memory benefit remains unmeasured.

What evidence exists in the paper. Table 3 is the sole real-device result and is explicitly weight-only. Section 4.5 states the hardware support limitation directly. The paper presents no simulated throughput analysis, no roofline model estimates, and no projections of W4A4 speedup based on theoretical compute intensity improvements. The W3A16 speed regression (83.4 vs. 134.2 tokens/sec for W4A16) provides a cautionary data point: moving to lower bit-widths can reduce throughput if kernel support lags.

Mitigation status. The paper acknowledges the limitation explicitly and frames hardware support as future work ("Enhancements to INT3/INT2 quantization speed are in our future roadmap"). However, the abstract and introduction present the W4A4 results without qualifying that the speed benefits are theoretical. A practitioner reading the abstract might reasonably assume that W4A4 deployment is as practical as W4A16, which the paper cannot substantiate. No projections, estimated speedups, or hardware roadmaps are provided to help practitioners decide whether to invest in OmniQuant for weight-activation quantization today versus waiting for kernel support.

6.3 Calibration Data Distribution and Volume Robustness Are Only Tested on Perplexity, Not on Downstream Task Accuracy

The assumption or constraint. The paper claims robustness to calibration data distribution and volume based on perplexity measurements (Tables A10, A11). For LLaMA-7B W3A16, WikiText2 perplexity varies from 6.47 to 6.69 when calibrating on different datasets, with variance of 0.009. For W4A4, the range is 11.23 to 12.17 with variance 0.17. The paper concludes that "the variance in perplexity across diverse calibration datasets is marginal" and that OmniQuant "converges with as few as 16 samples."

The consequence. Perplexity is a distributional match metric — it measures how well the model predicts the next token on average. Downstream tasks require specific capabilities (factual recall, logical reasoning, commonsense inference) that may be more sensitive to quantization artifacts than average token prediction. The paper evaluates zero-shot accuracy only on models calibrated with the default WikiText2-128 setting. If a practitioner calibrates on a different dataset (e.g., domain-specific text) or with fewer samples (e.g., 16 instead of 128), the zero-shot accuracy — which is arguably the more deployment-relevant metric — might degrade even if perplexity remains stable. The robustness claim therefore only partially transfers: we know calibration data choice doesn't substantially affect language modeling perplexity, but we do not know whether it affects the model's ability to answer questions correctly.

What evidence exists in the paper. All zero-shot accuracy results (Table 2, Figure 4, Tables A16, A17, A18) use the default WikiText2-128 calibration. Table A10 and A11 test calibration robustness only on WikiText2 and C4 perplexity for LLaMA-7B W3A16 and W4A4. There is no zero-shot accuracy evaluation with alternative calibration datasets or reduced sample counts. The paper does not report whether calibrating on C4 and evaluating on downstream tasks — which would simulate a practitioner using their own domain data — preserves the accuracy gains over baselines.

Mitigation status. Not addressed. The paper treats perplexity stability as sufficient evidence of robustness, without acknowledging that downstream task performance could be more brittle. A single experiment — e.g., reporting zero-shot accuracy for a model calibrated on C4 instead of WikiText2 — would substantially strengthen the claim. The paper also does not discuss whether the calibration data should ideally match the deployment data distribution, which is a standard consideration in PTQ that OmniQuant's block-wise optimization does not obviously eliminate.

6.4 The Optimization Introduces a 5× Training Time Overhead Compared to GPTQ, with Diminishing Returns at Higher Bit-Widths

The assumption or constraint. OmniQuant is positioned as maintaining "PTQ efficiency" while achieving QAT accuracy. The paper reports training times in Table A12: LLaMA-7B weight-only takes 1.1 hours, LLaMA-13B takes 2.2 hours, scaling linearly. However, the paper acknowledges (Appendix A4) that "our proposed method necessitates a training time that is approximately 5× greater than GPTQ."

The consequence. The 5× cost increase over GPTQ must be justified by commensurate quality improvements, and this justification varies dramatically by configuration. At W4A16g128 — the most deployment-relevant weight-only setting due to mature hardware support — OmniQuant improves WikiText2 perplexity over GPTQ by 0.02–0.04 PPL (Table 1). For LLaMA-7B, this means spending 1.1 GPU-hours instead of ~0.2 GPU-hours to gain ~0.03 PPL. A practitioner quantizing a single model once may accept this cost, but teams quantizing many model variants or iterating on quantization configurations will see the overhead compound. At W4A4 and W2A16 — where GPTQ outright fails — the 5× cost is irrelevant because there is no cheaper working alternative. But the paper does not provide guidance on when the extra training time is justified versus when GPTQ or AWQ is sufficient, leaving practitioners to discover the cost-quality knee points through their own experimentation.

Furthermore, the 5× figure is relative to GPTQ specifically, which is itself an optimization-based PTQ method (using Hessian-based weight compensation, not gradient descent). The paper does not compare training time against AWQ (grid search, likely faster than GPTQ) or SmoothQuant (instantaneous, hand-crafted). For weight-activation quantization, SmoothQuant requires zero optimization time — it applies a fixed formula to compute scaling factors. OmniQuant's 1.6 hours for LLaMA-7B W4A4 (Table A12) is therefore not 5× but infinitely more expensive than SmoothQuant's zero-cost transformation. Of course, SmoothQuant achieves only 38.41% accuracy at W4A4 versus OmniQuant's 52.65% (Table 2), so the cost is well-justified — but the framing of "PTQ efficiency" is misleading when the baseline it compares against is GPTQ's optimization cost, not the broader PTQ landscape that includes truly training-free methods.

What evidence exists in the paper. Table A12 provides training times across model sizes. The 5× GPTQ comparison is mentioned in Appendix A4. Table 1 shows the quality gap between OmniQuant and GPTQ across bit-widths — the cost-quality tradeoff can be inferred but is not explicitly analyzed. The paper does not include training time for baselines (GPTQ, AWQ) in the same table, nor does it plot a cost-vs-quality Pareto frontier that would guide practitioners.

Mitigation status. The paper acknowledges the training time increase but frames it as acceptable because "it remains markedly faster than QAT methods, which demand hundreds of GPU hours." This is true but sidesteps the more practical comparison: OmniQuant vs. other PTQ methods that are even cheaper. No guidance is provided on configuration selection based on cost tolerance. The data efficiency results (Table A11, convergence with 16 samples) hint that training time could potentially be reduced by using fewer epochs or samples without significant quality loss, but this tradeoff is not systematically explored.

6.5 Quantization Quality Is Evaluated Primarily on Perplexity and Multiple-Choice Accuracy, with Minimal Assessment of Generative Coherence or Factual Reliability

The assumption or constraint. The paper evaluates quantized models through two lenses: perplexity on language modeling benchmarks (WikiText2, PTB, C4) and accuracy on zero-shot multiple-choice tasks (PIQA, ARC, BoolQ, HellaSwag, Winogrande). Instruction-tuned models are additionally evaluated on the Vicuna benchmark (80 questions, GPT-4 judged). The paper does not evaluate open-ended generation quality, factual accuracy of generated statements, or coherence in long-form text production.

The consequence. Perplexity improvements do not guarantee that the quantized model produces useful text. A model with 5.74 perplexity (OmniQuant LLaMA-7B W4A16g128, Table 1) versus 5.83 (GPTQ) might generate equally coherent completions, or it might produce text that is locally more probable but globally less consistent. Multiple-choice accuracy measures the model's ability to rank pre-written answers, not to generate correct answers from scratch — a model that achieves 52.65% on zero-shot tasks at W4A4 (Table 2) might still produce nonsensical completions when asked to continue a paragraph. The Vicuna benchmark provides some signal about conversational quality through GPT-4 judgments, but with only 80 questions and reliance on an LLM judge whose own evaluation criteria are not calibrated for quantization artifacts, it is a coarse measure.

This limitation is particularly acute for the instruction-tuned model results (Figure 4), where the central claim is that OmniQuant preserves chatbot quality. The Vicuna benchmark tests the model's ability to produce preferred responses in a pairwise comparison, but does not measure whether the responses are factually correct, whether the model hallucinates more after quantization, or whether the diversity of responses collapses (a known failure mode of aggressive quantization). A practitioner deploying a quantized chatbot needs to know not just that its responses are preferred over RTN-quantized responses, but that it doesn't introduce new failure modes — repeated phrases, factual errors, or topic drift — that perplexity and win rates do not capture.

What evidence exists in the paper. The Vicuna evaluation (Figure 4) is the closest the paper comes to generation quality assessment, but the 80-question benchmark with GPT-4 judging is primarily measuring instruction-following and response quality, not factual accuracy or generation diversity. There is no human evaluation, no measurement of hallucination rates, and no analysis of generated text properties (repetition, diversity, length distribution). The zero-shot task results (Table 2) are multiple-choice only. The paper does not include example completions comparing FP16 and quantized model outputs, which would give qualitative insight into failure modes.

Mitigation status. The paper does not acknowledge this as a limitation. The evaluation protocol follows the conventions established by GPTQ, AWQ, and SmoothQuant — all of which rely primarily on perplexity and zero-shot accuracy — so the paper is consistent with prior work but does not advance beyond it. Given that OmniQuant pushes to lower bit-widths than prior PTQ methods, the absence of generative quality evaluation is more consequential: the failure modes at W2A16 or W4A4 may be qualitatively different from those at W4A16, and perplexity alone cannot distinguish between "slightly noisier" and "subtly incoherent" degradation.

6.6 The Method Provides No Mechanism for Per-Layer or Per-Channel Heterogeneity in Quantization Sensitivity

The assumption or constraint. OmniQuant applies the same quantization bit-width uniformly to all layers and all channels within a layer. LWC learns per-channel clipping strengths, but the quantization bit-width itself (e.g., 4 bits for all weights in W4A4) is fixed across the entire model. The paper does not explore mixed-precision quantization where more sensitive layers or channels receive higher precision. This is an intentional design choice — the paper emphasizes that OmniQuant produces "pure uniform integer quantized models" compatible with standard hardware — but it means the method cannot adapt to the well-documented phenomenon that different layers and channels exhibit different sensitivity to quantization error.

The consequence. Uniform bit-width allocation leaves performance on the table. Prior work (SpQR, SqueezeLLM, OWQ) has demonstrated that a small fraction of weights — often those corresponding to high-magnitude activation channels — are disproportionately important for model quality, and protecting them with higher precision or sparse storage yields better perplexity-vs-memory tradeoffs than uniform compression. OmniQuant's LWC partially addresses this by learning to clip outlier channels differently than regular channels (Figure A1 shows a long-tailed clipping distribution for W2A16g128), but clipping within the same bit-width is a cruder tool than allocating additional bits to sensitive weights. The comparison in Table A15 shows that at matched average bit-widths (~4.16 bits), SqueezeLLM (non-uniform quantization) achieves slightly better WikiText2 perplexity than OmniQuant's uniform W4A16 on LLaMA-7B (5.79 vs. 5.77 at 4.07 bits, 5.77 vs. 5.77 at 4.27 bits) and LLaMA-13B (5.17 vs. 5.17). At ~3.15 bits, SqueezeLLM achieves 6.20 vs. OmniQuant's 6.15 on LLaMA-7B. The differences are small, suggesting that OmniQuant's learned clipping partially compensates for the lack of mixed precision, but the uniform constraint is a fundamental ceiling.

What evidence exists in the paper. Table A15 directly compares OmniQuant against SpQR and SqueezeLLM at matched bit-widths, showing OmniQuant is competitive but not dominant. The paper frames uniform quantization as a practical advantage (hardware compatibility) and acknowledges the tradeoff ("our focus on uniform (INT) quantization provides simplicity and flexibility"). However, there is no ablation testing whether allowing even 1–2% of weights to be stored at higher precision would close the remaining gap to FP16 or improve W2A16 results further.

Mitigation status. The paper treats uniform quantization as a feature, not a limitation, because it ensures deployability on standard hardware (MLC-LLM, CUDA kernels expecting uniform INT formats). This is a legitimate engineering tradeoff, but the paper does not quantify how much performance is sacrificed for this simplicity. A mixed-precision ablation — e.g., protecting the top 1% of weights by activation magnitude at FP16 and quantizing the rest with OmniQuant — would help practitioners understand the cost of the uniformity constraint. The paper's explicit comparison against mixed-precision methods (SpQR, SqueezeLLM) partially addresses this by showing OmniQuant is competitive, but the comparison is at matched average bit-widths — OmniQuant achieves similar quality with a simpler format, but a hybrid approach combining OmniQuant's learned quantization parameters with mixed-precision allocation might outperform both.

7. Implications and Future Directions

How This Work Changes the Landscape

OmniQuant changes the landscape of LLM quantization not by proposing a single new technique that dominates all others, but by demonstrating that the prevailing dichotomy between hand-crafted PTQ (cheap but brittle at low bit-widths) and full QAT (accurate but prohibitively expensive) is a false choice enforced by parameterization decisions, not by any fundamental limitation of post-training optimization. This is a methodological reframing rather than a paradigm shift: the field already knew that gradient-based optimization could improve quantization quality (AdaRound, BRECQ), but the assumption — unstated but universally adopted — was that gradients in PTQ must flow through the model weights themselves, making the approach intractable for billion-parameter models. OmniQuant breaks this assumption by showing that learning only the quantization parameters — clipping thresholds and equivalent transformation factors, totaling a few thousand scalars per block — can match or exceed QAT while using 128 samples and a single GPU.

The shift is from asking "how do we hand-engineer better quantization heuristics?" to asking "what is the minimal set of learnable parameters that control the quantization process, and how should they be parameterized for stable optimization under distribution shift?" This opens a design space that prior work did not recognize existed: differentiable PTQ methods that learn the knobs of quantization rather than the weights themselves. The specific parameterization innovations — relative clipping strengths (γ, β) rather than absolute thresholds, and jointly learned channel-wise scaling and shifting with attention-aware extension — are the concrete instantiations of this principle, but the principle itself is what generalizes.

The paper reconciles several tensions in the prior literature. First, it explains why SmoothQuant and Outlier Suppression+ — both of which apply mathematically equivalent transformations to handle activation outliers — plateau at moderate bit-widths: their hand-crafted or grid-searched parameters cannot adapt to the complex interactions between weight and activation quantization that dominate at W4A4 and below. OmniQuant's gradient-based optimization of these same transformation types achieves 52.65% zero-shot accuracy on LLaMA-7B W4A4 versus SmoothQuant's 38.41% (Table 2) — a 14.24 percentage point gap that is entirely attributable to the optimization procedure, not the transformation concept. Second, it explains the conflicting results around weight clipping in quantization: PACT and LSQ work in QAT settings where weights adapt to the quantizer, but fail in PTQ settings where weights are frozen and equivalent transformations shift their distributions (Table A14, Figure A5). LWC's relative parameterization resolves this by making the clipping threshold invariant to the absolute scale changes that LET introduces.

A practical consequence of this reframing is that research attention should shift from designing new quantization heuristics to designing new learnable quantization parameterizations. The paper demonstrates that a well-parameterized optimization over a few thousand scalars can outperform complicated hand-crafted schemes (AWQ's grid search over activation-aware scaling, GPTQ's Hessian-based weight compensation) and even full model retraining (LLM-QAT). This suggests that future work on quantization should invest in what is learnable — which aspects of the quantization process can be parameterized as differentiable functions with well-behaved optimization landscapes — rather than in increasingly sophisticated closed-form or search-based heuristics. In particular, the paper's finding that relative parameterizations (scaling the existing min/max rather than learning absolute thresholds) are more stable under distribution shift provides a concrete design principle for future learnable quantization components.

The paper also establishes viability of uniform 2-bit and 3-bit quantization as a practical deployment target, which was not widely accepted before. GPTQ's 5,500 perplexity on LLaMA-13B W2A16 and AWQ's 280,000 (Table 1) had created a narrative that 2-bit quantization was fundamentally broken for LLMs except with specialized representations (mixed-precision, non-uniform, sparse-quantized). OmniQuant's 13.21 perplexity at W2A16 and 7.93 at W2A16g128 on the same model — while degraded from FP16's 5.09 — demonstrates that uniform 2-bit models can remain coherent. This finding shifts the conversation from "can we quantize below 4 bits?" to "how do we optimize the tradeoff between bit-width, group size, and calibration for uniform low-bit quantization?" The scaling law visualizations in Figures A4 and A6 reinforce this: OmniQuant's 3-bit quantization achieves perplexity-vs-model-size tradeoffs comparable to prior methods' 4-bit, effectively buying back roughly one bit-width of precision through better optimization.

Finally, the paper's deployment results (Table 3) introduce a sobering practical note: algorithmic advances in quantization currently outpace hardware support. OmniQuant produces accurate W3A16 and W2A16 models, but MLC-LLM's INT3/INT2 kernels are slower than INT4 kernels (83.4 vs. 134.2 tokens/second on LLaMA-7B) — meaning the memory savings of going below 4 bits come at a speed penalty. And W4A4 quantization, the paper's flagship result, cannot be deployed on real hardware at all because "out-of-the-box hardware support" is missing. This creates a feedback loop: the paper demonstrates that low-bit weight-activation quantization is algorithmically viable, which should incentivize hardware and kernel development to make it practically viable, but until that development occurs, the paper's strongest results remain theoretical. The gap between algorithmic capability and hardware readiness is itself a finding that should influence research prioritization: algorithmic work on W4A4 and below is ahead of its time, and the near-term practical impact of quantization research may come from weight-only methods (where hardware support is mature) rather than weight-activation methods (where it is not).

Follow-Up Research This Work Enables

Quantization-aware kernel development for W4A4 inference on consumer and datacenter GPUs. The paper demonstrates that accurate W4A4 models can be created (52.65% zero-shot accuracy on LLaMA-7B, Table 2) but cannot deploy them because hardware support is absent (Section 4.5). The most direct follow-up is a systems contribution: implement efficient W4A4 matrix multiplication kernels for CUDA (or Triton, or Metal for Apple Silicon) that can actually deliver the theoretical 4× compute reduction over FP16. A strong study would port OmniQuant's W4A4 LLaMA models to a custom kernel, measure end-to-end generation throughput against FP16 and W4A16 baselines, and characterize the gap between achieved and theoretical speedup using roofline analysis. The paper's finding that W3A16 kernels are currently slower than W4A16 (Table 3, 83.4 vs. 134.2 tok/s on LLaMA-7B) provides a baseline: the goal is to demonstrate that W4A4 can exceed W4A16 throughput, not just match it. This work is enabled by OmniQuant because prior W4A4 methods produced such low-accuracy models (SmoothQuant: 38.41%, Table 2) that kernel development was not worth the effort — now there is a target worth optimizing for.

Learned mixed-precision allocation combined with OmniQuant's differentiable parameterization. The paper chooses uniform bit-width allocation for hardware simplicity, but Table A15 shows that non-uniform methods (SqueezeLLM) achieve slightly better perplexity at matched average bit-widths. The natural extension is to make the bit-width assignment itself learnable within the OmniQuant framework: for each weight channel or group, learn a continuous "precision importance" score (parameterized via sigmoid or softmax) that determines whether it should be quantized to 2, 3, or 4 bits, with the average bit-width constrained by a regularizer in the block-wise objective (Equation 1). This would combine OmniQuant's stable differentiable optimization with the flexibility of mixed precision, potentially yielding near-lossless compression at aggressive average bit-widths. A crucial experiment would be measuring the actual inference speed of the resulting mixed-precision model on real hardware — the field needs to know whether the perplexity gains of mixed precision translate to throughput or are erased by irregular memory access patterns.

Cross-model transfer of learned quantization parameters to amortize calibration cost. The paper trains OmniQuant separately for each model, with LLaMA-7B W4A4 taking 1.6 hours (Table A12). An open question is whether the learned clipping strengths (γ, β) and equivalent transformation factors (s, δ, sₐ) transfer across model sizes within the same family — e.g., can the parameters learned on LLaMA-7B be applied (with appropriate interpolation or as initialization) to LLaMA-13B, reducing calibration time? If transfer works, it would suggest OmniQuant learns generic architectural properties (e.g., "the second attention head in the 15th layer always needs aggressive activation scaling") rather than weight-specific ones. A strong experiment would calibrate on LLaMA-7B, apply the learned transformations zero-shot to LLaMA-13B, measure the perplexity gap versus from-scratch calibration on 13B, and analyze which layers show the largest transfer gap. The paper's finding that calibration data distribution has minimal impact (Table A10) and that convergence saturates quickly (Table A11) hints that the optimization landscape may be smooth enough for transfer to work, but this is untested.

Extending equivalent transformation to additional operations: Softmax, LayerNorm, and residual connections. The paper applies LET to linear layers and attention score computation, but explicitly excludes the second FFN linear layer (due to post-nonlinearity gradient instability) and the Softmax output (due to long-tail distribution, kept at FP16, Table A7). Both are candidates for learnable equivalent transformations. For Softmax, a log2-based quantization scheme (as used in RepQ-ViT, cited in the paper) could be made learnable: instead of hand-crafting the log2 base, learn a per-head or per-layer compression factor that minimizes the block-wise reconstruction error. For LayerNorm, the affine parameters could be treated as learnable equivalent transformation targets — since LayerNorm already applies channel-wise scaling and shifting, making these parameters quantization-aware (jointly optimized with downstream LET) could further homogenize activation distributions. For residual connections, which sum the attention output with the input, a learnable scaling factor on one branch (absorbable into the preceding linear projection) could balance the two operands' magnitudes before summation to reduce quantization error in the residual stream. A systematic study applying learnable equivalent transformations to all operations in a transformer block would quantify how much of the remaining FP16-to-quantized gap is addressable through better transformation parameterizations versus being an irreducible consequence of reduced precision.

OmniQuant-calibrated models as initialization for parameter-efficient fine-tuning (QLoRA-style). The paper notes complementarity with QLoRA and INT2.1 (Section 2.2) but does not explore it. A natural combination: quantize a base model with OmniQuant (e.g., LLaMA-7B W4A16g128 achieving 5.77 WikiText2 PPL, Table 1), then fine-tune on a downstream task using LoRA adapters in FP16. The question is whether OmniQuant's learned clipping and equivalent transformations preserve the model's fine-tunability better than GPTQ or AWQ quantization. Specifically, does OmniQuant's lower quantization error translate to higher downstream task accuracy after fine-tuning, or do the learned transformations distort the model's representations in ways that impede adaptation? A strong study would compare OmniQuant vs. GPTQ vs. AWQ as the base quantizer for QLoRA across several commonsense reasoning and instruction-following tasks, measuring both final accuracy and training convergence speed. The paper's instruction-tuned model results (Figure 4: OmniQuant's W3A16g128 LLaMA-2-chat outperforms AWQ's) provide suggestive evidence that OmniQuant preserves more of the original model's capability, but a controlled fine-tuning study would isolate the effect.

Stress-testing OmniQuant at the boundary: quantization below 2 bits and on models larger than 180B. The paper pushes to W2A16 and shows coherent models (LLaMA-13B W2A16g128: 7.93 PPL, Table 1), but does not explore W1A16 or ternary quantization. With LWC's relative clipping and LET's activation-to-weight migration, can uniform 1.5-bit or ternary ({-1, 0, 1}) quantization be made viable? The failure modes at W2A16 (increased epochs needed — 40 vs. 20, Table A9 — and continued improvement through 80 epochs) suggest the optimization becomes harder as bit-width decreases, but the trend is smooth rather than catastrophic. Extending to Falcon-180B-style models (Table A18 shows W3A16g512 works at 180B) would test whether the learned parameters scale to the largest available models, where the economic incentive for aggressive quantization is strongest (335GB → 65GB memory reduction). A negative result — e.g., W2A16 diverging on 180B models despite working on 70B — would help characterize the scaling limits of differentiable PTQ and identify whether model size, architecture, or training data distribution is the binding constraint.

Practical Applications and Downstream Use Cases

Single-GPU deployment of 30B–70B models for local inference. Table 3 provides the concrete numbers: LLaMA-30B FP16 requires 60.6GB of weight memory and 66.1GB of running memory — exceeding any single consumer GPU. With OmniQuant's W4A16g128, the same model requires 16.7GB of weight memory and 21.7GB running memory, fitting comfortably on an RTX 4090 (24GB) or A100-40G, while delivering 43.6 tokens/second (1.8× faster than FP16 on an A100-80G would be if it fit). LLaMA-65B at W4A16g128 requires 33.0GB weight memory — within reach of an A100-80G or an Apple M2 Ultra with 64GB unified memory — where FP16 would run out of memory entirely. For practitioners building local chatbots, code assistants, or document analysis tools, this means a single high-end consumer GPU can run models that previously required multi-GPU server setups. The memory savings are proven on hardware; the speed improvements for non-4-bit configurations await better kernel support.

Cost-efficient batch inference at scale for LLM API providers. For organizations serving LLM inference at scale (e.g., offering API access to LLaMA-family models), memory bandwidth is often the throughput bottleneck. OmniQuant's W4A16g128 reduces weight memory by approximately 3.3× (e.g., LLaMA-70B: 134GB → 33GB for weights, inferred from Table 3 scaling), meaning 3.3× more model instances can fit in the same GPU memory, and each instance runs 1.7–1.9× faster (Table 3: LLaMA-7B: 134.2 vs. 69.2 tok/s; LLaMA-13B: 91.3 vs. 52.5 tok/s). For a batch inference workload where multiple requests are processed in parallel, the combined effect is a roughly 5–6× increase in throughput per GPU. The paper's demonstration that W4A16g128 OmniQuant models achieve near-FP16 perplexity (LLaMA-13B: 5.17 vs. 5.09, Table 1) and zero-shot accuracy (implied by the strong W4A4 results in Table 2 — W4A16 would be even better) means this throughput gain comes with minimal quality degradation. The 1–2 hour per-model calibration cost (Table A12) is negligible when amortized over millions of inference requests.

On-device LLM inference for privacy-sensitive applications. The paper's Falcon-180B result (Table A18) shows a 335GB model compressed to 65GB (W3A16g512) — a 5.2× reduction — while maintaining 79.40% zero-shot accuracy versus 80.58% FP16. Extrapolating to smaller models: LLaMA-7B at W4A16g128 uses 3.8GB of weight memory (Table 3), which fits on a flagship smartphone (12–16GB RAM) alongside the operating system and other apps. Combined with recent advances in on-device inference engines (llama.cpp, MLC-LLM, MediaPipe), this enables LLM-powered features — email summarization, document Q&A, text completion — to run entirely on-device without sending data to cloud servers. The privacy benefit is that user text never leaves the device; the latency benefit is eliminating network round-trips. OmniQuant's contribution is making the on-device model accurate enough to be useful — prior W3A16 methods (GPTQ: 5,500 PPL on 13B, Table 1) would produce incoherent outputs that undermine the user experience.

Quantization as a preprocessing step for domain-specific fine-tuning pipelines. Organizations that fine-tune LLMs on proprietary data (legal documents, medical records, financial reports) often face a compute bottleneck: fine-tuning a 70B model requires multiple GPUs even with parameter-efficient methods. A workflow enabled by OmniQuant: quantize the base model once (e.g., LLaMA-70B to W4A16g128, 1–2 hours on a single A100), then distribute the quantized model to domain teams who fine-tune LoRA adapters on single GPUs. The quantized base model serves as a frozen backbone; only the lightweight adapters are trained. After fine-tuning, the adapters plus quantized base are deployed. The key enabler is that OmniQuant's quantization preserves the model's representational quality (low perplexity degradation, Table 1), so the adapters learn meaningful domain-specific transformations rather than fighting quantization artifacts. The paper does not demonstrate this pipeline directly, but the combination of OmniQuant's strong weight-only results and the established QLoRA paradigm makes it a natural next step — and the paper explicitly notes complementarity with QLoRA in Section 2.2.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration2023· inference-optimization← LLM-judged similar
- READ MORE2023· inference-optimization← in-domain predecessor
- "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization2024· inference-optimization← similar topic
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding2015· inference-optimization← similar topic
Accelerating LLM Inference with Staged Speculative Decoding2023· inference-optimization← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models2023· inference-optimization→ in-domain predecessor
- YARN: EFFICIENT CONTEXT WINDOW EXTENSION OF LARGE LANGUAGE MODELS2023· training-methods→ LLM-judged similar
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models2023· inference-optimization→ in-domain predecessor