Inference Optimization2025~99 minarXiv:2509.23202

Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization

score 9Vage Egiazarian, Roberto L. Castro, Denis Kuznedelev +8NVIDIA↑ 29★ 98Sep 27, 2025cs.LGhttps://github.com/IST-DASLab/FP-Quant

🎯 Pitch

Rotations—a standard trick for improving quantization—actually worsen NVFP4 accuracy due to its tiny block size, but the new MR-GPTQ algorithm can rescue MXFP4 to near-NVFP4 quality while delivering up to 3.6× GPU speedups. This paper reveals that neither hardware FP4 format is a drop-in replacement for INT4, but careful format-aware optimization unlocks a new Pareto frontier.

1. Executive Summary

This paper presents the first comprehensive study of microscaling 4-bit floating-point formats—MXFP4 and NVFP4—for post-training quantization of large language models, revealing that state-of-the-art quantization methods struggle with these formats due to NVFP4's small group size neutralizing outlier mitigation and MXFP4's power-of-two scale quantization inducing severe accuracy degradation. To bridge this gap, the authors introduce Micro-Rotated-GPTQ (MR-GPTQ) , a variant of the GPTQ algorithm that employs block-wise Hadamard transforms, format-specific scale optimization, and static activation reordering—achieving up to 3.6× layer-wise and 2.2× end-to-end speedups over FP16 on NVIDIA B200 GPUs via the accompanying QuTLASS kernel library. The method recovers MXFP4 accuracy to within 1–2% of NVFP4 on Llama-3 and Qwen-3 models, establishing that while FP4 is not an automatic upgrade over INT4, format-specialized techniques can unlock a new accuracy–performance frontier only when tailored to each format's quantization error properties.

2. Context and Motivation

The Core Problem: Hardware Promise Without Algorithmic Proof

The fundamental tension this paper addresses is deceptively simple: hardware vendors have shipped FP4 microscaling formats with bold accuracy claims, but no one has systematically verified whether these claims hold up under rigorous evaluation with state-of-the-art quantization methods. This gap is not merely academic — it has direct consequences for anyone deploying LLMs at scale. NVIDIA's Blackwell architecture and AMD's CDNA4 GPUs now accelerate MXFP4 and NVFP4 matrix multiplications in silicon, and the industry narrative (e.g., NVIDIA's technical marketing [40; 43]) positions FP4 as a straightforward accuracy improvement over INT4. If this narrative is wrong, practitioners who adopt these formats expecting lossless compression will encounter silent accuracy degradation in production. If it's right, the community needs to understand why and when so that existing quantization tooling (GPTQ, SmoothQuant, QuaRot, SpinQuant) can be adapted accordingly.

The paper identifies the specific knowledge gap in the introduction:

"Despite these developments, little is known about the accuracy of these formats on real models or their practical performance."

This is not an overstatement. Prior to this work, there was no comprehensive study of how NVFP4 and MXFP4 interact with weight-and-activation quantization at 4-bit precision across modern LLM families (Llama-3, Qwen-3) using standard zero-shot benchmarks. The formats exist, the hardware exists, but the evaluation does not.

Why This Matters: More Than Academic Curiosity

The practical stakes are high for several reasons the paper implicitly raises throughout Sections 1 and 2:

1. LLM inference is memory-bandwidth-bound, not compute-bound — until it isn't. In small-batch autoregressive decoding, throughput is limited by how fast weights can be streamed from GPU memory. Reducing weight precision from FP16 to FP4 cuts memory traffic by 4×, which directly translates to latency and throughput improvements if the model's accuracy is preserved. The paper's QuTLASS kernels demonstrate that these theoretical bandwidth savings do materialize: 3.6× layer-wise speedups on B200, 6× on RTX5090 (Figures 5, 11). But speed without accuracy is worthless. A practitioner choosing between a fast FP4-quantized model that produces wrong answers and a slower FP16 model that produces correct ones will always choose the latter. The paper's accuracy analysis (Table 1, Figures 7, 13) is what makes the speedup numbers actionable rather than speculative.

2. The microscaling design space is not neutral — format choices create winners and losers among quantization algorithms. MXFP4 and NVFP4 differ in two seemingly minor ways: group size (32 vs. 16) and scale representation (E8M0 powers-of-two vs. E4M3 full FP8). The paper's error analysis in Section 3 proves that these choices have qualitatively different effects on quantization error that propagate through to downstream accuracy. Specifically: NVFP4's E4M3 scales provide enough precision that top elements in each group are effectively "promoted" to higher precision, naturally mitigating outliers without any algorithmic intervention — but this also means that Hadamard rotations hurt NVFP4 accuracy under round-to-nearest quantization (Table 1, RTN vs. RTN+HT for NVFP4: 74.73% → 74.05% average). Conversely, MXFP4's E8M0 scales are so coarse that Hadamard rotations help substantially (RTN vs. RTN+HT for MXFP4: 69.32% → 70.45%), but still cannot close the gap to NVFP4. These are not obvious results — the community has largely treated rotations as a universal good for quantization since QuIP/QuaRot — and they have direct engineering implications: if you're deploying NVFP4, skip the rotations; if you're deploying MXFP4, you need them.

3. The gap between MXFP4 and NVFP4 is larger than the hardware cost difference would predict. The average bits-per-element difference is small: 4.25 for MXFP4 vs. 4.5 for NVFP4. Yet the accuracy gap in Table 1 is substantial: for Llama-3.1-8B-Instruct W4A4 with GPTQ, NVFP4 averages 75.72% vs. MXFP4's 70.62% — a ~5 percentage point gap, or roughly 6% relative accuracy loss. This matters because hardware support for the two formats is not symmetric: MXFP4 is an open standard (OCP) with broader ecosystem support (AMD, future Intel GPUs), while NVFP4 is NVIDIA-proprietary. If the industry standardizes on MXFP4 for portability, it incurs a hidden accuracy tax that prior literature had not documented.

Where Existing Approaches Fall Short

The paper identifies specific, non-obvious failure modes across multiple families of prior work:

Pretrained quantization methods were never evaluated on FP4 microscaling. The GPTQ algorithm [21] was developed and validated on INT4 and INT8 formats with uniform grids and group-wise FP16 scales. Its effectiveness on FP4's non-uniform grid and quantized scales was entirely unknown. Similarly, SmoothQuant [54] was designed for INT8 activation quantization, operating on the assumption that activation outliers can be redistributed between weights and activations via per-channel scaling factors. The paper shows (Table 1) that SmoothQuant works moderately well across NVFP4, MXFP4, and INT4, but is not uniformly the best method — its success depends on the format and the presence or absence of rotations. AWQ [36] relies on per-channel scaling based on activation magnitudes, which implicitly assumes reasonably large group sizes where outlier channels can be identified and protected. The small group sizes in microscaling formats (16 or 32) already perform implicit outlier mitigation via absmax scaling, making AWQ's saliency-based protection redundant or counterproductive.

Rotation-based methods (QuaRot, SpinQuant) make assumptions that break for NVFP4. The core insight of QuaRot [4] and SpinQuant [38] is that Hadamard rotations spread outlier energy across all dimensions, normalizing weight and activation distributions and reducing quantization error. This is well-motivated for INT quantization with large group sizes or per-channel scaling. But the paper's Lemma 1 and Section 3 analysis reveal a previously unnoticed interaction: for microscaling formats with absmax scaling, applying a rotation spreads the top-element error across the entire group. Under NVFP4, the top element naturally enjoys high precision because it determines the shared scale (effectively "promoting" it to the scale's precision). Rotations destroy this property, trading sure-thing precision on the most important element for better average behavior on the rest — a trade that Table 1 shows is net-negative for NVFP4 with RTN. QuaRot and SpinQuant were not designed with this tradeoff in mind, and their default application to FP4 formats yields accuracy below simple RTN (QuaRot + NVFP4: 74.10%, vs. RTN: 74.73%).

Integer quantization research created blind spots. The community's extensive work on INT quantization created a set of assumptions that do not transfer to FP4: that quantization grids are uniform (FP4 is not), that larger group sizes are always better for outlier mitigation (microscaling's small groups already handle this), that rotations universally help (they don't for NVFP4), and that scale quantization is a secondary concern (MXFP4's E8M0 scales are the primary accuracy bottleneck). The paper's Section 3 formalizes these differences mathematically, while Section 5 quantifies their practical impact. Prior work had not drawn these distinctions because FP4 microscaling formats simply did not exist when those methods were developed.

Industry claims lacked independent verification. The paper notes in the introduction that NVIDIA's technical materials claim FP4 superiority over INT4 [40; 43]. The related work section (Section 2) observes that "rigorous evaluation remains absent so far, likely due to the recent introduction of this format." The paper's comparison of INT4 (group size 32, FP16 scales — matching NVFP4's average bit-width) against NVFP4 and MXFP4 in Table 1 fills this gap: INT4 with RTN+HT achieves 74.75%, comparable to NVFP4 RTN at 74.73%, while GPTQ-based INT4 (73.21%) lags behind GPTQ NVFP4 (75.72%). The formats are not uniformly superior or inferior — their performance depends on the quantization method, a nuance absent from simplified vendor comparisons.

How This Paper Positions Itself

The paper frames itself not as proposing a single new method, but as providing three interconnected contributions that together constitute a complete picture:

1. An analytical framework for understanding FP4 microscaling error. Section 3 develops a model of quantization error under Laplace (native weights/activations) and Normal (post-rotation) distributions, deriving the asymptotic behavior of per-element MSE and top-element MSE as functions of group size. The key insight — that the E8M0 scale format's coarseness relative to E2M1 base elements makes MXFP4 intrinsically higher-error, and that NVFP4's E4M3 scale format effectively "promotes" top elements — is supported analytically (Lemmas 1–4) and validated numerically on both synthetic and real-model tensors (Figure 3). This is not an incremental empirical observation; it's a principled explanation of why the formats behave differently that enables engineering decisions (when to rotate, when not to) to be derived rather than discovered by trial-and-error.

2. A format-specialized quantization algorithm (MR-GPTQ) that bridges the gap. Rather than treating all FP4 formats identically, MR-GPTQ adapts GPTQ along three axes that the error analysis shows matter: (a) block-wise Hadamard rotations sized to match the quantization group size (16 for NVFP4, 32 for MXFP4), (b) format-specific scale optimization (MSE-optimized grid search for NVFP4, static scale fitting for MXFP4), and (c) static activation reordering that provides the accuracy benefits of GPTQ's dynamic act-order without its runtime overhead. The paper explicitly acknowledges that some components are not universally beneficial — rotations hurt NVFP4 under RTN but help under GPTQ — and designs MR-GPTQ to exploit these conditional effects.

3. Production-grade GPU kernels (QuTLASS) that eliminate the rotation overhead. A common objection to rotation-based quantization is that the online Hadamard transform of activations adds latency that negates the speedup from reduced precision. The paper addresses this head-on with custom CUDA kernels that fuse the rotation, quantization, and scale computation into a single memory-bound kernel — meaning the rotation is essentially free because the limiting factor is memory bandwidth, not compute (Section 4.2, Tables 13–14). The fact that MXFP4 with Hadamard rotations can outperform NVFP4 throughput (Figure 5) despite the additional transform is a non-obvious result enabled by MXFP4's larger group size and simpler scale arithmetic (power-of-two multiplications).

The paper explicitly positions itself relative to the training-inference tradeoff that dominates quantization research: it does not claim that FP4 is a finished product ready for universal deployment. The Results discussion in Section 5.1 (point 1) is candid: "No Lossless Format: Across all formats, the accuracy drop is noticeable." Instead, the paper positions MR-GPTQ as demonstrating what is possible when formats are taken seriously as first-class design variables rather than interchangeable backends — and implicitly argues that hardware vendors and the ML community need to collaborate on format design rather than treating it as a fixed constraint.

3. Technical Approach

This is primarily an empirical analysis paper whose core idea is that the effectiveness of any quantization method for FP4 microscaling formats depends critically on how that method interacts with the format's specific error properties—MXFP4 suffers from coarse scale quantization (E8M0 powers-of-two), while NVFP4's small group size and precise scales (E4M3) naturally mitigate outliers but nullify the benefits of Hadamard rotations—and that a format-specialized variant of GPTQ can bridge the accuracy gap between them.

3.1 Reader Orientation

The paper builds a system for quantizing LLM weights and activations to 4-bit microscaling floating-point formats (MXFP4 and NVFP4) such that the quantized model runs 2–4× faster on modern GPUs while preserving as much of the original FP16 accuracy as possible. The core idea is that no single quantization recipe works across both formats; instead, the paper develops Micro-Rotated-GPTQ (MR-GPTQ) , which adapts the classic GPTQ algorithm along three axes—block-wise Hadamard rotations, format-specific scale optimization, and static activation reordering—each motivated by the format's distinct error characteristics, and supports it with custom GPU kernels (QuTLASS) that make the online rotation cost negligible.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Error Analysis Framework (Section 3) — a mathematical model that predicts how NVFP4 and MXFP4 quantization error behaves under different weight/activation distributions (Laplace for native tensors, Normal for rotated tensors) and identifies which format properties cause accuracy degradation. This is the analytical foundation that motivates all design choices.
Format-Specialized Quantization Grid and Scale Optimizer — for NVFP4, this solves an alternating optimization over per-block scales and a global tensor scale to minimize MSE; for MXFP4, this fits the excessively wide E8M0 scale range to the actual data range via exponent rescaling (Section 4.1, "Ingredient 1").
Block-Wise Hadamard Rotation — applies k × k Hadamard transforms (H_k) to weights and activations, where k is chosen to match the quantization group size (16 for NVFP4, 32 for MXFP4). This normalizes heavy-tailed distributions toward Gaussian, reducing per-element MSE for MXFP4 and—crucially—enabling GPTQ's error compensation to work better across both formats (Section 4.1, "Ingredient 3").
MR-GPTQ Quantization Algorithm — extends GPTQ's column-wise weight quantization with: (a) rotations applied before quantization grid computation, (b) static activation reordering that shuffles columns by Hessian diagonal magnitude after scale computation but before GPTQ's iterative quantization, avoiding the 10–20% runtime overhead of dynamic reordering (Section 4.1, "Ingredient 2"), and (c) the rotated weights are stored in the quantized format, with rotations pre-fused into them.
QuTLASS GPU Kernels — a suite of CUDA kernels for NVIDIA Blackwell GPUs that: (a) fuse the online activation rotation and quantization into a single memory-bound kernel (the rotation is essentially free because it's limited by memory bandwidth, not compute), (b) handle the hardware-mandated scale rearrangement for tcgen05.mma instructions, and (c) support plug-and-play matmul backends (CUTLASS, FlashInfer). The key insight is that for block sizes k < 256, the rotation matrix fits in registers/cache, so any orthogonal transform (not just Hadamard) costs the same (Section 4.2).

Information flows as follows: a pretrained FP16 model enters → the error analysis informs which format-specific optimizations to apply → block-wise Hadamard rotations are applied to weights (pre-fused) → per-group and per-tensor scales are optimized via MSE minimization (NVFP4) or exponent rescaling (MXFP4) → weight columns are statically reordered by Hessian importance → GPTQ quantizes weights column-by-column, compensating error via second-order updates → at inference time, activations are rotated online by QuTLASS's fused kernel → the FP4 matrix multiplication executes on tensor cores → the result is dequantized to FP16/BF16 for accumulation.

3.3 Roadmap for the Deep Dive

First, the formal quantization error model (Section 3): how microscaling formats are mathematically defined, what distributional assumptions are made for native vs. rotated tensors, and the asymptotic MSE bounds that predict when rotations help or hurt. This is essential because every subsequent design choice is justified by these predictions.
Second, the scale optimization and grid design (Section 4.1, "Ingredient 1"): how MR-GPTQ computes quantization scales differently for NVFP4 (alternating optimization) vs. MXFP4 (static exponent fitting), and why the naive absmax approach is suboptimal.
Third, the block-wise Hadamard rotation mechanism (Section 4.1, "Ingredient 3"): what it does to weight/activation distributions, why the block size matters (matching group size), and how it is supported at runtime without overhead via QuTLASS.
Fourth, the static activation reordering (Section 4.1, "Ingredient 2"): how it differs from GPTQ's dynamic act-order, why it provides similar accuracy benefits without runtime cost, and its interaction with the microscaling group structure.
Fifth, the QuTLASS kernel design (Section 4.2): how the online rotation is fused with quantization, why MXFP4 can outperform NVFP4 throughput despite additional transforms, and the hardware-specific optimizations for SM100 (B200) and SM120 (RTX5090).

3.4 Detailed, Sentence-Based Technical Breakdown

Formal Modeling of Microscaling Quantization Error

The paper's analytical framework in Section 3 models microscaling block floating-point (MFP) quantization as a two-stage process: first, elements within a block of size G share a common scale s (computed via absmax: s = max_i |X_i|); second, each normalized element U_i = X_i / s ∈ [-1, 1] is rounded to the nearest value on a finite grid Q ⊂ [-1, 1]. The grid Q is determined by the FP4 base format (E2M1: 1 sign bit, 2 exponent bits, 1 mantissa bit), which provides 7 distinct positive values {0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0} plus zero and their negatives.

Distribution Assumptions (Definition 1). The paper models native (unrotated) weights and activations as following a Laplace distribution W ~ Laplace(0, b) with density:

$f_W(w) = \frac{1}{2b} e^{-|w|/b}$

where b = 1/√2 gives unit variance (Var(W) = 2b² = 1). The magnitude Z = |W| is then exponentially distributed with rate λ = 1/b = √2. This is empirically motivated: Figure 2 shows that Llama-3.1-8B-Instruct's native weights have kurtosis 1.47 and native activations have kurtosis 8.75—both substantially heavier-tailed than Gaussian (kurtosis 0), making Laplace a better fit.

After applying a Hadamard rotation, weights and activations follow a Normal distribution V ~ N(0, 1) with kurtosis near zero (0.05 and 0.02 respectively, Figure 2). The magnitude Z = |V| is half-normal. This normalization property has been proven in prior work (QuIP, QuIP#) and is the theoretical motivation for rotation-based quantization—the rotated distribution has no outliers, making it easier to quantize with uniform grids.

Quantization Metrics (Definition 3). The paper tracks two key error quantities:

$\text{MSE}(G) = \mathbb{E}[(X_1 - \hat{X}_1)^2]$

$\text{MSE}_{\text{top}}(G) = \mathbb{E}[(X_{I^\star} - \hat{X}_{I^\star})^2]$

where G is the group size, X_1 is an arbitrary element (by symmetry), I* = arg max_i |X_i| is the index of the maximum-magnitude ("top") element, and X̂_i is the de-normalized quantized value X̂_i = s · RTN_Q(X_i/s). MSE(G) captures the average per-element error; MSE_top(G) captures the error on the element that determined the shared scale.

The Dead-Zone Effect (Remark 1). The first positive quantization level in the FP4 grid, denoted q_min, induces a dead-zone half-width δ = q_min / 2 on the normalized domain [0, 1]. Any normalized element with |U_i| < δ gets rounded to zero. For FP4's E2M1 grid, q_min = 0.5, so δ = 0.25. This dead-zone is the primary source of quantization error for small-magnitude elements.

Asymptotic Error Behavior. The paper proves a critical asymptotic result: as group size G → ∞, MSE(G) → Var(X_1) = 1 for both Laplace and Normal distributions. This happens because as G grows, the block maximum max_i |X_i| diverges, so |U_1| = |X_1|/max_i|X_i| → 0 in probability—the normalized element falls into the dead-zone and gets quantized to zero. The "preserved mass" (what escapes underflow) is:

$R(G) = 1 - \text{MSE}(G) = \mathbb{E}[X_1^2 \cdot \mathbf{1}_{\{|U_1| \geq \delta\}}]$

Why Laplace Beats Normal at Small G but Loses at Large G (Lemma 2). The paper derives the asymptotic rates of R(G) for both distributions:

Laplace: R_L(G) = Θ((log G)² · G^(-δ)) with rate δ = 0.25
Normal: R_N(G) = Θ(√(log G) · G^(-δ²)) with rate δ² = 0.0625

Since 0 < δ² < δ < 1, we get a crossover phenomenon: at small group sizes, the Laplace MSE is lower (better preservation) because the heavier tails concentrate mass in fewer, larger elements that survive the dead-zone. At sufficiently large G, the Normal distribution's faster decay rate (G^(-δ²) vs. G^(-δ)) reverses this—the Normal MSE becomes lower.

What this means for quantization: For NVFP4's small group size G = 16, native Laplace-distributed weights should quantize better than rotated Normal weights. For MXFP4's G = 32, the Normal distribution might start winning for RTN, but the crossover depends on the specific δ. This predicts that rotations should hurt NVFP4 accuracy at small group sizes but help at larger ones, and should help MXFP4 more consistently—exactly what Table 1 empirically confirms (NVFP4 RTN: 74.73% → RTN+HT: 74.05%, a small decline; MXFP4 RTN: 69.32% → RTN+HT: 70.45%, a clear improvement).

Top-Element Error and the Rotation Trade-off (Lemma 1). The paper proves a fundamental result about what happens to outliers under rotations. When a vector x ∈ R^G with i.i.d. N(0, 1) coordinates is Hadamard-transformed to y = (1/√G) H x, quantized, and inverse-transformed back to x̂ = (1/√G) H^T ŷ, the expected squared error on the original top coordinate equals the per-element MSE:

$\text{MSE}_{\text{top}}(G) = \text{MSE}(G)$

What this means mechanically: Without rotation, MSE_top(G) = 0 for absmax scaling—the top element determines the scale, so it is always perfectly preserved (its normalized value is ±1, which is exactly on the FP4 grid). With rotation, the top element's error becomes exactly the average error across all elements. This is a direct trade-off: rotations eliminate the "free lunch" of perfect outlier preservation in exchange for better average performance. For NVFP4, where the E4M3 scale effectively promotes top elements to ~8-bit precision, destroying this property is harmful. For MXFP4, where the E8M0 scale is too coarse to provide meaningful promotion anyway, the trade-off favors rotation.

Shared Scale Quantization Effects (Section 3.2). The paper extends the analysis to account for quantized scales—the fact that s itself is stored in a reduced-precision format (E8M0 for MXFP4, E4M3 for NVFP4). The key observation (Figure 4) is that the dynamic ranges of real weight and activation scales comfortably fit within E4M3's range [2^(-9), 448], and certainly within E8M0's enormous range [2^(-127), 2^128]. This means:

For NVFP4 (S = E4M3): The scale's mantissa precision (3 bits) is finer than the base element's mantissa (1 bit for E2M1), so the effective precision on top elements is dominated by the scale, not the element. Top elements are effectively "promoted" to E4M3 precision. This is proven in Lemma 4: for a smooth distribution over a wide dynamic range, the relative top-element MSE is translation-invariant to power-of-two shifts and depends only on the finer of the two precisions (here, E4M3).
For MXFP4 (S = E8M0): The scale has zero mantissa bits—it can only represent exact powers of two. This is coarser than the base E2M1 element precision. The top element inherits its effective precision from the base format (E2M1), not the scale, because the scale cannot represent intermediate values between powers of two. The relative top-element MSE is constant regardless of G, and is determined by E2M1's quantization grid.

This explains Figure 3's most striking empirical finding: for NVFP4, MSErel_top is flat across all group sizes (the E4M3 promotion works regardless of G); for MXFP4, MSErel_top is also flat but at a higher level (stuck at E2M1 precision). Only after applying the Hadamard transform—which spreads the top-element error across the group via Lemma 1—does MSErel_top decrease with G, because the per-element MSE benefits from averaging over more elements.

Relative Error Metrics (Definition 4). For real tensors with varying group variances, the paper defines normalized versions:

$\text{MSE}_{\text{rel}}(G) = \mathbb{E}\left[\frac{\sum_{i=1}^{G} (X_i - \hat{X}_i)^2}{\sum_{i=1}^{G} X_i^2}\right]$

$\text{MSE}_{\text{rel}}^{\text{top}}(G) = \mathbb{E}\left[(X_{I^\star} - \hat{X}_{I^\star})^2 / X_{I^\star}^2\right]$

These are the metrics plotted in Figure 3, Figure 10 (for INT4 microscaling), and Figure 8–9 (appendix). MSErel is the key predictor of downstream accuracy: Malinovskii et al. (2025) showed a linear relationship between MSErel and end-to-end accuracy decline, and recent lattice-based PTQ methods (QuIP#, QTIP) explicitly optimize for it.

Validation on Real Model Tensors (Figure 3). The paper validates its analytical predictions by measuring MSErel and MSErel_top on actual Llama-3.1-8B-Instruct weight and activation matrices. The results in Figure 3 (top row) show:

For NVFP4 (G=16, E4M3 scales): MSErel is ~0.010–0.012 across group sizes, and MSErel_top is ~0.001–0.003 (very low). Adding Hadamard transform (HT) increases MSErel to ~0.015 at G=16—confirming the prediction that rotations hurt at small group sizes. At larger G (64–128), HT reduces MSErel—confirming the crossover prediction.
For MXFP4 (G=32, E8M0 scales): MSErel is substantially higher (~0.015–0.020) and MSErel_top is ~0.010–0.015 (much worse than NVFP4). Adding HT reduces MSErel across all group sizes, with the gap widening at larger G—confirming that rotations are unambiguously beneficial for MXFP4.

The Laplace distribution samples (left column) and the model tensors (right columns) show qualitatively identical patterns, validating that the Laplace/Normal modeling captures the essential dynamics.

Integer Microscaling Formats (Appendix D, Figure 10). The paper extends the analysis to hypothetical INT4-based microscaling formats (NVINT4: G=16, E4M3 scales; MXINT4: G=32, E8M0 scales) with a symmetric uniform grid of 15 values [-7, ..., +7]. The analysis predicts that NVINT4 should benefit from HT (unlike NVFP4, because INT4 has no "element precision" advantage to lose), while MXINT4 should behave similarly to MXFP4. Figure 10 validates both predictions, and Tables 1 and 3–5 confirm that NVINT4 with MR-GPTQ achieves the highest accuracy of all formats (97.12% recovery on Llama-3.1-8B, vs. 96.08% for NVFP4 and 93.31% for MXFP4).

MXFP Scale Fitting (Appendix H). The paper identifies that MXFP4's E8M0 format has a dynamic range [2^(-127), 2^128] that is vastly wider than the actual range of weight and activation scales (roughly [2^(-15), 2^5], as shown in Figure 4). This wastes most of the 256 available quantization levels on values that never occur. The proposed fix: rescale the exponent to fit the data range.

The original MXFP4 scale quantization (with 4/3 bias correction from Tseng et al., 2025) is:

$s_{\text{E8M0}} = (4/3) \cdot 2^{\text{clamp}(\text{round}(\log_2 s), -128, 127)}$

The modified version estimates the minimum and maximum log-scales log_2(s_min) and log_2(s_max) from the tensor, then remaps:

$s_{\text{E8M0}} = 2^{(\log_2 s_{\max} - \log_2 s_{\min}) \cdot \text{clamp}(\text{round}(255 \cdot \frac{\log_2 s - \log_2 s_{\min}}{\log_2 s_{\max} - \log_2 s_{\min}}), 0, 255) + \log_2 s_{\min}}$

What this computes: Instead of using the full E8M0 range, it computes a linear mapping of the log-scale to the 256 quantization bins, then exponentiates. The effective representation becomes s = 2^(αq + β) with α ∈ (0, 1), which is a fractional exponent—intermediate between powers of two.

Why this matters: Standard E8M0 can only represent exact powers of two, introducing quantization error proportional to the gap between adjacent levels (a factor of 2). The rescaled version can represent values at fractional powers of two, reducing the quantization step size by making the exponent step α smaller than 1. The cost is that the weight and activation scales in a layer must share the same exponent α for efficient hardware multiplication (s_A · s_W = 2^(α_A q_A + β_A + α_W q_W + β_W) requires α_A = α_W), which the paper's implementation ensures. Table 11 shows this modification (denoted MXFP4†) boosts RTN accuracy by 6.5 points on Llama-3-8B and 2.6 points on Qwen-3-8B, bringing MXFP4 to within 0.4–2.6 points of NVFP4 without any other algorithmic changes.

MSE-Optimized Grid and Scale Selection (Ingredient 1)

The paper's first algorithmic addition to GPTQ is format-specific scale optimization. The key insight is that microscaling formats have two levels of scaling: a per-tensor scale s_T (global) and per-group scales s_Gi (one per block of G elements). The quantized value is:

$\hat{X}_i = s_T \cdot s_{Gj} \cdot Q\left(\frac{X_i}{s_T \cdot s_{Gj}}\right)$

where Q is the FP4 quantization function and j indexes the group containing element i.

For NVFP4, the paper solves a joint optimization over both levels of scales:

$\min_{s_T, s_{G1}, \ldots, s_{Gk}} \sum_i \|\hat{X}_i - X_i\|_2^2$

What this computes: Given a tensor divided into k groups, find the per-tensor scale and each per-group scale that minimize the total squared quantization error. This is a non-convex problem with k+1 continuous variables.

How it is solved: The paper uses alternating optimization: fix s_T, solve for each s_Gj independently (which reduces to a 1D line search per group since the FP4 grid is small), then fix all s_Gj and solve for s_T (another 1D search), and repeat until convergence. The authors state this yields "consistent improvements" for NVFP4 without rotations.

For MXFP4 with rotations, the paper found that a single static value for the per-tensor scale works stably across all layers—the rotated distributions are sufficiently normalized that the optimal per-tensor scale doesn't vary much. This simplifies the implementation: one scale per layer, computed once during calibration, with no runtime optimization needed.

Why this approach over simple absmax scaling: Absmax scaling sets s_T · s_Gj = max_i |X_i in group j|, which exactly preserves the top element (zero error on it) but may waste precision on other elements. The MSE-optimized scales can reduce the scale below the maximum, allowing some top-element clipping in exchange for better average precision—exactly the trade-off that Lemma 1 shows is beneficial when the top-element precision is already good (NVFP4 with E4M3 scales). For MXFP4, the scale quantization error (E8M0) dominates anyway, so sophisticated scale optimization is less impactful than the exponent rescaling described above.

Block-Wise Hadamard Rotation (Ingredient 3)

The paper's second algorithmic addition applies block-wise Hadamard transforms to both weights and activations. For a linear layer with weight matrix W (size d_out × d_in) and activation input X (size batch × d_in), the computation becomes:

$\text{Output} = Q(W H_k) \cdot Q(X H_k)^T$

where H_k is a k × k Hadamard matrix (or block-diagonal concatenation of such matrices) applied along the input dimension, and Q is the quantization function. Crucially, the rotation is pre-fused into the weights—the stored quantized weights are already Q(W H_k), so no online weight rotation is needed.

How it works at inference time: The activations X arrive at the layer in FP16/BF16. The QuTLASS kernel applies H_k to X online, quantizes the result to FP4, and feeds it to the FP4 matrix multiplication with the pre-rotated weights Q(W H_k). The output is mathematically equivalent to the unrotated computation because (W H_k)(X H_k)^T = W H_k H_k^T X^T = W X^T (Hadamard matrices are orthogonal: H_k H_k^T = k I). The scale normalization 1/√k is absorbed into the quantization scales.

Why block size matters: The paper sweeps block sizes k ∈ {16, 32, 64, 128, 256} and finds that the optimal choice is matching the quantization group size: k = 16 for NVFP4, k = 32 for MXFP4. The intuition (from Lemma 1 and Figure 3) is that the rotation spreads error within each block; if the block is larger than the quantization group, elements that share a scale come from different rotation blocks, breaking the normalization property. If the block is smaller, the distribution within each quantization group is not fully normalized (not enough averaging). The Platinum benchmark results (Figure 13) confirm this: for NVFP4, Had16 outperforms Had128 (96.6% vs. 93.9% recovery); for MXFP4, Had32 is competitive with Had128 (92.1% vs. 92.3%) but the gap to non-rotated variants is much larger (~92% vs. ~85%).

The key property enabling zero overhead: For k < 256, the k × k Hadamard matrix fits entirely in GPU registers or shared memory, and the transformation is memory-bound, not compute-bound. This means the rotation costs essentially the same as simply loading the activations from memory—which has to happen anyway before the matrix multiplication. Tables 13 and 14 in Appendix K show that different transform types (Hadamard, DCT, DST, GSR) and different block sizes (16 to 256) produce nearly identical throughput for the same quantization format, confirming the memory-bound nature. The QuTLASS kernel fuses the rotation, quantization, and scale computation into a single kernel launch with a custom epilogue function, eliminating separate kernel launch overhead and intermediate memory writes.

Static Activation Reordering (Ingredient 2)

The third algorithmic addition modifies GPTQ's activation reordering heuristic. The original GPTQ algorithm optionally reorders weight columns in descending order of the corresponding Hessian diagonal entry H_{ii} = (1/N) ∑_j (X_{ji})^2 before quantization. This "dynamic act-order" groups important columns (those with high activation variance) first, so that GPTQ's error compensation—which accumulates error in later columns—places the residual error on less important weights. However, this column reordering must be reversed at runtime, requiring an online shuffle that costs 10–20% end-to-end inference slowdown.

MR-GPTQ's innovation: static reordering. Instead of reordering before computing quantization scales and grids, MR-GPTQ performs these steps:

Compute per-group scales s_Gj and the quantization grid using the original, arbitrary column order. This ensures the scales are correct for the actual layout.
After scales are fixed, reorder the columns by descending Hessian diagonal magnitude.
Apply GPTQ's iterative quantization on the reordered matrix, using the pre-computed scales.
After quantization, reorder the columns back to their original positions, maintaining the microscaling group structure.

What this achieves: The benefit of act-order (important columns get lower quantization error because they're processed first, when the error compensation budget is largest) is preserved, but the online reshuffling penalty is eliminated—the model's weights are stored in the original column order, and no runtime permutation is needed. The paper states this "provides similar improvements to standard 'dynamic' act-order, without the runtime overheads."

Why it works: The key insight is that the quantization scales and grid are determined by the weight values, not their column order—a weight's optimal quantization bin depends on its magnitude relative to its group's maximum, which is invariant to column permutation within the group. Therefore, the scale computation can happen in the original order, the reordering affects only which weights get priority in GPTQ's error allocation, and this priority ordering is a heuristic that works regardless of when the scales were computed.

The Platinum benchmark results (Figure 13) validate this: for NVFP4, variants with ActOrder consistently outperform DefOrder regardless of other choices (Hadamard size, scale method). For MXFP4, ActOrder is uniformly beneficial as well, with the top MXFP4 variant (Had128-MinMax+ActOrder) achieving 92.3% recovery vs. 84.9% for the worst (Ident-MinMax+DefOrder).

Integration into the GPTQ Algorithm

The full MR-GPTQ algorithm integrates these three ingredients into GPTQ's existing framework. The standard GPTQ algorithm [21] works as follows for a weight matrix W of size d_row × d_col:

Compute the inverse Hessian H^(-1) = (2X^T X + λI)^(-1) from calibration inputs X, where λ is a dampening factor (set to 10^(-2) in all experiments).
Process columns sequentially (left to right). For each column i:
- Quantize all weights in column i using the pre-computed grid and scales.
- Compute the quantization error ε_i = W_{:,i} - Ŵ_{:,i} for all rows.
- Update all remaining (unquantized) columns j > i: W_{:,j} ← W_{:,j} - ε_i · (H^(-1)_{i,j} / H^(-1)_{i,i}).
- Update the inverse Hessian to remove column i (a rank-1 update).

The complexity is O(max(d_row · d_col², d_col³))—a quadratic factor in d_col from processing each column and updating all subsequent columns, but linear in d_row because all rows share the same Hessian and column processing order.

MR-GPTQ's modifications to this pipeline:

Before quantization: Apply block-wise Hadamard rotation to W → W' = W H_k, where H_k is applied along the input dimension (columns). Compute optimal scales s_T and s_Gj on W' using either alternating optimization (NVFP4) or static fitting (MXFP4). Compute Hessian H from rotated activations X' = X H_k.
During quantization: Reorder columns of W' by descending H_{ii}. Process in GPTQ's standard order, using the pre-computed scales and the FP4 quantization grid. After quantization, restore the original column order. Store Q(W H_k) (the quantized rotated weights) in the original column layout.
At inference: Apply H_k to input activations online, quantize to FP4, multiply with stored quantized weights, and dequantize the output to FP16/BF16.

Why this integration is non-trivial: The Hadamard rotation changes the weight distribution (Laplace → Normal), which changes the optimal quantization scales. Computing scales after rotation is essential—rotating after scale computation would invalidate the scales because the per-group maximum would change. The activation reordering must also happen after scale computation (to avoid permuting the group structure) but before GPTQ's iterative updates (so the error allocation benefits from the ordering). The paper's careful ordering of these steps is what distinguishes MR-GPTQ from simply applying rotations and GPTQ independently.

QuTLASS GPU Kernel Design (Section 4.2)

The QuTLASS kernel library implements the inference-time operations needed for MR-GPTQ on NVIDIA Blackwell GPUs (SM100 for B200, SM120 for RTX5090). It consists of two categories of kernels:

1. Quantization-related kernels (online rotation + quantization). These implement the computation Q(X H_k) where X is the FP16/BF16 activation input. The kernel is structured as:

Input: FP16 activation matrix X (size batch × d_in), pre-computed k × k rotation matrix H_k (fits in registers/shared memory for k < 256).
Operation: For each block of k columns, load a batch × k tile of X and the k × k rotation matrix, perform the dense matrix multiplication X_tile × H_k (memory-bound operation), compute per-group scales via either absmax or MSE (selectable at runtime), quantize each element to FP4 using the computed scales and E2M1 grid, and write the quantized values and scales to output buffers in the hardware-required layout for tcgen05.mma instructions.
Key optimization: The rotation, scale computation, and quantization are fused into a single kernel with a custom epilogue function. This avoids writing the rotated FP16 activations to memory (which would require a separate read in the matmul kernel) and eliminates kernel launch overhead. The template-based design allows plugging in new quantization methods (absmax, MSE, or custom) without modifying the rotation logic.

The kernel supports "unimodal" block diagonal matrices with k ∈ {16, 32, 64, 128}. For k < 256, any orthogonal transform (Hadamard, DCT, DST, GSR) costs essentially the same because the limiting factor is loading X from memory, not the k × k multiply (which is tiny compared to the batch × d_in data movement). This is validated in Appendix K (Tables 13–14), where throughput differences between transform types are within measurement noise.

2. Matmul-related narrow precision kernels. These handle the FP4 matrix multiplication C = Q(W H_k) × Q(X H_k)^T. The kernel must:

Rearrange scales into the hardware-mandated layout for tcgen05.mma (Blackwell's tensor core instruction for microscaling formats). The NVIDIA cuBLAS documentation specifies that block scales must be interleaved with data in a specific pattern; QuTLASS implements this rearrangement using a Triton kernel.
Execute the FP4 matmul using one of multiple backends: CUTLASS (NVIDIA's template library, providing maximum control over tensor core instructions) or FlashInfer (optimized for LLM inference patterns with variable sequence lengths). The plug-and-play backend selection allows choosing the optimal implementation per workload shape and batch size.

Why MXFP4 matmul can be faster than NVFP4 (Figure 5). Despite NVFP4's smaller group size (which should mean less overhead from scale loading), MXFP4 achieves up to ~15% higher throughput on B200. The paper hypothesizes three contributing factors:

Power-of-two scale arithmetic: MXFP4's E8M0 scales are exact powers of two, so the dequantization step (multiplying the dot product by the scale product) is a simple integer exponent addition followed by a bit shift—much cheaper than E4M3's full FP8 multiply.
Larger group size (G=32 vs. 16): Fewer scale elements to load from memory per output element, reducing memory bandwidth pressure for the scale stream.
Hardware implementation details: The tensor core instruction scheduling may be more efficient when scales are coarser, though this is speculative and noted as an area for future investigation.

Performance bounds (Figure 5). The "ideal" curve represents the raw FP4 matrix multiplication throughput without any quantization-related overhead—just the matmul itself. The "actual" curves include the online Hadamard rotation, scale computation, and quantization. The gap between ideal and actual shrinks as batch size increases because the matmul cost grows with O(batch) while the rotation cost is a one-time per-layer overhead. At batch size 8192, MXFP4 with H_128 achieves ~5500 TFLOPs/s out of an ideal ~5800—a ~5% overhead. NVFP4 achieves ~4800 out of an ideal ~5300—a ~9% overhead.

End-to-end inference (Figure 6, Figure 12). Integrated into vLLM, QuTLASS achieves:

B200: 2.2× speedup over BF16 for Llama-3.3-70B with MXFP4 at batch size 8192, reaching ~15,000 tokens/sec (Figure 12). For small batch sizes (memory-bound regime), weight-only NVFP4 quantization provides ~20% lower latency than FP8 (Figure 12, right panel).
RTX5090 (SM120): Up to 3.8× speedup for Qwen3-8B with MXFP4 (Figure 11, right panel), and ~6× layer-wise speedup (Figure 11, left panel). The SM120 architecture's higher FP4 throughput (relative to FP16) accounts for the larger speedup vs. B200.

The paper explicitly notes that small-batch inference is memory-bound and benefits only from weight quantization (activation quantization adds computation without reducing memory traffic). This is a practical deployment consideration: use FP8 or weight-only FP4 for interactive single-user inference, and full W4A4 for high-throughput batch processing where the matmul dominates.

4. Key Insights and Innovations

Innovation 1: Recasting FP4 Microscale Quantization as a Distribution-Format Interaction Problem, Not a Grid-Optimization Problem

The dominant mental model in post-training quantization since at least GPTQ (Frantar et al., 2022) has been: given a fixed quantization grid, optimize the weight assignment and error compensation to minimize output distortion. This framing treats the quantization format as a constraint—a fixed target that algorithms must work around—and focuses innovation entirely on the algorithm side: better Hessian approximations, smarter rounding, learned rotations. What makes this paper's analytical framework distinctive is that it inverts this relationship: it treats the format's error properties as the object of study and derives which algorithmic interventions are appropriate as a function of the format, not as universally applicable techniques.

The critical conceptual move is in Section 3's pairing of distributional assumptions (Laplace for native tensors, Normal for rotated tensors) with format-specific error decomposition (per-element MSE vs. top-element MSE, base precision vs. scale precision). Before this work, the community understood rotations as "making distributions more Gaussian, which helps quantization." The papers on QuIP (Chee et al., 2024) and QuaRot (Ashkboos et al., 2024) established this empirically and theoretically for INT formats. What no one had articulated is that rotations interact with microscaling formats through a specific mechanism—the top-element error spreading documented in Lemma 1—and that this mechanism has opposite effects depending on whether the scale format provides more or less precision than the base element format.

This is not an incremental refinement of the "rotations help" narrative. It is a diagnostic framework that explains when that narrative holds (MXFP4: E8M0 scales are coarser than E2M1 elements, so spreading top-element error is net-beneficial) and when it fails (NVFP4 with RTN: E4M3 scales are finer than E2M1 elements, so destroying the implicit outlier promotion by spreading top-element error is net-harmful). This is visible in Table 1: NVFP4 RTN drops from 74.73% to 74.05% with rotations; MXFP4 RTN improves from 69.32% to 70.45%.

The significance extends beyond performance. This framework provides a decision procedure for practitioners: measure whether your format's scale precision exceeds its element precision. If yes, rotations are likely harmful under RTN but may help under error-compensating algorithms like GPTQ (where the per-element MSE reduction compensates for lost top-element precision). If no, rotations are unambiguously beneficial. Prior work offered no such diagnostic; practitioners would have to run costly ablation sweeps to discover these conditional effects empirically.

The asymptotic analysis in Lemma 2 is another example of this reframing. The crossover phenomenon—Laplace distributions quantize better than Normal at small group sizes but worse at large ones—is not just a theoretical curiosity. It explains why NVFP4's group size of 16 is a sweet spot where native distributions outperform rotated ones, and why MXFP4's group size of 32 pushes the trade-off in the other direction. This connects format design choices (group size) to algorithmic choices (whether to rotate) through a single, interpretable mechanism—dead-zone survival probability—rather than through opaque empirical sweeps.

The extension to hypothetical INT4 microscaling formats (Appendix D, Figure 10) demonstrates the framework's generality. NVINT4, with E4M3 scales and INT4 base elements, has the same scale precision advantage as NVFP4—but INT4 elements are uniform, so the "top-element promotion" logic is less clear. The prediction (and empirical validation) that NVINT4 does benefit from rotations unlike NVFP4 suggests that the framework captures something beyond format-specific quirks: it isolates the interaction between scale quantization granularity and element quantization granularity as the fundamental variable.

This reframing is fundamental, not incremental, because it changes what questions the field should ask. Instead of "which quantization algorithm works best for FP4?", the question becomes "for a given format specification (group size, scale precision, element precision), which distributional properties of the tensor determine the optimal algorithmic intervention?" This is a more general, more predictive question that applies to future formats (FP6, MXFP8 variants, custom microscaling specifications) without requiring new empirical sweeps.

Innovation 2: Static Activation Reordering as a Zero-Cost Accuracy Improvement That Decouples Quantization Order From Runtime Layout

GPTQ's dynamic activation ordering—reordering weight columns by descending Hessian magnitude before quantization, then reversing the permutation at runtime—was known to improve accuracy by 1–2% on INT4 quantization (Frantar et al., 2022). It was also known to impose a 10–20% inference slowdown due to the online column shuffle. The standard trade-off was clear: if you need maximum accuracy, pay the runtime cost; if you need maximum throughput, skip act-order. The field accepted this as a fundamental tension between accuracy and performance.

MR-GPTQ's static activation reordering breaks this assumed trade-off by recognizing that the accuracy benefit of act-order comes from the order in which GPTQ allocates error-compensation budget during quantization, not from any property of the runtime layout. The runtime reshuffling is an implementation artifact of the original GPTQ design, where scales were computed after reordering. By decoupling these steps—compute scales and grids in the original column order, reorder for GPTQ's iterative compensation, then restore the original order—the method preserves the error-allocation benefit while eliminating the runtime penalty entirely.

This is a small refinement mechanically but a significant conceptual contribution because it refutes an assumed constraint. The prior work had implicitly baked the reordering into the quantization pipeline as an atomic operation; MR-GPTQ shows that the operation has two separable effects (scale computation dependence on column order, error allocation dependence on column order) and that only the second matters for accuracy. This is not obvious—one might have hypothesized that computing scales on the original order before reordering would misalign them with the reordered columns during GPTQ's updates, but the empirical results (Figures 13, 14, Table 12) show that ActOrder variants consistently outperform DefOrder variants regardless of other design choices, with no degradation from the static approach.

The Platinum benchmark results (Figure 13a) quantify this: for NVFP4, the top three variants all use ActOrder, with Had16-MSE+ActOrder achieving 96.57% recovery. For MXFP4 (Figure 13b), ActOrder variants occupy the top six positions, with a ~7-point gap between the best ActOrder variant (92.32%) and the best DefOrder variant (85.93%). The standard deviation analysis (Table 12) confirms these gaps are statistically significant relative to the measured variance (0.3–0.8 points for NVFP4, 0.4–2.3 points for MXFP4).

The broader implication is that other quantization algorithms with similar runtime-layout coupling (e.g., SpQR's sparse-dense separation, SqueezeLLM's sensitivity-based grid allocation) may have analogous decoupling opportunities. This paper doesn't explore those, but the concept of separating quantization-time ordering from runtime layout is generalizable beyond GPTQ.

Innovation 3: Empirical Demonstration That Format Choice Is Not a Secondary Concern—It Determines Which Algorithms Work

The dominant assumption in the LLM quantization literature, reinforced by vendor marketing (NVIDIA's FP4 accuracy claims [40; 43]), is that floating-point formats are inherently more accurate than integer formats at the same bit-width because they allocate more representational capacity to small values and provide better dynamic range. Under this assumption, the choice between NVFP4 and MXFP4 should be a straightforward assessment of accuracy-vs-cost: NVFP4 costs 0.25 more bits per element for supposedly better accuracy; MXFP4 is cheaper but might be slightly less accurate. The algorithm you use (RTN, GPTQ, SmoothQuant) should transfer straightforwardly.

The paper's comprehensive evaluation (Table 1, Tables 3–10, Figure 7) systematically demolishes this assumption. The key empirical findings are:

1. No format is lossless. Even the best method on the best format (NVFP4 with MR-GPTQ on Llama-3.1-8B-Instruct) achieves only 96.08% average recovery—a ~3-point drop from FP16. For smaller models (Llama-3.2-1B), recovery drops to 87.42% even with MR-GPTQ (Table 3). Weight-only quantization (Appendix A, Table 2) shows a 2-point drop on average even for NVFP4. The formats are lossy, and practitioners should expect accuracy degradation proportional to model size and task difficulty.

2. The same algorithm yields opposite effects across formats. Hadamard rotations improve MXFP4 RTN by ~1.1 points but hurt NVFP4 RTN by ~0.6 points (Table 1). QuaRot—a method designed specifically to make quantization easier—achieves only 74.10% average on NVFP4, below simple RTN at 74.73%, and catastrophically fails on MXFP4 at 62.90% (vs. 69.32% for RTN). SmoothQuant, the standard activation quantization method, helps NVFP4 (95.90% recovery) but barely moves MXFP4 (89.06%). There is no universally effective algorithm across formats.

3. MXFP4's accuracy penalty is substantially worse than its bit-rate savings would suggest. At 0.25 fewer bits per element than NVFP4, MXFP4 with RTN achieves 87.83% recovery vs. 94.67% for NVFP4—a ~7-point gap. Applying the best method narrows this to ~3 points (MR-GPTQ: 93.31% vs. 96.08%), but even the MXFP4† modification (Appendix H, Table 11) only partially closes it. The implication is that MXFP4's power-of-two scale constraint is not a minor implementation detail—it is the dominant source of quantization error, and it cannot be fully compensated by algorithmic cleverness.

4. INT4 with microscaling-style group sizes is competitive with FP4, and NVINT4 with rotations is the best overall format. Tables 1, 3–5, and 7 consistently show that NVINT4 (hypothetical INT4 with E4M3 scales, G=16) with MR-GPTQ achieves the highest recovery of all formats: 97.12% on Llama-3.1-8B (vs. 96.08% for NVFP4), 96.27% on Llama-3.2-3B (vs. 95.51% for NVFP4 GPTQ). This finding—that a non-existent format outperforms the shipping hardware formats—is both a validation of the paper's distribution-format interaction framework (which predicted this in Section 3.2 and Appendix D) and an implicit critique of the OCP standard's choice to specify only floating-point base types.

This is a fundamentally important negative result disguised as a benchmarking exercise. It demonstrates that the industry has converged on microscaling format specifications (MXFP4 as an open standard, NVFP4 as a proprietary alternative) without systematic evaluation of whether these specifications are near-optimal for the actual distributional properties of LLM weights and activations. The paper's analysis suggests that a hypothetical format combining INT4 base elements with E4M3 scales (NVINT4) would outperform both existing formats, and that the OCP's choice of E8M0 for scales is the primary bottleneck in MXFP4. These are actionable findings for future format standardization efforts.

Innovation 4: Verifier Over-Optimization Does Not Apply Here, but Format Over-Optimization Does—A New Category of Quantization Failure Mode

The quantization literature has long recognized that minimizing layer-wise MSE does not guarantee end-to-end accuracy preservation, because quantization errors can compound across layers in ways that simple distortion metrics miss. This has motivated output-reconstruction methods like GPTQ (which minimizes layer output error rather than weight error) and block-wise methods like OBC/OBQ. The implicit assumption is that better optimization within the quantization algorithm (more sophisticated error compensation, better Hessian approximations) will monotonically improve accuracy, up to the limit imposed by the format itself.

The paper's results reveal a more nuanced picture: algorithmic sophistication can over-optimize to the format's specific error profile, improving MSE at the cost of distributional properties that matter for end-to-end accuracy. This is most evident in two places:

Hadamard rotations worsen NVFP4 under RTN but improve it under GPTQ. Under RTN, rotating NVFP4 from native Laplace to Normal distributions increases per-element MSE at G=16 (Figure 3, top row, NVFP4: MSErel goes from ~0.010 to ~0.015 with HT). This directly explains the 0.6-point accuracy drop in Table 1. Yet under MR-GPTQ, the rotated NVFP4 variant matches or exceeds the unrotated variant (96.08% vs. 95.92% for GPTQ without rotations). The mechanism is that GPTQ's error compensation can "fix" the increased per-element MSE from rotation by shifting error to less important weights—something RTN cannot do because it has no error compensation. The rotation's distributional normalization (making all weights equally "important" for MSE allocation) actually helps GPTQ's Hessian-based compensation work better, even though the raw per-element distortion is higher.

The Had16 vs. Had128 block size results (Figure 13) show a format-dependent optimum. For NVFP4, Had16 (matching the group size) achieves 96.57% recovery vs. 93.95% for Had128. For MXFP4, Had32 achieves 91.73% vs. 92.32% for Had128—essentially tied, with both far above the non-rotated variants (~85%). The optimal rotation block size is not simply "as large as possible for maximum normalization" (the QuIP# assumption), but rather matches the quantization group size for NVFP4 and is less sensitive for MXFP4. This is a previously unknown interaction between rotation granularity and microscaling block structure.

This is not an incremental optimization finding—it identifies a new category of interaction: the quantization algorithm's error compensation mechanism (GPTQ's column-wise updates) has a different optimal input distribution than the format's raw quantization fidelity (RTN). Methods that optimize for one can regress on the other. This has implications beyond FP4: any format with non-uniform error properties across elements (e.g., non-uniform grids, per-group scaling, mixed-precision schemes) may exhibit similar algorithm-format interactions where the algorithm's compensation strategy needs to be matched to the format's error structure, not just optimized in isolation.

This also contextualizes the paper's seemingly contradictory advice: "don't use rotations for NVFP4 with RTN, but do use them with GPTQ." The contradiction is real—rotations simultaneously hurt raw quantization fidelity and help error compensation—and resolving it requires understanding that GPTQ is not just "RTN with error correction added." RTN optimizes per-element fidelity; GPTQ optimizes per-output-channel fidelity, which is a different objective that responds differently to distributional changes. The paper's contribution is demonstrating that this distinction matters at the level of format choice and algorithm selection, not just as an implementation detail of GPTQ.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses a subset of tasks from the Open LLM Leaderboard V1 [5]: GSM8K [10] for grade-school math, MMLU [26; 9] for world knowledge and reasoning, HellaSwag [58] and WinoGrande [48] for language understanding. Additional analyses use the PlatinumBench benchmark [53], a carefully curated suite designed to reduce noise in LLM evaluation. All experiments use standard zero-shot evaluation protocols.

Base model(s). The main experiments center on Llama-3.1-8B-Instruct [18], with additional results across the Llama-3 family (Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.3-70B-Instruct) and the Qwen-3 family [55] (Qwen-3-8B, Qwen-3-14B, Qwen-3-32B). The paper argues this coverage "spans a representative range of modern LLM scales and architectures" (Section 5, introductory text). For Qwen-3 models, the authors turn off thinking mode because long reasoning chains-of-thought proved detrimental on GSM8k and MMLU-CoT (Appendix B note).

Metrics. The primary metric is average accuracy recovery (%) — the ratio of the quantized model's average score across tasks to the FP16 baseline's average score, expressed as a percentage. Individual task accuracies are also reported. For the quantization error analysis (Section 3), the paper uses relative per-element MSE (MSErel) and relative top-element MSE (MSErel_top), as defined in Definition 4. For kernel performance, TFLOPs/s and tokens/sec are reported.

Baselines. For weight-and-activation quantization (W4A4), the paper compares against: (1) Round-to-nearest (RTN) quantization to the target format with absmax scales; (2) RTN with block-wise Hadamard rotations matching the quantization group size, denoted RTN+HT; (3) SmoothQuant [54] diagonal rescaling with tuned α=0.6; (4) QuaRot [4] with RTN post-rotation; (5) SpinQuant [38] with trained rotations on 1024 FineWeb calibration sequences; (6) GPTQ [21] weight quantization with RTN on activations, using 1024 FineWeb calibration sequences, absmax scales, and standard Hessian dampening factor λ=10⁻². For weight-only quantization (W4A16, Appendix A), additional baselines include AWQ [36]. INT8 and FP8 baselines are included as near-lossless reference points.

Generation budget / compute accounting. In the simulated quantization experiments (PyTorch), the "budget" is not measured in generations but in the quantization group size and bit-width, which determine memory footprint. NVFP4 uses group size 16 and E4M3 scales, totaling 4.5 bits per element on average. MXFP4 uses group size 32 and E8M0 scales, totaling 4.25 bits per element. INT4 baselines use group size 32 with FP16 scales, matching NVFP4's average bit-width of 4.5 bits per element. For the real-kernel experiments, computation is measured in observed throughput (TFLOPs/s) and end-to-end tokens/sec relative to BF16.

Cross-validation / statistical protocol. For the simulated quantization results in Table 1, the paper reports variance estimates: NVFP4 experiments have approximately 0.3 average points standard deviation over 5 distinct seeds, while INT4 experiments show approximately 1 point standard deviation. For the Platinum benchmark comparisons (Appendix I, Figure 13, Table 12), the paper reports averages and standard deviations across five random seeds for calibration set sampling, scale selection, and quantization ordering strategies. Top-performing entries within 2 standard deviations are marked in bold.

Main Quantitative Results

W4A4 Quantization on Llama-3.1-8B-Instruct (Table 1)

The headline finding is that no format achieves lossless compression, and the best method on the best format (NVFP4 with MR-GPTQ) recovers only 96.08% of FP16 accuracy on average, corresponding to a drop from 78.93% to 75.84% across the four evaluation tasks.

Format ranking by best achievable accuracy:

NVFP4: 75.84% average with MR-GPTQ (96.08% recovery), or 75.72% with GPTQ (95.92% recovery). SmoothQuant achieves 75.70% (95.90% recovery). These three results are within variance of each other.
INT4: 74.75% with RTN+HT (94.71% recovery), though GPTQ-based INT4 achieves only 73.21% (92.75% recovery). The paper notes that INT4 has higher variance (~1 point) than NVFP4 (~0.3 points), making the top NVFP4 and INT4 results statistically indistinguishable.
MXFP4: 73.65% with MR-GPTQ (93.31% recovery), substantially below both NVFP4 and INT4. RTN achieves only 69.32% (87.83% recovery).

The format gap is method-dependent:

Under RTN alone, NVFP4 achieves 74.73% vs. MXFP4's 69.32% — a 5.41-point gap.
Under MR-GPTQ, NVFP4 achieves 75.84% vs. MXFP4's 73.65% — a 2.19-point gap, showing that format-specialized methods can partially but not fully close the gap.
SmoothQuant is the best method for NVFP4 under RTN-like activation handling (75.70%), but barely helps MXFP4 (70.30%).

Hadamard rotation effects are format-conditional:

Adding HT to RTN quantization: NVFP4 drops from 74.73% to 74.05% (−0.68 points), while MXFP4 improves from 69.32% to 70.45% (+1.13 points).
Adding HT to GPTQ (MR-GPTQ): NVFP4 improves from 75.72% to 75.84% (+0.12 points, within variance), while MXFP4 improves from 70.62% to 73.65% (+3.03 points).

QuaRot and SpinQuant perform poorly on FP4:

QuaRot on NVFP4 achieves 74.10% (below RTN at 74.73%). On MXFP4, it collapses to 62.90% — a 16-point drop from FP16 and substantially worse than any other method. This is the single worst result in Table 1 for MXFP4.
SpinQuant on NVFP4 achieves 73.70%, also below RTN. On MXFP4, it achieves 69.40%, comparable to RTN.

INT8 and FP8 are near-lossless: INT8 RTN achieves 78.73% (99.74% recovery), FP8 RTN achieves 78.65% (99.64% recovery). The drop is negligible — these formats serve as "controls" confirming that the evaluation harness can detect accuracy degradation when it exists.

Real Quantization Results Across Model Families (Tables 3–10, Table 10)

The paper validates its simulated results with real-kernel evaluations in vLLM using QuTLASS, extending to additional model sizes and families. Table 10 provides a per-model recovery summary.

Model size matters substantially:

Llama-3.2-1B-Instruct with NVFP4 RTN: 83.91% recovery (Table 3). With MR-GPTQ: 87.35% recovery — a substantial drop for small models.
Llama-3.2-3B-Instruct with NVFP4 RTN: 94.45%; with MR-GPTQ: 93.68% (Table 4). The paper reports a degradation from MR-GPTQ vs. GPTQ alone (95.51%) at this scale, suggesting the rotation trade-off is more delicate for intermediate sizes.
Llama-3.1-8B-Instruct: recovery exceeds 95% for the best NVFP4 methods (Table 5).
Llama-3.3-70B-Instruct with NVFP4 GPTQ: 99.07% recovery (Table 6). At this scale, both NVFP4 and MXFP4 exceed 98% recovery with the best methods.

Architecture family differences are pronounced:

Qwen-3-8B with NVFP4 RTN achieves 98.86% recovery (Table 7) vs. Llama-3.1-8B's 94.80% — a ~4-point advantage for Qwen. This pattern holds across formats: Qwen models consistently achieve higher recovery rates than comparably-sized Llama models.
Qwen-3-32B with NVFP4 RTN reaches 99.76% recovery (Table 9), while MXFP4 RTN achieves only 91.83% — the largest format gap in the study, at nearly 8 points.
MXFP4 on Qwen-3-32B with RTN+Had32 suddenly jumps to 98.67% recovery, dramatically outperforming the unrotated variant. The paper does not explain this anomalous improvement in detail, but it is consistent with the prediction that MXFP4 benefits from rotations more at larger model scales where weight distributions are better normalized.

Quantization-Aware Training (QAT) benefits are format-dependent:

For NVFP4, QAT provides marginal improvements: Llama-3.1-8B QAT achieves 95.40% recovery vs. 95.70% for GPTQ (Table 5). QAT+Hadamard reaches 96.50%, within range of MR-GPTQ's 95.80%.
For MXFP4, QAT provides more substantial gains: Llama-3.1-8B QAT achieves 92.30% recovery vs. 89.66% for GPTQ; QAT+Had32 reaches 95.40%, closing the gap to NVFP4 methods. On Llama-3.2-1B, MXFP4 QAT+Had32 achieves 84.55% recovery vs. unquantized 53.47% average — still poor in absolute terms but substantially better than RTN (67.70%) or GPTQ (68.45%) for that model (Table 3).

Integer Microscaling Formats (Tables 1, 3–5, 7; Figure 10)

The paper introduces hypothetical NVINT4 and MXINT4 formats and evaluates them alongside the real FP4 formats.

NVINT4 with rotations achieves the best overall PTQ accuracy:

Llama-3.1-8B-Instruct: NVINT4 MR-GPTQ reaches 76.66% average (97.12% recovery), surpassing NVFP4 MR-GPTQ at 75.84% (96.08%) and all other combinations (Table 1).
Llama-3.2-3B-Instruct: NVINT4 MR-GPTQ achieves 68.82% (96.27% recovery), vs. NVFP4 GPTQ at 68.28% (95.51%) (Table 4).
The NVINT4 advantage is consistent but modest (~1 point on average), and the paper's error analysis in Figure 10 predicted this: NVINT4's E4M3 scales are finer than INT4's uniform grid, so rotations spread top-element error beneficially, unlike NVFP4 where E4M3's promotion of top elements is "destroyed" by rotation.

MXINT4 is substantially worse than MXFP4:

Llama-3.1-8B: MXINT4 RTN achieves 63.05% (79.87% recovery) vs. MXFP4 RTN at 69.32% (87.83%) — a 6-point disadvantage for the integer variant (Table 1).
With MR-GPTQ, MXINT4 reaches 72.51% (91.86% recovery), still well below MXFP4 MR-GPTQ at 73.65% (93.31%).
On small models, MXINT4 is catastrophic: Llama-3.2-1B achieves only 38.83% recovery with MR-GPTQ (Table 3).

QuTLASS Kernel Performance (Figures 5–6, 11–12)

Layer-wise throughput (Figure 5, B200):

Actual MXFP4 with Hadamard (H_128) reaches ~5,500 TFLOPs/s at batch size 8,192, vs. ~5,800 for ideal MXFP4 (no quantization overhead) — a ~5% overhead.
Actual NVFP4 reaches ~4,800 TFLOPs/s vs. ~5,300 ideal — a ~9% overhead.
BF16 achieves ~1,500 TFLOPs/s. The speedup of MXFP4 over BF16 is ~3.6× at batch size 8,192.
MXFP4 outperforms NVFP4 throughput by up to ~15% at large batch sizes, despite the additional Hadamard transform. The paper attributes this to MXFP4's power-of-two scale arithmetic and larger group size reducing scale-loading overhead.

RTX5090 performance (Figure 11, left):

For a single layer from Qwen3-32B, MXFP4 achieves ~1,200 TFLOPs/s vs. ~200 for BF16 — a ~6× speedup.
Ideal W4A4 (no quantization ops) achieves ~1,300 TFLOPs/s, meaning the actual quantization overhead is ~8%.
Both Had32 and Had128 variants achieve essentially identical throughput, confirming the memory-bound nature of the rotation.

End-to-end inference (Figure 6, B200):

Llama-3.3-70B with MXFP4 achieves up to 2.2× speedup over BF16 across batch sizes 32 to 8,192 in vLLM.
The speedup is relatively flat across batch sizes, suggesting that the matmul dominates end-to-end time even at moderate batch sizes for large models.

End-to-end on RTX5090 (Figure 11, right):

Qwen3-8B with MXFP4 achieves up to 3.8× speedup over BF16 using the Transformers library integration.
The speedup increases with batch size, consistent with the matmul becoming more dominant relative to attention and other operations.

Small-batch vs. large-batch regimes (Figure 12):

For large batches (≥64), MR-MXFP4 achieves the highest throughput (~15,000 tok/sec at batch 512) — 2.2× over BF16 and 1.3× over FP8.
For small batches (≤32, corresponding to single-user text generation), the FP4 quantized formats show "little to no improvement over BF16 and FP8" because inference is memory-bound and activation quantization provides no benefit. In this regime, weight-only FP4 quantization provides ~20% lower latency than FP8 (Figure 12, weight-only curve).

Detailed Format and Method Ablation on PlatinumBench (Figures 13–17, Appendix I)

The standard evaluation harness struggles to distinguish between top methods due to high noise in some tasks (the paper notes that "GSM8k and MMLU-CoT have particularly high variance"). The PlatinumBench benchmark, with curated low-noise tasks, provides finer discrimination.

NVFP4 component ablation (Figure 13a, 13c, 16):

The top configuration is Had16-MSE+ActOrder at 96.57% recovery (standard deviation 0.746).
The gap between the best (Had16-MSE+ActOrder) and worst (Ident-MinMax+ActOrder at 93.27%) among the non-trivial variants is ~3.3 points, which is significant relative to the measured standard deviations (0.3–1.7 points).
Hadamard rotations with block size 16 (matching the quantization group size) outperform block size 128 (96.57% vs. 93.95% for Had128-MSE+ActOrder), and both outperform identity transforms (93.27–94.20%).
MSE scale optimization is slightly but consistently better than MinMax (e.g., Had16-MSE+ActOrder at 96.57% vs. Had16-MinMax+ActOrder at 93.77%), confirming that the joint scale optimization provides measurable gains.
Activation ordering consistently helps: all four ActOrder variants appear in the top five configurations, and every ActOrder variant outperforms its corresponding DefOrder variant.

MXFP4 component ablation (Figure 13b, 13d, 15):

The top configuration is Had128-MinMax+ActOrder at 92.32% recovery (standard deviation 1.128).
The gap between rotated variants (~91–92%) and unrotated variants (~85%) is ~6–7 points — dramatically larger than for NVFP4, confirming that rotations are substantially more important for MXFP4.
Had32 and Had128 are statistically tied (91.73% vs. 92.32%), unlike NVFP4 where block size matching the group size was clearly optimal. This suggests MXFP4 benefits from normalization at multiple granularities.
MSE vs. MinMax optimization shows inconsistent effects: for Had32, MSE outperforms MinMax (91.73% vs. 91.73% — identical); for Had128, MinMax outperforms MSE (92.32% vs. 91.86%). The differences are within measurement noise, suggesting scale optimization is less impactful for MXFP4 than the rotation itself.

Ablation Studies and Robustness Checks

Weight-only quantization (Appendix A, Table 2): For Llama-3.1-8B-Instruct, INT4 RTN achieves 77.12% (97.71% recovery), NVFP4 RTN achieves 77.37% (98.02%), and MXFP4 RTN achieves 75.44% (95.58%). Even in the weight-only setting where the 4-bit format only compresses weights (activations remain FP16), MXFP4 suffers a ~2% absolute accuracy drop. Hadamard rotations hurt both NVFP4 (77.37% → 76.41%) and MXFP4 (75.44% → 73.81%) in the weight-only setting. AWQ on NVFP4 achieves 77.40% (98.06%), marginally better than RTN. The errors from weight and activation quantization are "roughly evenly split" according to the paper.

Scale quantization format comparison (Appendix C, Figures 8–9): An ablation over FP8 scale formats (E1M6 through E6M1, plus INT8) for weight and activation quantization with group size 16 shows that:

E4M3 and E8M0, the formats used by NVFP4 and MXFP4 respectively, are not optimal for weight scale quantization. E4M3 increases MSErel by ~10% relative to FP16 scales, while E8M0 increases it by ~40%.
Formats with larger mantissa (E1M6–E3M4) and INT8 perform closer to unquantized FP16 scales, suggesting that future microscaling format designs should allocate more bits to mantissa precision in the scale representation.
The activation pattern is similar except for the down_proj layer, which has heavier-tailed distributions and is more sensitive to scale quantization.

MXFP4 scale fitting (Appendix H, Table 11): The proposed MXFP4† modification (exponent rescaling to fit the data range) substantially improves accuracy:

Llama-3.1-8B RTN: 87.83% → 94.30% recovery (+6.47 points).
Llama-3.1-8B MR-GPTQ: 93.31% → 94.87% recovery (+1.56 points).
Qwen-3-8B RTN+HT: 93.56% → 96.30% recovery (+2.74 points).
MXFP4† brings MXFP4 to within 0.4–2.6 points of NVFP4 across methods, demonstrating that MXFP4's accuracy gap is primarily attributable to the E8M0 scale format's excessive dynamic range, not to the FP4 element format itself.

Different linear transforms and block sizes (Appendix K, Tables 13–14): A sweep over Hadamard, DCT, DST, and GSR transforms with block sizes 16, 32, 64, 128, 256 shows:

For NVFP4 with RTN, all transforms perform similarly to identity (averages ~0.695–0.706 vs. 0.708 for identity), with a slight degradation for most settings. No transform-block-size combination substantially outperforms the unrotated baseline.
For MXFP4 with RTN, Hadamard with block size 32 achieves the best result (0.694 average vs. 0.678 for identity), confirming that rotations help MXFP4. GSR with block sizes 32–256 is competitive. The optimal block size is not sharply defined — many configurations are within noise.
For NVFP4 with GPTQ, the best configuration is GSR with block size 256 (0.714 average), but the variation across transforms and sizes is small (range ~0.692–0.714), suggesting GPTQ's error compensation is relatively robust to the specific rotation choice.
For MXFP4 with GPTQ, Hadamard with block size 64 achieves the best result (0.703) among rotated variants, substantially above identity GPTQ (0.684). The transform type matters more for MXFP4 than NVFP4.

GPTQ component-level ablation on the standard benchmark suite (Figure 14, Table 12): Unlike the Platinum benchmark where differences are stark, the standard benchmark suite groups many methods within overlapping standard deviations:

For NVFP4, all eight configurations achieve recovery within ~95.5–96.4%, with standard deviations of 0.16–0.66 points. The paper cannot statistically distinguish between most variant pairs on this benchmark.
For MXFP4, the rotated variants (92.8–93.8%) are clearly above the unrotated variants (89.2–90.5%), but the differences among rotated variants are within noise.
This demonstrates a negative but important result: the standard LLM evaluation harness is too noisy to reliably compare FP4 quantization methods that differ by less than ~2 points in average accuracy. The Platinum benchmark provides the necessary resolution.

Critical Assessment

Does the paper demonstrate that state-of-the-art methods struggle with FP4 formats?

Yes, with strong evidence from multiple angles. The claim is supported by:

RTN baseline comparisons (Table 1): The simplest quantization method (round-to-nearest) already reveals the format gap — NVFP4 RTN at 94.67% recovery vs. MXFP4 RTN at 87.83%. This gap cannot be explained by implementation details; it's inherent to the formats.
Rotation-based methods regressing on NVFP4 (Table 1): QuaRot, explicitly designed to improve quantization, achieves 93.80% recovery on NVFP4 — below RTN at 94.67%. SpinQuant achieves 93.40%. These are not marginal degradations; they demonstrate that applying methods developed for INT quantization to FP4 without format-specific adaptation actively harms accuracy.
MXFP4's catastrophic failure with QuaRot (Table 1: 62.90%): A method that achieves 74%+ on NVFP4 and INT4 collapses to near-random performance on MXFP4. This is prima facie evidence that existing methods do not transfer.

However, the claim is slightly overstated in one respect: SmoothQuant (adapted with tuned α=0.6) actually works well across all formats — 95.90% on NVFP4, 89.06% on MXFP4, matching or exceeding GPTQ in several cases. It's not that "all methods struggle" — SmoothQuant, which was designed for activation quantization and does not assume a specific weight format, transfers more gracefully than rotation-based methods. The paper's narrative emphasizes the failures more than this partial success.

Does the paper demonstrate that NVFP4's small group size provably neutralizes traditional outlier mitigation?

Partially. The analytical proof is strong; the empirical demonstration is more nuanced.

The analytical argument (Section 3, Lemma 1 and Remark 2) proves that with absmax scaling, the top element in each group is perfectly preserved — zero quantization error. For a group size of 16, this means outliers within each block of 16 elements are automatically protected. This does indeed "neutralize" the need for explicit outlier mitigation techniques (per-channel scaling, mixed-precision outlier retention) that methods like SmoothQuant, AWQ, and SpQR were designed to provide when using larger group sizes or per-tensor scaling.

However, the empirical evidence for this claim is largely indirect — it comes from the fact that adding Hadamard rotations (which destroy the automatic outlier protection) hurts NVFP4 accuracy under RTN (Table 1: 74.73% → 74.05%). This is consistent with the theory, but the paper does not run a direct ablation where traditional outlier mitigation is applied to a large-group-size baseline and shown to be redundant at G=16. Such an experiment — e.g., comparing SmoothQuant with and without per-channel scaling at G=16 vs. G=128 — would provide direct evidence but is absent.

The INT4 comparison complicates the narrative: INT4 with group size 32 (Table 1) achieves similar accuracy to NVFP4 with group size 16 (74.75% vs. 74.73%), even though INT4's larger group size should leave outliers more exposed. The fact that INT4 handles outliers well at G=32 suggests that the "neutralization" effect is not exclusively a G=16 phenomenon — it may also be influenced by the uniform grid of INT4, which provides better coverage for outliers than FP4's non-uniform grid.

Does the paper demonstrate that MR-GPTQ bridges the gap between MXFP4 and NVFP4?

Yes, but with an important caveat about what "bridges the gap" means.

On Llama-3.1-8B-Instruct (Table 1), MR-GPTQ brings MXFP4 from 87.83% recovery (RTN) to 93.31% — closing the gap to NVFP4 RTN (94.67%) to within 1.36 points, and to NVFP4 MR-GPTQ (96.08%) to within 2.77 points. The paper's claim that MR-GPTQ brings MXFP4 "within 1-2% of NVFP4" is approximately correct for this specific model size.

However, the gap varies substantially across model scales (Table 10):

Llama-3.2-1B: MXFP4 MR-GPTQ at 79.80% vs. NVFP4 MR-GPTQ at 87.35% — a 7.5-point gap. The format gap is not bridged for small models.
Llama-3.2-3B: MXFP4 at 92.69% vs. NVFP4 at 93.68% — a 1-point gap. Bridged.
Llama-3.3-70B: MXFP4 at 98.40% vs. NVFP4 at 99.07% — a 0.7-point gap. Bridged.
Qwen-3-32B: MXFP4 RTN+Had32 at 98.67% vs. NVFP4 at 99.76% — a 1.1-point gap. But note this is achieved with RTN+Had32, not MR-GPTQ, suggesting that for Qwen models, simple rotation without GPTQ compensation suffices to bridge the MXFP4-NVFP4 gap.

The "1-2% within NVFP4" claim is model-scale-dependent and holds best for models in the 3B–70B range. For 1B models (and presumably smaller), MXFP4 remains substantially worse regardless of method. The paper does not emphasize this qualification, but the data in Table 10 make it clear.

Does the paper demonstrate that QuTLASS supports MR-GPTQ with negligible overhead?

Yes, with convincing evidence, though the claim is qualified by batch size.

Figure 5 shows that the "actual" MXFP4+H128 throughput (~5,500 TFLOPs/s) is within ~5% of "ideal" MXFP4 (no quantization overhead, ~5,800 TFLOPs/s) at large batch sizes (8,192). This is negligible overhead — the Hadamard transform, scale computation, and quantization add at most 5% to the raw matmul time.

At smaller batch sizes, the overhead grows: at batch size 512, the gap between actual and ideal is more visible, and at batch size 32 (not shown in Figure 5 but discussed in the text), FP4 quantization provides "little to no improvement over BF16." This is because small-batch inference is memory-bound — the matmul is not the bottleneck, so reducing its precision doesn't help. The paper is transparent about this (Figure 12 discussion), noting that weight-only quantization is preferable for the small-batch regime.

The RTX5090 results (Figure 11) show a similar pattern: ~6× layer-wise speedup over BF16, with the overhead being ~8% (1,200 actual vs. 1,300 ideal TFLOPs/s). The end-to-end speedup of ~3.8× on Qwen3-8B (Figure 11, right) is lower than the layer-wise speedup because the full model includes attention, layer norm, and other operations that are not accelerated by FP4.

A missing experiment: the paper does not benchmark QuTLASS against a naively implemented rotation (e.g., a separate cuBLAS gemm for the Hadamard followed by a separate quantization kernel) to quantify how much the fusion optimization matters. The claim that the fused kernel is "essential" for low overhead is plausible but not directly demonstrated.

Genuine weaknesses and missing experiments

The test set is small and task selection is narrow. The main evaluation uses only 4 tasks: GSM8k, MMLU, HellaSwag, WinoGrande. While these are standard, they do not cover instruction-following, code generation, long-form reasoning, or safety-critical behaviors. The Platinum benchmark (Appendix I) adds 14 tasks but is still limited to English text understanding and reasoning. The paper acknowledges this implicitly by evaluating across model families and scales, but the task diversity limitation means that practitioners deploying FP4 for code generation, multilingual applications, or agentic workflows cannot extrapolate from these results.

No evaluation of generation quality beyond accuracy. All tasks are multiple-choice or exact-match. The paper does not measure perplexity on held-out text, ROUGE/BLEU scores on generation tasks, or human preference ratings. For deployment use cases where LLMs generate free-form text (chatbots, summarization, creative writing), accuracy on benchmark tasks is an imperfect proxy for generation quality. The weight-only quantization results (Appendix A) showing ~2% accuracy drops on standard benchmarks may understate the perceptual quality degradation in open-ended generation.

The INT4 baseline is not fully apples-to-apples. The INT4 comparison uses group size 32 with FP16 scales to match NVFP4's bit-width (4.5 bits/element). However, FP16 scales are substantially more precise than E4M3 scales (10 bits of mantissa vs. 3 bits), and they are not quantized. This makes the INT4 baseline stronger than a hardware-realistic INT4 microscaling format would be — it gets the memory savings of 4-bit elements but the precision of FP16 scales. The hypothetical NVINT4 format (Appendix D) fixes this by quantizing scales to E4M3, and its strong performance (97.12% recovery) suggests that real INT4 microscaling hardware could outperform FP4 — but this is speculative. The paper could have included an INT4 baseline with E4M3-quantized scales to make the comparison fully hardware-realistic for both formats.

The QAT experiments are limited in scope. QAT is applied only to Llama-3.1-8B and Qwen-3-8B (Table 10), not to the full range of model scales. For MXFP4, QAT provides substantial benefits (Table 5: 92.30% → 95.40% with Had32), suggesting it could partially rescue MXFP4 on small models where PTQ methods fail. Without QAT results for 1B and 3B models, we cannot assess whether training-aware methods could make MXFP4 viable at all scales. The QAT training setup is also minimally described: 92,995 Tülu 3 samples, balanced GJS divergence loss — there is no ablation over training data quantity, learning rate, or number of epochs. The paper notes QAT "only provides very limited benefits" for NVFP4 but does not investigate why.

No latency measurements for interactive inference. All kernel benchmarks report throughput (TFLOPs/s, tokens/sec) at batch sizes ranging from 16 to 8,192. For interactive single-user inference, the relevant metric is time-to-first-token and per-token latency at batch size 1, not throughput at batch 512. The paper mentions (Figure 12 discussion) that small-batch inference is memory-bound and recommends weight-only quantization, but provides no actual latency numbers. A latency comparison between FP8 weight-only and FP4 weight-only at batch size 1 would be directly actionable for practitioners building chatbots or coding assistants.

The Hadamard block size sweep is incomplete. The paper's theoretical framework predicts that matching the block size to the group size is optimal for NVFP4 (because the rotation should normalize within each quantization group). The Platinum benchmark results (Figure 13) confirm this for Had16 vs. Had128 on NVFP4. However, the paper does not systematically evaluate the interaction between block size and group size — for example, Had8 or Had4 on NVFP4 (G=16), or Had16 and Had64 on MXFP4 (G=32). The Appendix K sweep (Tables 13–14) includes sizes 16, 32, 64, 128, 256, but the per-size results are averaged across tasks into a single scalar (0.69–0.71), making it difficult to see whether specific block-size/group-size ratios are consistently optimal.

The claim that NVFP4's small group size "provably neutralizes traditional outlier mitigation" is an analytical claim, not an empirically demonstrated one. The paper proves (Lemma 1, Remark 2) that absmax scaling perfectly preserves the top element in each group. But "outlier mitigation" in the quantization literature encompasses more than preserving the top element — it also involves preventing outlier-induced quantization error from spreading to non-outlier elements in the same group (the core motivation for SmoothQuant's per-channel scaling). The paper does not measure or discuss this secondary effect. An ablation comparing per-channel vs. per-group scaling for NVFP4 and a large-group-size baseline would directly test this claim but is absent.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Renders the Theoretical Efficiency Gains Aspirational Rather Than Achieved

The assumption or constraint. The entire compute-optimal framework—and by extension the practical deployment of MR-GPTQ's format-conditional design choices—requires knowing which format to apply, with which algorithmic components, for a given model and deployment scenario. The paper's approach to determining this is an exhaustive empirical sweep: evaluating multiple quantization methods (RTN, GPTQ, SmoothQuant, QuaRot, SpinQuant), multiple format configurations (NVFP4, MXFP4, INT4, with and without rotations), multiple block sizes, multiple scale optimization strategies, and multiple reordering schemes across multiple models and benchmarks. The paper acknowledges in Section 5's introductory discussion that the standard benchmark suite is too noisy to reliably distinguish between top methods, and that reliable comparison requires the lower-variance PlatinumBench (Appendix I, Figures 13–17).

The consequence. A practitioner deploying a new model—not one of the Llama-3 or Qwen-3 variants evaluated in the paper—faces a combinatorial explosion of design choices with no principled method for navigating them. The paper provides diagnostics (compare scale precision to element precision; if scales are finer, rotations may hurt under RTN; if coarser, rotations help), but these diagnostics are qualitative, not quantitative. They tell you which direction to explore but not which specific configuration (Had16 vs. Had128, MSE vs. MinMax scales, ActOrder vs. DefOrder) is optimal for a given model. The cost of running the full ablation that the paper itself required—sweeping across formats, block sizes, scale methods, and reordering strategies—is enormous: quantizing a single large model 50+ times (8 configurations × 2 formats × multiple seeds, per Appendix I) and evaluating on multiple benchmarks. This is not a one-time cost amortized across deployments; it must be repeated for each new model architecture, each new quantization format specification, and potentially each new task distribution.

What evidence exists in the paper. The paper does not provide a lightweight difficulty estimation or configuration selection method. The error analysis in Section 3 provides explanations for observed behavior but not predictions that could replace empirical sweeps. For example, Lemma 2 predicts a crossover phenomenon (Laplace beats Normal at small G, Normal beats Laplace at large G), but the precise crossover point depends on the dead-zone halfwidth δ and is not computed numerically for the specific FP4 grid. Figure 3 shows that the crossover occurs somewhere between G=16 and G=64 for MSErel, but this range spans the two groups sizes of interest (16 and 32), making the prediction too coarse to unambiguously select a method without empirical validation. The analysis can explain post hoc why NVFP4 RTN degrades with rotations while MXFP4 improves, but it cannot predict the magnitude of the effect (0.6 points vs. 1.1 points in Table 1), which matters for deployment decisions.

Mitigation status. The paper does not address this limitation. Section 8 (Conclusion) does not mention it as a direction for future work. The paper's primary contribution is demonstrating that format-specialized methods work, not how to select them efficiently for new models. A practitioner reading this paper can adopt MR-GPTQ for Llama-3-8B with the specific configurations reported in Table 1, but they cannot confidently extrapolate to Llama-4, Mistral, Gemma, or custom fine-tuned models without re-running substantial portions of the experimental pipeline.

6.2 The Kernels Are Optimized for Throughput, Not Latency—Making the Method Unsuitable for Interactive Inference

The assumption or constraint. The QuTLASS kernel design (Section 4.2) and all performance benchmarks (Figures 5–6, 11–12) focus on throughput (TFLOPs/s, tokens/sec) at batch sizes ranging from 16 to 8,192. The kernel optimizations—fused online rotation, scale computation, and quantization into a single memory-bound kernel—are explicitly motivated by the observation that for block sizes k < 256, the rotation is memory-bound and therefore "essentially free." This logic holds when the matmul itself is large enough to amortize the per-layer rotation overhead, which is true at large batch sizes where the matmul dominates total runtime.

The consequence. For interactive single-user inference (batch size 1, or small batches of 1–4 for speculative decoding), the workload is memory-bandwidth-bound, not compute-bound. In this regime, activation quantization to 4 bits provides no latency benefit because the bottleneck is loading weights from GPU memory, not computing the matrix multiplication. The paper explicitly acknowledges this in the Appendix G discussion of Figure 12:

"For small batch workloads... [FP4 formats] show little to no improvement over BF16 and FP8. In that regime, which is characterized by memory-bound inference, it is preferable to use weight-only quantization, as activation quantization brings no benefits to inference speed."

This means that MR-GPTQ—which is a weight-and-activation (W4A4) quantization method with online activation rotations—provides zero speedup for the single most common LLM deployment scenario: a chatbot or coding assistant serving one user at a time. The paper's headline speedups (2.2× on B200, 4× on RTX5090) apply only to batched inference serving multiple concurrent users or processing large evaluation workloads.

Furthermore, even in the batched regime, the online rotation adds latency that is not separately measured. The paper compares "actual" throughput (including rotation) against "ideal" throughput (matmul only) and reports a ~5–9% gap (Figure 5). But this gap is measured at batch size 8,192, where the matmul is massive relative to the rotation. At batch size 16 or 32—a more realistic batch for many deployments—the relative overhead of the rotation would be larger because the matmul time scales with batch size while the rotation time is a fixed per-invocation cost. The paper does not report throughput at batch sizes below 16, leaving the latency penalty for low-batch inference unquantified.

What evidence exists in the paper. Figure 12 explicitly shows the batch-size dependence: the FP4 speedup over BF16 grows with batch size, reaching 2.2× only at batch sizes ≥256. At batch size 32 (the smallest shown for end-to-end measurements), the speedup is approximately 1.2–1.3×. The paper does not show time-to-first-token or per-token latency at batch size 1 for any FP4 configuration, only the note that weight-only FP4 provides ~20% lower latency than FP8 in the small-batch regime. The layer-wise benchmarks (Figures 5, 11 left) extend down to batch size 16, where the throughput advantage over BF16 is approximately 2× (not 3.6×) on B200, but these are single-layer measurements that exclude the overhead of attention, layer norm, and the full model pipeline.

Mitigation status. The paper partially addresses this by recommending weight-only quantization for small-batch inference (Section 4.2 discussion, Appendix G), and provides accuracy results for weight-only FP4 in Appendix A (Table 2). However, the weight-only results show a ~2% accuracy drop even for NVFP4 on Llama-3.1-8B (98.02% recovery), and the weight-only configuration does not use MR-GPTQ's Hadamard rotations (since activations are not quantized, there is no activation rotation to fuse). This means the paper's primary algorithmic contribution—micro-rotations to bridge the MXFP4-NVFP4 accuracy gap—is irrelevant to the deployment scenario where FP4 quantization is most latency-competitive (weight-only, small-batch). The paper does not develop or evaluate a latency-optimized kernel variant that applies rotations only to weights (pre-fused offline, as described in Section 4.1) with FP16 activations, which could potentially combine MR-GPTQ's accuracy benefits with weight-only deployment.

6.3 The MXFP4–NVFP4 Accuracy Gap Is Model-Scale-Dependent, and Small Models Remain Fundamentally Unserved

The assumption or constraint. The paper's central claim—that MR-GPTQ bridges the gap between MXFP4 and NVFP4, bringing MXFP4 "within 1-2% of NVFP4" (Section 5 Results discussion, point 1)—is based primarily on the Llama-3.1-8B-Instruct results in Table 1, where MXFP4 MR-GPTQ achieves 93.31% recovery vs. NVFP4 MR-GPTQ at 96.08% (a 2.77-point gap). The claim is implicitly generalized to "large models" in the abstract and conclusion.

The consequence. The format gap is not constant across model scales—it is much larger for small models and narrower for very large models. Table 10 in Appendix B provides the per-model recovery rates:

Model	NVFP4 MR-GPTQ	MXFP4 MR-GPTQ	Gap
Llama-3.2-1B	87.35%	79.80%	7.55 points
Llama-3.2-3B	93.68%	92.69%	0.99 points
Llama-3.1-8B	95.80%	93.31%	2.49 points
Llama-3.3-70B	98.32%	98.40%	−0.08 points (MXFP4 wins)
Qwen-3-8B	97.43%	95.23%	2.20 points
Qwen-3-14B	98.94%	97.28%	1.66 points
Qwen-3-32B	98.26%	95.58%	2.68 points

For Llama-3.2-1B, MR-GPTQ leaves MXFP4 trailing NVFP4 by 7.55 points—a gap that is larger than the improvement MR-GPTQ provides over RTN for that model (MXFP4 RTN: 67.70% → MXFP4 MR-GPTQ: 79.80%, a 12.1-point improvement, but still far from NVFP4 MR-GPTQ at 87.35%). For Llama-3.3-70B, MXFP4 MR-GPTQ slightly exceeds NVFP4 MR-GPTQ (98.40% vs. 98.32%), but this comparison is complicated by the fact that different Hadamard block sizes were used for the two formats (Had32 for MXFP4, Had16 for NVFP4)—it is not a controlled comparison of formats alone.

The implication is that MR-GPTQ does not "bridge the gap" in any uniform sense. For the smallest models (1B parameters), the gap is so large that MXFP4 is not a viable format regardless of algorithmic sophistication—a practitioner would be better served by NVFP4, INT4, or even FP8. For models 3B and above, the gap narrows to 1–3 points, which may be acceptable depending on the application. The paper's abstract and conclusion do not surface this scale-dependence, potentially misleading practitioners working with sub-3B models (increasingly common for on-device deployment) into expecting MR-GPTQ to make MXFP4 competitive.

What evidence exists in the paper. The per-model results are in Tables 3–10 and summarized in Table 10. The paper's main text (Section 5) focuses on the 8B model and mentions that "smaller models (< 8B) and Llama-family models tend to have lower recovery rates," but does not quantify the format gap separately for each scale. The 7.55-point gap for Llama-3.2-1B is visible in Table 10 but is not highlighted or discussed in the main text.

Mitigation status. The paper does not address this scale-dependence as a limitation. The proposed MXFP4† modification (Appendix H, Table 11) improves MXFP4 accuracy on Llama-3-8B substantially (MR-GPTQ: 93.31% → 94.87%) but is not evaluated on 1B or 3B models, leaving open the question of whether scale fitting could rescue MXFP4 at small scales. QAT provides larger benefits for MXFP4 than for NVFP4 on the 8B model (Table 5: MXFP4 QAT+Had32 reaches 95.40% vs. MR-GPTQ at 93.31%), but QAT is similarly evaluated only at 8B scale (Table 10 shows QAT results for 1B and 3B Llama models but only in NVFP4, not MXFP4). The missing experiment is MXFP4 QAT at 1B and 3B scales, which would reveal whether training-aware methods can close the gap where PTQ methods fail.

6.4 The Evaluation Is Limited to Multiple-Choice and Exact-Match Benchmarks—Generation Quality and Safety Are Unmeasured

The assumption or constraint. All accuracy evaluations in the paper use tasks with deterministic ground-truth answers: GSM8k (exact match on final numeric answer), MMLU (multiple choice), HellaSwag (multiple choice), WinoGrande (multiple choice), and the PlatinumBench tasks (various formats, all with verifiable correct answers). The paper does not evaluate perplexity on held-out text, open-ended generation quality, instruction-following ability, or safety-critical behaviors (harmful output rates, hallucination frequency, refusal boundary shifts). The paper also does not evaluate on code generation benchmarks (HumanEval, MBPP), which are standard in the quantization literature and particularly relevant because code generation is often run in batched evaluation settings where FP4 throughput benefits would apply.

The consequence. The accuracy recovery rates reported in the paper are lower bounds on the true quality degradation for deployment use cases involving free-form text generation. Quantization error that preserves a model's ability to select the correct multiple-choice answer may nonetheless degrade the fluency, coherence, factuality, or stylistic appropriateness of generated text in ways that human evaluators would notice. This is especially concerning for the smaller models where recovery rates drop to 80–87% (Llama-3.2-1B with NVFP4, Table 3)—a 13–20% relative accuracy loss on multiple-choice tasks likely corresponds to a qualitatively worse generation experience.

More specifically, the paper does not address how quantization interacts with safety alignment. LLM safety training (RLHF, DPO) operates on the model's output distribution; quantization perturbs this distribution, potentially shifting refusal boundaries or reactivating harmful behaviors that were suppressed during alignment. This is a known concern in the quantization literature (e.g., Li et al., 2024; Lee et al., 2024), and the paper's finding that "no format is lossless" (Section 5.1, point 1) implies that such shifts are occurring—but their nature and severity are unmeasured. A practitioner deploying an MR-GPTQ quantized chatbot cannot assess from this paper whether the quantized model is more or less likely to produce harmful content than the FP16 baseline.

What evidence exists in the paper. The paper's evaluation is limited to the tasks listed above. The results section is transparent about this scope: "We use a subset of tasks from the Open LLM Leaderboard V1" (Section 5 introductory text). The PlatinumBench extension adds lower-noise tasks but does not change the task type (all are verifiable-answer benchmarks). The paper does not claim to evaluate generation quality or safety, and does not flag this as a limitation.

Mitigation status. The paper does not attempt to address this gap. There is no perplexity evaluation on a held-out corpus (e.g., WikiText-2, C4), which is the standard minimal extension for assessing language modeling quality under quantization. There is no evaluation on instruction-following benchmarks (MT-Bench, AlpacaEval) or safety benchmarks (TruthfulQA, ToxiGen, RealToxicityPrompts). The absence of these evaluations is not acknowledged as a limitation.

6.5 The Algorithm Design Was Validated on Only Two Model Families—Transfer to Other Architectures Is Unverified

The assumption or constraint. All experiments use models from the Llama-3 family (Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct) and the Qwen-3 family (Qwen-3-8B, Qwen-3-14B, Qwen-3-32B). The paper states that this coverage "spans a representative range of modern LLM scales and architectures" (Section 5 introductory text), but both families share architectural features common to modern dense transformer LLMs: SwiGLU activations in feed-forward layers, grouped-query attention, and pre-normalization with RMSNorm.

The consequence. The paper's key design recommendations—block-wise Hadamard rotations with the block size matching the quantization group size, MSE scale optimization for NVFP4, static activation reordering—are empirically derived from these two model families and may not transfer to models with different architectural properties. Several concrete risks:

Mixture-of-Experts (MoE) models (Mixtral, DeepSeek-V2, Qwen-MoE) have different weight distribution characteristics: expert layers are sparsely activated, leading to different activation statistics (the calibration data distribution may not cover all experts equally) and potentially different optimal quantization strategies. The paper's error analysis (Section 3) models weights and activations as i.i.d. within blocks, which may not hold for MoE experts where different experts specialize to different input clusters.
Non-SwiGLU activation functions (GeLU in older models like GPT-3, BLOOM) produce different activation distributions. The paper's Laplace/Normal modeling (Figure 2) was validated on SwiGLU-based models; whether the same distributional assumptions hold for GeLU or other activations is untested.
Encoder-decoder architectures (T5, BART) and vision-language models (LLaVA, Qwen-VL) have additional components (cross-attention, visual encoders) that the paper does not evaluate. The Hadamard rotation mechanism assumes a standard linear layer structure; its interaction with cross-attention or visual feature projections is unknown.
Very deep or very wide models at the extremes of the scale range: the paper covers 1B to 70B, but scaling trends are not always monotonic and may reverse at larger scales (e.g., Llama-3-405B, not evaluated).

The paper's own results provide indirect evidence of architecture-dependence: Qwen-3 models consistently achieve higher recovery rates than comparably-sized Llama-3 models (Table 10: Qwen-3-8B NVFP4 RTN at 98.86% vs. Llama-3.1-8B at 94.80%). This 4-point difference for the exact same quantization method suggests that architectural factors (training data, tokenizer, layer count, weight initialization) significantly affect quantization robustness. If two fairly similar dense transformer families differ by 4 points, the variance across more architecturally diverse families could be larger.

What evidence exists in the paper. The paper's ablation over model scales (Tables 3–10) shows consistent trends within each family (recovery improves with scale), but also substantial inter-family variation. The difference between Llama and Qwen is noted in the Results discussion ("Qwen3 models can achieve more than 99% average recovery in NVFP4") but not analyzed—the paper does not investigate why Qwen models quantize better, which would be necessary to predict behavior on new architectures.

Mitigation status. The paper does not address transfer to other architectures as a limitation. The abstract and conclusion frame the results as general findings about FP4 microscaling formats ("we conclude that, while FP4 is not an automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock a new frontier"), without qualifying that the evidence comes from two model families. No experiments on MoE, encoder-decoder, or vision-language models are suggested for future work.

6.6 The Theoretical Analysis Predicts Qualitative Trends but Cannot Guide Quantitative Configuration Selection

The assumption or constraint. Section 3's error analysis models FP4 microscaling quantization under simplified assumptions: i.i.d. elements within blocks, Laplace or Normal distributions, absmax scaling with an unquantized scale, and asymptotic (G → ∞) or large-G approximations. The analysis derives asymptotic rates for MSE decay (Lemma 2: Θ((log G)² · G^(-δ)) for Laplace, Θ(√(log G) · G^(-δ²)) for Normal) and comparative predictions (Laplace MSE < Normal MSE for small G; Normal MSE < Laplace MSE for large G; rotations spread top-element error via Lemma 1; MXFP4's E8M0 scales are coarser than E2M1 elements, while NVFP4's E4M3 scales are finer).

The consequence. The analysis predicts qualitative trends—whether a rotation should help or hurt, whether increasing group size should improve or degrade accuracy—but provides no quantitative guidance for selecting specific hyperparameters. The crossover point where Normal MSE drops below Laplace MSE depends on the dead-zone halfwidth δ = 0.25 for the FP4 grid, but the exact G at which the crossover occurs is not computed (the Θ notation hides constant factors). The prediction that "rotations hurt NVFP4 RTN but may help under GPTQ" is borne out empirically (Table 1), but the analysis cannot predict how much they hurt (~0.6 points) or whether the GPTQ recovery will fully compensate (it does, within variance).

This gap is most consequential for the block size selection—a hyperparameter that the paper shows matters substantially for NVFP4 (Figure 13: Had16 outperforms Had128 by 2.6 points) but less for MXFP4. The theoretical framework predicts that the rotation block size should match the quantization group size (so that normalization occurs within each quantization block), but this prediction is qualitative ("match" vs. "don't match"). It does not predict the sensitivity to mismatch: would Had8 or Had32 (integer multiples of G=16) perform better or worse than Had16? The Appendix K sweep (Tables 13–14) includes sizes 16, 32, 64, 128, 256, but the per-size results are averaged across tasks into a single accuracy scalar, making it impossible to test whether block sizes that are integer divisors or multiples of the group size have systematic advantages.

For scale optimization, the analysis provides even less guidance. The paper's alternating optimization for NVFP4 scales (Section 4.1, Ingredient 1) is motivated by "we have found this to yield consistent improvements," not by any theoretical prediction about when joint scale optimization helps vs. when absmax scaling suffices. The PlatinumBench results (Figures 13c, 13d) show that MSE optimization provides a measurable but inconsistent benefit: for NVFP4 with Had16, MSE beats MinMax by 2.8 points (96.57% vs. 93.77%); for MXFP4 with Had128, MinMax beats MSE by 0.5 points (92.32% vs. 91.86%), though this is within noise. The theory does not explain this format-dependence.

What evidence exists in the paper. The analytical predictions (Section 3.2) are validated qualitatively by Figure 3: NVFP4 MSErel increases with HT at G=16, decreases at larger G; MXFP4 MSErel decreases with HT across all G. These are qualitative matches. The numerical values in Figure 3 (e.g., NVFP4 MSErel_top ~0.001 for native weights, rising to ~0.005 with HT at G=16) are empirical measurements, not theoretical predictions. Nowhere does the paper compute a predicted MSErel value from its model and compare it to the measured value; the comparison is always directional.

Mitigation status. The paper does not frame the gap between qualitative prediction and quantitative guidance as a limitation. The analysis is presented as successful because it correctly predicts the direction of effects (Section 3.2 Discussion: "our analysis so far showed that the MXFP4 format induces higher MSE for RTN quantization relative to NVFP4, and is worse at outlier preservation"). The inability to predict magnitudes—which would be necessary to make the framework actionable for new formats without re-running full empirical sweeps—is not acknowledged. The obvious next step, a numerical integration of the MSE expressions for the specific FP4 grid (E2M1 with δ=0.25) to produce predicted curves that could be compared against Figure 3, is not taken.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around low-precision LLM quantization in a specific, falsifiable way: it demonstrates that format choice is not a secondary implementation detail but a first-class design variable that determines which algorithms work, and how well. The dominant narrative in the quantization community—driven by hardware vendors shipping FP4 tensor cores and by the intuitive appeal of floating-point's non-uniform grid for heavy-tailed distributions—has been that FP4 is a straightforward accuracy improvement over INT4 at the same bit-width. This paper is the first to systematically test that narrative against modern LLMs and state-of-the-art quantization methods, and the results are sobering: FP4 is not uniformly better than INT4, MXFP4 is substantially worse than NVFP4 despite only 0.25 fewer bits per element, and methods developed for INT quantization (QuaRot, SpinQuant) can actively harm FP4 accuracy when applied without format-specific adaptation.

This is a reframing, not a paradigm shift. It does not introduce a new fundamental principle like Chinchilla scaling laws did for pretraining. But it reframes what the community should be optimizing: instead of treating quantization methods as interchangeable backends and formats as fixed hardware constraints, the paper argues that method and format must be co-designed—the choice of Hadamard rotation block size, scale optimization strategy, and activation reordering depends on whether the format's scale precision exceeds its element precision (NVFP4) or not (MXFP4). This is a diagnostic insight that changes how practitioners should approach new formats: before running expensive accuracy sweeps, first measure whether the format's scale quantization granularity is finer or coarser than its element quantization granularity. If finer (E4M3 scales over E2M1 elements, as in NVFP4), rotations will hurt under RTN but may help under error-compensating methods. If coarser (E8M0 scales over E2M1 elements, as in MXFP4), rotations are unambiguously beneficial and scale optimization is critical.

The paper also resolves an emerging tension in the literature that practitioners may not have recognized as a tension at all. Prior work on rotation-based quantization (QuIP, QuaRot, SpinQuant) established that Hadamard transforms normalize weight and activation distributions, making them easier to quantize. This was proven and validated for INT formats with large group sizes or per-channel scaling. The straightforward extrapolation—"rotations help quantization, so they should help FP4 too"—is what this paper shows is false for NVFP4 under RTN. The mechanism is specific and previously undocumented: microscaling's absmax scaling perfectly preserves the top element in each group, effectively promoting it to the scale's precision. Rotations destroy this property by spreading top-element error across the group (Lemma 1), and whether this trade-off is net-beneficial depends on whether the scale precision exceeds the element precision. This resolves the apparent contradiction: rotations are not universally good or bad for FP4; they are conditionally beneficial based on a measurable property of the format (scale precision vs. element precision). This is a testable, diagnostic criterion that did not exist before.

Less visibly but perhaps more consequentially, the paper shifts research attention from algorithmic sophistication to verifier/formatter robustness. In the quantization context, the "verifier" is the MSE- or output-reconstruction-based loss that GPTQ optimizes; the "formatter" is the quantization grid and scale representation. The paper's findings—that lookahead-style optimization (more sophisticated scale search, larger rotation blocks) can be counterproductive (Had128 underperforms Had16 for NVFP4 in Figure 13; MSE optimization is not uniformly better than MinMax for MXFP4 in Figure 13d); that the primary bottleneck for MXFP4 is not algorithmic but format-inherent (E8M0 scale coarseness); and that the hypothetical NVINT4 format (INT4 base elements with E4M3 scales) outperforms both real FP4 formats (Table 1, 97.12% recovery)—collectively argue that improving the format specification itself is more impactful than improving the quantization algorithm that targets it. This is analogous to how the RLHF community recognized reward model robustness as more important than policy optimization sophistication. The paper's analysis of integer microscaling formats (Appendix D, Figure 10) provides a concrete template for what a better format specification would look like: INT4 base elements with E4M3 scales (NVINT4) combines the best properties of both existing formats—uniform grid for even error distribution, fine-grained scales for outlier protection—and the paper's own error analysis predicts and validates its superiority.

The research directions that become more attractive after this work:

Format-algorithm co-design for emerging precision standards. The paper demonstrates that format specifications (group size, scale precision, element precision) interact non-trivially with quantization algorithms. As new formats emerge (FP6, MXFP8 variants, Microsoft's FP4 proposals), this framework provides a principled way to evaluate them before hardware is available—simulate the format, run the error analysis, predict which algorithm components will be needed.
Efficient difficulty estimation for quantization configuration. The paper's results show that the optimal configuration (Hadamard block size, scale method, act-order) is not universal across models or even within a model family. Developing lightweight methods to predict the best configuration for a new model without exhaustive sweeps becomes important.
Verifier/formatter improvements as the primary lever. The finding that format specification limits accuracy more than algorithmic sophistication suggests that resources spent on better scale representations (more mantissa bits, data-aware range fitting like MXFP4†) will yield larger gains than resources spent on more complex quantization algorithms.

The research directions that become less attractive:

Applying existing rotation-based methods to FP4 without modification. The paper shows that QuaRot and SpinQuant underperform simple RTN on NVFP4, and QuaRot catastrophically fails on MXFP4 (62.90% recovery). The straightforward application of these methods to FP4 is dead-on-arrival; any future use must incorporate format-specific adaptations of the kind MR-GPTQ provides.
Sole focus on layer-wise or per-element MSE as a universal proxy. The paper reveals that the same MSE metric can be reduced by a change that hurts end-to-end accuracy (NVFP4 RTN: rotations reduce MSErel at larger G but increase it at G=16, where the format operates). The relationship between MSE and accuracy is mediated by format-specific error structure (top-element preservation vs. average-element precision), not just by aggregate distortion.

Follow-Up Research This Work Enables

1. Lightweight difficulty estimation for quantization configuration selection. The most immediate barrier to deploying MR-GPTQ on a new model is the combinatorial cost of sweeps: 8+ configurations (Hadamard block size × scale method × act-order × format) × multiple seeds × multiple benchmarks. The paper's error analysis provides qualitative diagnostics (compare scale precision to element precision) but cannot predict which specific block size is optimal. A concrete follow-up: train a lightweight classifier on the calibration data alone—using features such as the kurtosis of weight/activation distributions before and after rotation at different block sizes, the ratio of MSErel_top to MSErel for each candidate configuration, and the Hessian spectrum—to predict which configuration will maximize PlatinumBench accuracy. The training data would be the paper's existing sweep results across Llama-3 and Qwen-3 models (Tables 3–10, Figures 13–16). The test would be held-out model families (Mistral, Gemma). Success means reducing the deployment cost from 50+ quantization runs to a single forward pass over calibration data.

2. MXFP4† with QAT at small model scales. The paper shows that MXFP4's accuracy gap is worst for small models (Llama-3.2-1B: 79.80% recovery with MR-GPTQ vs. 87.35% for NVFP4), and that QAT helps MXFP4 on 8B models (Table 5: QAT+Had32 reaches 95.40% vs. 93.31% for MR-GPTQ). The missing experiment: apply QAT with the MXFP4† scale-fitting modification to Llama-3.2-1B and Llama-3.2-3B, measuring whether the combination of better scale representation (MXFP4†) and training-aware error compensation (QAT) can close the 7.5-point gap to NVFP4. A negative result—that even QAT+MXFP4† cannot make MXFP4 viable at 1B scale—would establish a fundamental lower bound on model size for MXFP4 deployment, which is actionable for on-device deployment planning. A positive result would validate MXFP4 as a universal format across scales.

3. Latency-optimized MR-GPTQ for weight-only small-batch inference. The paper's QuTLASS kernels are optimized for throughput (large-batch matmul) and provide negligible speedup at batch sizes below 32. The paper recommends weight-only quantization for small-batch inference but does not evaluate MR-GPTQ in this setting. A direct follow-up: implement a kernel that applies the Hadamard rotation to weights only (pre-fused offline, as described in Section 4.1) and leaves activations in FP16, then benchmark time-to-first-token and per-token latency at batch sizes 1–8 for Llama-3.1-8B and Llama-3.3-70B. Compare against FP8 weight-only (the current production baseline) and against FP4 weight-only without rotations. The accuracy results in Appendix A (Table 2) already exist; the missing piece is latency. If weight-only MR-GPTQ achieves lower latency than FP8 weight-only while matching or exceeding its accuracy, it would immediately become the preferred format for interactive inference.

4. Extension to non-SwiGLU architectures and MoE models. The paper validates on Llama-3 and Qwen-3, both using SwiGLU activations and dense transformer architectures. The error analysis (Section 3) assumes i.i.d. elements within blocks following Laplace or Normal distributions, which may not hold for: (a) MoE models where different experts see different input distributions, making per-expert weight statistics non-identical; (b) GeLU-based models where activation distributions have different tail behavior; (c) encoder-decoder architectures with cross-attention. A targeted follow-up: apply MR-GPTQ to Mixtral-8x7B (MoE), Mistral-7B (dense, but with different training recipe), and T5-XXL (encoder-decoder), measuring whether the format-dependence patterns (NVFP4 without rotations works well; MXFP4 needs rotations) replicate. The specific question: does the diagnostic "compare scale precision to element precision" correctly predict rotation effectiveness across architectures, or are there architecture-specific distributional properties that override the format-level prediction?

5. INT4 microscaling as a standard proposal with hardware-aware evaluation. The paper's finding that hypothetical NVINT4 (INT4 base elements, E4M3 scales, G=16) outperforms both NVFP4 and MXFP4 on PTQ accuracy (Table 1: 97.12% vs. 96.08% for Llama-3.1-8B) is an existence proof that the OCP standard's choice of only floating-point base types is suboptimal. A concrete next step: develop a reference implementation of NVINT4 matrix multiplication (either via emulation on existing tensor cores or via FPGA prototyping), measure the hardware cost (area, power, cycle time) relative to NVFP4, and quantify the accuracy-vs-hardware-cost Pareto frontier. If INT4 microscaling can be implemented with similar or lower hardware complexity than FP4 microscaling (plausible because uniform quantization is simpler than non-uniform), combined with the accuracy advantage demonstrated here, this would make a compelling case for including integer base types in the next revision of the OCP microscaling specification. The paper's QuTLASS kernel infrastructure provides a starting point for the software emulation side.

6. Distribution-aware scale format design using the error analysis framework. The paper's analysis of scale quantization formats (Appendix C, Figures 8–9) shows that the standard E4M3 and E8M0 formats are not optimal for the distribution of LLM weight and activation scales—formats with more mantissa bits (E1M6–E3M4) reduce MSErel by 10–40% relative to the current standards. This analysis was done at group size 16; a systematic extension would sweep group sizes (8, 16, 32, 64) and scale formats (all combinations of exponent and mantissa bits summing to 8) across the full range of model scales (1B–70B) and architectures, producing a scale format recommendation as a function of group size and model properties. The output would be a lookup table: for group size G, model scale S, and architecture family A, the scale format (EeMm) that minimizes accuracy degradation while respecting hardware constraints (total bits per element). This is directly actionable for hardware designers and standards bodies.

Practical Applications and Downstream Use Cases

1. High-throughput batch inference for evaluation and data generation. Organizations running large-scale LLM evaluation (e.g., scoring thousands of student essays, generating synthetic training data, or evaluating model checkpoints on benchmark suites) operate in the batched regime where QuTLASS's throughput benefits apply directly. For Llama-3.3-70B on B200 GPUs, MR-MXFP4 delivers ~15,000 tokens/sec vs. ~7,000 for BF16—a 2.2× throughput improvement (Figure 12)—while recovering 98.4% of FP16 accuracy (Table 10). For a workload processing 1 million tokens, this reduces GPU-hours from ~143 to ~67 on B200. On consumer hardware (RTX 5090), the speedup reaches 3.8× end-to-end for Qwen3-8B (Figure 11), making it feasible to run batch evaluation on a single workstation that previously required multiple GPUs or cloud instances. The key caveat is that these speedups require batch sizes ≥64; for smaller evaluation runs, the benefit shrinks.

2. On-device deployment of 3–8B models with format-specific configuration. For applications deploying 3–8B parameter LLMs on edge devices with FP4-capable GPUs (projected in future AMD and NVIDIA consumer hardware), MR-GPTQ provides a concrete recipe: use NVFP4 without rotations for the simplest deployment (94–99% recovery across Llama and Qwen families, Tables 4–5, 7), or MXFP4 with MR-GPTQ if hardware only supports the OCP standard (92–95% recovery). The paper's per-model recovery rates (Table 10) provide the exact accuracy expectation: a Llama-3.2-3B deployed with NVFP4 GPTQ will achieve ~95.5% of FP16 accuracy; a Qwen-3-8B with NVFP4 RTN will achieve ~98.9%. These are specific enough for product decisions. For models smaller than 3B, the paper's results caution that FP4 quantization (even with MR-GPTQ) incurs substantial accuracy loss—practitioners should benchmark carefully or use higher precision for sub-3B models.

3. Cost-efficient serving with format routing based on query difficulty. The paper's finding that format accuracy recovery varies with model scale—and that different models have different format affinities (Qwen quantizes better than Llama)—enables a deployment architecture where queries are routed to differently-quantized model replicas based on estimated difficulty or domain. For example, a coding assistant service could serve routine queries (syntax questions, boilerplate generation) using a Qwen-3-14B model quantized to MXFP4 (97.3% recovery, highest throughput), moderate queries using NVFP4 (98.9% recovery, slightly lower throughput), and complex queries (multi-file refactoring, algorithmic reasoning) using FP8 or BF16. The QuTLASS kernels' plug-and-play backend support (CUTLASS vs. FlashInfer) makes switching between precision levels at runtime feasible. The challenge—unsolved by this paper—is estimating query difficulty cheaply enough to route before inference; the quantization difficulty estimation problem is parallel to the one identified in the compute-optimal test-time scaling literature.

4. Self-improvement data generation pipelines with MXFP4-accelerated models. LLM self-improvement loops (e.g., STaR, ReST, or rejection sampling fine-tuning) require generating large volumes of candidate solutions from a model, scoring them, and fine-tuning on the successful ones. The generation phase operates at large batch sizes and benefits directly from FP4 throughput. If the base model is quantized to MXFP4 with MR-GPTQ and achieves 93–98% recovery (depending on size), the generated data will be slightly lower quality than FP16 generations—but the 2–4× throughput improvement means 2–4× more candidates can be generated for the same compute budget. Whether the increased quantity compensates for the slight quality degradation is an empirical question the paper does not answer, but the throughput numbers (Figures 5–6) make this a compelling ablation for any team running self-improvement pipelines at scale. The paper's finding that Qwen models achieve >99% NVFP4 recovery on 32B+ scales (Table 10) is particularly relevant: at these scales, the accuracy loss from quantization is negligible, making FP4 acceleration essentially free for data generation.

When to Prefer This Method

The paper does not position MR-GPTQ against a single named alternative with a clearly articulated tradeoff (its contribution is demonstrating that format-specialized methods outperform format-agnostic ones, not that MR-GPTQ dominates a specific competing algorithm). However, the experimental results do imply a decision procedure for practitioners choosing among the evaluated methods:

Prefer MR-GPTQ (Hadamard rotations + MSE scales + static act-order) when:

Deploying on MXFP4 hardware (OCP standard, AMD CDNA4, future Intel GPUs) and accuracy is paramount—MR-GPTQ provides 3–5 points higher recovery than unrotated GPTQ (Table 1, Llama-3.1-8B: 93.31% vs. 89.47%), and the alternative of simply accepting MXFP4's lower accuracy may be unacceptable for the application.
Deploying on NVFP4 hardware and using GPTQ-based quantization (rather than RTN)—MR-GPTQ's rotated NVFP4 variant is within variance of standard GPTQ (96.08% vs. 95.92% on Llama-3.1-8B, Table 1), and the PlatinumBench results (Figure 13a) suggest a small but consistent advantage for Had16 variants with MSE scales and act-order (96.57% vs. 93.27–94.20% for unrotated variants).
Deploying NVINT4 or other hypothetical integer microscaling formats—the paper shows that rotations are unambiguously beneficial for NVINT4 (Table 1: 97.12% vs. 95.42% for unrotated GPTQ), and MR-GPTQ provides the strongest results.

Prefer RTN without rotations when:

Deploying on NVFP4 hardware with the simplest possible inference pipeline (no GPTQ calibration, no online rotation)—RTN achieves 94.80% recovery on Llama-3.1-8B (Table 1), which is within 1 point of the best GPTQ-based methods, and requires no calibration data, no Hessian computation, and no error compensation.
The inference batch size is small (≤4) and latency is critical—RTN weight-only NVFP4 quantization (Appendix A, Table 2) achieves 98.02% recovery with no activation quantization overhead, and the paper's Figure 12 shows that weight-only FP4 provides ~20% lower latency than FP8 in the small-batch regime.

Prefer SmoothQuant when:

Activation quantization is the primary bottleneck (weights are already compressed or will be quantized separately), and format portability is important—SmoothQuant achieves 95.90% recovery on NVFP4 and 89.06% on MXFP4 for Llama-3.1-8B (Table 1), outperforming GPTQ-based activation quantization (which the paper does not implement—GPTQ only quantizes weights) while being simpler to implement than full MR-GPTQ.

Prefer QAT with Hadamard rotations when:

Training budget exists and the target format is MXFP4—QAT+Had32 achieves 95.40% recovery on Llama-3.1-8B MXFP4 (Table 5), substantially above MR-GPTQ's 93.31%, and QAT+Had16 achieves 96.50% on NVFP4, matching or exceeding all PTQ methods.
The model is small (1–3B parameters) where PTQ methods show the largest accuracy drops—though the paper does not provide QAT results at these scales for MXFP4, the trend from 8B results suggests QAT could be the only viable path to acceptable MXFP4 accuracy for sub-3B models.

These preferences are clearly supported by the paper's Tables (1, 2, 5, 10) and Figures (13–14). The paper does not articulate them as a decision procedure, but the experimental design—sweeping across formats, methods, and model scales—implicitly provides the necessary comparisons.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training2025· inference-optimization← LLM-judged similar
- Microscaling Data Formats for Deep Learning2023· low-precision← LLM-judged similar
- Sleep-time Compute: Beyond Inference Scaling at Test-time2025· inference-optimization← in-domain predecessor
- EfficientLLM: Efficiency in Large Language Models2025· evaluation← topic thread
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration2024· inference-optimization← similar topic
Set Block Decoding is a Language Model Inference Accelerator2025· inference-optimization← in-domain predecessor
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads2024· inference-optimization← LLM-judged similar
- VibeVoice Technical Report2025· inference-optimization← in-domain predecessor
The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm2025· llm-systems← topic thread
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale2022· inference-optimization← LLM-judged similar
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities2025· llm-systems← in-domain predecessor
- Quartet: Native FP4 Training Can Be Optimal for Large Language Models2025· training-methods← topic thread
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution2024· llm-systems← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

The Pitfalls of KV Cache Compression2025· inference-optimization→ in-domain predecessor
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction2026· inference-optimization→ LLM-judged similar
- Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models2025· inference-optimization→ in-domain predecessor
- Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost2025· llm-systems→ topic thread
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression2024· inference-optimization→ similar topic
- Inference-Time Hyper-Scaling with KV Cache Compression2025· inference-optimization→ similar topic
Pretraining Large Language Models with NVFP42025· pretraining→ topic thread
- Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation2026· training-methods→ LLM-judged similar
- RLP: Reinforcement as a Pretraining Objective2025· pretraining→ in-domain predecessor
- INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats2025· inference-optimization→ topic thread
SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training2025· inference-optimization→ similar topic
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation2025· inference-optimization→ in-domain predecessor
- An Empirical Study of Qwen3 Quantization2025· low-precision→ topic thread
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference2023· inference-optimization→ similar topic
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration2024· inference-optimization→ similar topic
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps2026· inference-optimization→ similar topic