Training Methods2025~124 minarXiv:2505.14669

Quartet: Native FP4 Training Can Be Optimal for Large Language Models

score 9Roberto L. Castro, Andrei Panferov, Soroush Tabesh +5NVIDIA↑ 78★ 120May 20, 2025cs.LGhttps://github.com/IST-DASLab/Quartet

🎯 Pitch

Fully-native FP4 training can be the optimal choice for large language models—not just a compromise—when its efficiency gains fully offset its accuracy losses, as quantified by new low-precision scaling laws. On a Blackwell GPU, Quartet achieves up to ~2× forward-pass speedup over FP8 while reducing validation loss by 10% over prior 4-bit methods at scale, demonstrating that the hardware’s doubled throughput makes 4-bit the “accuracy-efficiency” sweet spot for models like Llama3.

1. Executive Summary

This paper introduces Quartet, a new algorithm for fully-native FP4 training of large language models that executes all three matrix multiplications of a linear layer in MXFP4 precision—a format supported in hardware by NVIDIA's Blackwell architecture. Through pre-training Llama-family models on the C4 dataset, the authors develop a new low-precision scaling law that decomposes the impact of quantized training into two fitted efficiency factors: parameter efficiency (linked to forward-pass compression error, maximized via QuEST's RMSE-based clipping) and data efficiency (linked to gradient estimator bias, measured by a novel projection-magnitude misalignment metric and maximized via stochastic rounding on the backward pass). Quartet achieves up to ~1.6× overall training speedup and ~2× forward-pass speedup over highly-optimized FP8 kernels on an NVIDIA RTX 5090, while reaching the lowest validation loss among prior 4-bit training methods—for example, improving upon LUQ-INT4 by a 10% relative loss reduction at a 100× tokens-per-parameter ratio. The paper's scaling-law framework establishes that FP4 can be the "optimal" precision on the accuracy-vs-speed Pareto frontier, but only in specific regimes where the reduced parameter and data efficiencies are fully offset by the hardware throughput gains—a condition satisfied by models at the scale of larger Llama3 or Qwen2.5 under realistic compute budgets.

2. Context and Motivation

The Core Problem: We Cannot Train LLMs Natively in Hardware-Supported FP4 Without Significant Accuracy Loss

The fundamental question this paper tackles is: can we train large language models entirely in hardware-native 4-bit floating-point precision without sacrificing accuracy relative to 8-bit or 16-bit training? The answer, prior to this work, was essentially "no" — existing methods either lose precision, become unstable, or must fall back to higher-precision computation for selected matrix multiplications, meaning they do not achieve true end-to-end 4-bit training.

This gap matters because of a specific hardware development: NVIDIA's Blackwell architecture introduces native hardware support for 4-bit floating-point formats — specifically MXFP4 and NVFP4 — that can nearly double arithmetic throughput and roughly halve energy consumption compared to 8-bit operations. The B200 GPU, for instance, peaks at 18 PFLOPS of dense FP4 compute. However, this hardware capability is useless without algorithms that can actually train models accurately in these formats. The paper states this directly in the introduction:

"today's algorithmic support for accurate end-to-end training in such low precision is missing. State-of-the-art quantized training methods such as Switchback, Jetfire, HALO, and INT4-Transformers either (i) lose precision and stability when training current models in 4-bit formats, or (ii) fall back to higher precision for selected matrix multiplications."

The significance is both practical and conceptual. Practically, training frontier models has become astronomically expensive — the FLOPs required to train a frontier model double every few months, as the paper notes in Section 1. Lower-precision computation offers near-linear gains in throughput and energy efficiency because the matrix multiplications that dominate training workloads run faster at lower bit-widths. Conceptually, the paper asks whether there exists a regime where the accuracy penalty from 4-bit quantization is fully compensated by the higher throughput — meaning that, under a fixed compute budget, FP4 training could actually produce a better model than FP8 training because you can train on more data or with more parameters in the same wall-clock time. This question had no answer before this work.

Why This Problem Matters Now: The Inference-Training Precision Gap

The paper situates itself at a peculiar inflection point in the field. On the inference side, 4-bit quantization — and even lower — is well-established as a way to preserve accuracy while dramatically reducing model size and latency. Techniques like GPTQ, QuaRot, and QuIP have shown that sophisticated calibration and rotation schemes make INT4 or FP4 inference near-lossless relative to FP16. On the training side, the precision frontier has been pushed more slowly: the field moved from FP16 to FP8, with methods like FP8 training in DeepSeek-V3 being responsible for significant efficiency breakthroughs. But going below 8 bits for training has proven extremely challenging.

Why is training harder than inference? The paper points to a fundamental asymmetry: inference only requires the forward pass (approximately 33% of training compute), while training requires both the forward pass and the backward pass (the remaining ~66%). The backward pass involves propagating gradients through the network, and these gradients must be computed with sufficient fidelity that the optimizer can make meaningful updates. Quantization errors in the backward pass accumulate across layers and can bias the gradient estimates in ways that derail convergence over long training runs. The paper's key insight — developed formally in Section 4.3 — is that forward-pass and backward-pass quantization impose different requirements: forward-pass quality is primarily about minimizing reconstruction error (MSE), while backward-pass quality is primarily about maintaining unbiased gradient estimates. Most prior 4-bit training methods did not systematically disentangle these two requirements, applying the same quantization strategy uniformly to both passes, which explains their poor performance.

Prior Approaches and Where They Fall Short

The paper provides a detailed taxonomy of existing work in Section 2, which we can analyze through the lens of what they quantize (forward only, backward only, or both) and how they handle the distinct challenges of each pass.

Forward-Only Quantization (Quantization-Aware Training, or QAT)

A substantial body of work focuses on quantizing only the forward pass, leaving gradients and weight updates in higher precision. These methods are motivated primarily by inference cost reduction: the model learns to be robust to the quantization it will encounter at deployment time. The paper identifies two sub-families:

Noise-injection approaches (e.g., Baskin et al., 2021): These add quantization noise during training to make the network robust to the discretization it will face at inference. The intuition is that by exposing the model to quantization-like perturbations during training, it learns parameter configurations that are resilient to the specific error profile of the target format.

Error-minimization approaches (e.g., LSQ by Esser et al., 2019; QuEST by Panferov et al., 2025): These focus on finding the best possible quantized representation of the continuous weights and activations at each forward pass, typically by learning the quantization scale or clipping parameters to minimize mean squared error between the full-precision and quantized tensors. QuEST specifically uses RMSE-based clipping combined with optional Hadamard rotations to handle outliers, achieving state-of-the-art parameter efficiency among forward-only methods.

The limitation of forward-only approaches is that they only address half the problem. They leave the backward pass — which dominates training compute — in higher precision, so the training throughput gains are modest (at best ~1.5× from forward-pass speedup alone, since the backward pass is 2× more compute-intensive than the forward pass). True end-to-end efficiency requires quantizing the backward pass as well.

Wang et al. (2024) explored FP4 QAT specifically, introducing a "smoother" gradient estimator, outlier clamping, and compensation mechanisms. While their approach shows good accuracy, the paper notes it is "fairly complex and not validated in terms of efficient support" — meaning the algorithm exists on paper but hasn't been implemented with optimized GPU kernels that could demonstrate real speedups.

Backward-Only Quantization

A more recent line of work investigates quantizing only the backward pass, keeping the forward pass in higher precision. Tseng et al. (2025) specifically investigated backward-only quantization with the MXFP4 format, demonstrating the importance of stochastic rounding and outlier mitigation in low-precision backpropagation. Their work provides one of the key ingredients — stochastic rounding on the backward pass — that Quartet adopts, but by itself it only addresses half the problem.

Full End-to-End 4-Bit Training (Both Forward and Backward)

This is where Quartet's primary competitors lie. The paper identifies four prior methods that attempt full 4-bit training for both forward and backward passes:

LUQ (Chmiel et al., 2023): Logarithmic Unbiased Quantization combines a log-scale FP4-type format (to cover wider dynamic range) with stochastic unbiased rounding on the backward pass. On ResNet50/ImageNet, LUQ incurs a 1.1% top-1 accuracy drop. Critically, LUQ has not been validated on hardware-supported FP formats — the paper notes it is "far from being supported in hardware" — meaning its theoretical advantages cannot be realized as actual speedups on existing GPUs. It also was developed for CNNs, not Transformers, and its behavior on LLM pre-training at scale was unknown prior to this work.

LSS / INT4-Transformers (Xi et al., 2023): This method trains Transformers using INT4-effective precision by combining block-wise Hadamard transforms and learned step-size quantization (LSQ) for outlier mitigation on the forward pass, with leverage score sampling on the backward pass to exploit structured sparsity. On BERT-family models, it achieves 1-2% accuracy gap relative to FP16, with 2.2× speedup on individual matrix multiplies (relative to 4× theoretical) and up to 35% faster end-to-end training. The paper's experiments reveal that LSS "is competitive only for shorter runs, and diverges for longer training budgets, beyond 50×" tokens-per-parameter ratio. This is a critical failure mode: LSS works for fine-tuning or short training runs but breaks down under the data-saturated conditions typical of modern LLM pre-training.

Jetfire (Xi et al., 2024): Originally developed for INT8 training, Jetfire uses per-block quantization to handle activation and gradient outliers by partitioning matrices into 32×32 blocks with independent scaling. It achieves ~40% end-to-end speedup and 1.49× memory reduction in INT8. The paper adapts Jetfire to FP4 for comparison, but notes it "incurs large degradation and is unstable when ported to FP4." The dynamic range of FP4 is simply too narrow for Jetfire's block-wise approach without additional safeguards.

HALO (Ashkboos et al., 2025): This method uses Hadamard rotations to mitigate outliers, evaluated at their most accurate HALO-2 setting. Like Jetfire, it was designed for INT8 and degrades severely when pushed to FP4. The paper's experiments show HALO incurs "large degradation" alongside Jetfire.

The Mixed-Precision Fallback Problem

A common pattern across prior work is the mixed-precision fallback: methods that claim "4-bit training" often execute some matrix multiplications in higher precision (FP8, FP16, or even FP32) to maintain stability. This is not a minor implementation detail — it fundamentally limits the achievable speedup because the higher-precision operations become the bottleneck, and the hardware's FP4 throughput advantage cannot be fully realized if the critical path still contains FP8 or FP16 computations. The paper explicitly calls this out as a gap:

"State-of-the-art quantized training methods such as Switchback, Jetfire, HALO, and INT4-Transformers either (i) lose precision and stability when training current models in 4-bit formats, or (ii) fall back to higher precision for selected matrix multiplications."

Quartet's key differentiator is that it executes all three matrix multiplications of a linear layer — forward (input × weights), backward for activations (output gradient × weights^T), and backward for weights (output gradient^T × input) — entirely in MXFP4. There are no mixed-precision fallbacks in the linear layer computation, making it a true end-to-end FP4 training algorithm.

How This Paper Positions Itself

The paper frames itself not as proposing a single new quantization trick, but rather as providing a systematic framework for understanding and designing quantized training algorithms, with Quartet as the concrete instantiation that achieves state-of-the-art results. This framework has four components, which the paper calls "ingredients":

Ingredient 1: A common evaluation framework via scaling laws. Rather than comparing methods at a single model size or training budget (which can be misleading — a method that looks good at 30M parameters may collapse at 200M), the paper proposes fitting a parametric scaling law that explicitly models how forward and backward precision affect the effective parameter count and effective data size. Two methods can then be compared by examining their fitted efficiency coefficients. This is a meta-methodological contribution: it provides a rigorous way to determine whether one quantized training approach is genuinely better than another, rather than relying on point comparisons that may not generalize across scales.

Ingredient 2: A mixed-precision optimality analysis. The paper argues that previous work set an unnecessarily high bar by requiring lower precision to suffer no accuracy loss. Instead, the right question is: under a fixed compute budget, does the higher throughput of lower precision compensate for any accuracy degradation, resulting in a net better model? The paper formalizes this by incorporating real hardware speedup measurements into the scaling law, computing the "optimality region" where FP4 beats FP8 (the red vs. orange regions in Figure 1). This analysis reveals that FP4 can indeed be optimal for models at the scale of larger Llama3 or Qwen2.5.

Ingredient 3: Explicit separation of forward and backward requirements. This is the paper's central algorithmic insight. Forward-pass quality is linked to parameter efficiency and is maximized by minimizing MSE (via QuEST's RMSE-based clipping with Hadamard rotation). Backward-pass quality is linked to data efficiency and is maximized by eliminating gradient bias (via stochastic rounding). The paper formalizes this trade-off through a novel projection magnitude misalignment metric, derived from prior theoretical work on distributed mean estimation. Table 2 demonstrates that QuEST achieves the best forward-pass MSE but introduces misalignment on the backward pass, while stochastic rounding achieves perfect alignment (zero misalignment) but has higher forward-pass MSE. Quartet combines both: QuEST on the forward pass, stochastic rounding on the backward pass — a "best of both worlds" approach that no prior method had attempted with MXFP4.

Ingredient 4: A highly-optimized GPU implementation. Even the best algorithm is useless without an efficient implementation. The paper devotes significant attention (Section 4.4 and Appendix A.3) to describing a custom CUDA kernel built on CUTLASS 3.9 that fuses the Hadamard transform, quantization, scale calculation, and clipping mask generation into a single kernel (Stage 1), followed by a dedicated GEMM kernel that uses Blackwell's native tcgen05.mma instructions for block-scaled matrix multiplication (Stage 2). The implementation explicitly handles the alignment requirements of the MXFP4 format (scales computed over groups of 32 elements along the K-dimension) and optimizes the Hadamard transform by implementing it as a direct small GEMM against a fixed 32×32 Hadamard matrix in shared memory. Without this level of kernel engineering, the theoretical advantages of Quartet would not translate to real speedups — the paper notes that "a direct implementation of the above pattern would be slower than FP16 unquantized training, let alone optimized FP8."

The Scaling Law Premise: A Deeper Look

The paper's scaling law framework deserves special attention because it is the intellectual foundation for all subsequent claims about "optimality." The core equation (Section 4.1) is:

$L(N, D, P_{\text{forward}}, P_{\text{backward}}) = \left( \frac{A}{(N \cdot \text{eff}_N(P_{\text{forward}}))^\alpha} + \frac{B}{(D \cdot \text{eff}_D(P_{\text{backward}}))^\beta} \right)^\gamma + E$

This is a modification of the standard Chinchilla-style scaling law, with two multiplicative efficiency factors. The parameter efficiency $\text{eff}_N(P_{\text{forward}})$ captures the idea that quantizing the forward pass reduces the model's effective capacity — a model trained with FP4 forward pass behaves like a smaller model trained in full precision, because the quantization error acts as a regularizer or capacity bottleneck. This is consistent with prior work showing that compression (sparsity, quantization) can be modeled as a multiplicative reduction in effective parameter count.

The data efficiency $\text{eff}_D(P_{\text{backward}})$ captures the idea that quantizing the backward pass degrades the quality of the training signal — biased or noisy gradients mean the model learns less from each training example, effectively requiring more data to reach the same loss. This is modeled as a multiplicative reduction in effective data size: with backward-pass quantization, you need approximately $1/\text{eff}_D$ times more data to achieve the same loss. This is a novel parametrization that the paper validates experimentally (Figure 2) and theoretically (via the connection to unbiased gradient estimation in optimization theory, citing Alistarh et al., 2017).

The power of this framework is that it decomposes the complex behavior of a quantized training algorithm into two scalar parameters. If method A has both higher $\text{eff}_N$ and higher $\text{eff}_D$ than method B, then A is strictly superior — it will achieve lower loss at any combination of model size and data budget. If A has higher $\text{eff}_N$ but lower $\text{eff}_D$ , the comparison depends on whether the deployment scenario is parameter-bound or data-bound (the trade-off analyzed in Ingredient 2 and Figure 1).

The paper's experimental validation of this framework (Table 2) is particularly instructive. For forward-pass quantization, the ranking by $\text{eff}_N$ (QuEST > LSQ > RTN > SR) correlates perfectly with the ranking by MSE on Gaussian data — lower reconstruction error means higher effective parameter count. For backward-pass quantization, the ranking by $\text{eff}_D$ (SR > RTN) correlates with the projection magnitude misalignment metric — lower bias means higher effective data usage. This clean separation validates the paper's central thesis: forward and backward quantization impose fundamentally different requirements, and the optimal strategy treats them differently.

3. Technical Approach

3.1 Reader Orientation

The paper designs a full training pipeline — Quartet — that executes all three matrix multiplications of a Transformer linear layer natively in MXFP4, a 4-bit floating-point format supported in hardware by NVIDIA's Blackwell architecture. The system solves the problem of accurate end-to-end 4-bit LLM training by applying different quantization strategies to the forward and backward passes: minimizing reconstruction error (via QuEST's RMSE-based clipping with Hadamard transforms) on the forward pass where it directly reduces the model's effective parameter count, and eliminating gradient bias (via stochastic rounding) on the backward pass where unbiased gradient estimates are critical for convergence over long training runs. The solution's "shape" is a fused two-stage GPU kernel — a quantization-and-transform stage followed by a dedicated block-scaled GEMM stage — wrapped in a scaling-law framework that predicts when the throughput gains of FP4 fully compensate for its accuracy penalty, making it the "optimal" precision choice under a fixed compute budget.

3.2 Big-Picture Architecture (Diagram in Words)

The Quartet system has five major components, organized as a pipeline that processes each linear layer during both the forward and backward passes:

Hadamard Transform Module — applies a fixed, block-wise randomized or deterministic Hadamard rotation to the input tensors (activations $X$ , weights $W$ , or gradients $dy$ ) to "spread out" outlier values across all dimensions, making the tensor more amenable to uniform quantization. Implemented as a direct small GEMM against a 32×32 Hadamard matrix in GPU shared memory.
Forward-Pass Quantizer (QuEST) — takes the Hadamard-transformed activations and weights, computes per-group RMSE-optimal clipping factors, and quantizes them to MXFP4 with deterministic round-to-nearest. Produces the quantized tensors $X_q$ , $W_q$ plus clipping masks $M_x$ , $M_w$ that encode which values were clipped.
Backward-Pass Quantizer (Stochastic Rounding) — takes Hadamard-transformed gradient tensors, applies a 0.75× rescaling (to prevent saturation of the FP4 dynamic range), then quantizes to MXFP4 using stochastic rounding (probabilistic up/down rounding proportional to distance from grid points). This ensures the quantized gradient is an unbiased estimator of the true gradient.
Dedicated MXFP4 GEMM Engine — executes the actual matrix multiplications (forward: $X_q \times W_q$ ; backward: $G_q \times W_q^\top$ and $G_q^\top \times X_q^\top$ ) using Blackwell's native tcgen05.mma instructions, which natively support block-scaled matrix multiply with on-the-fly rescaling via per-32-element scale factors.
Scaling-Law Optimality Analyzer — an offline component that fits the parametric scaling law $L(N, D, P_{\text{forward}}, P_{\text{backward}})$ to training runs across multiple model sizes and data budgets, extracts the $\text{eff}_N$ and $\text{eff}_D$ coefficients for different quantization strategies, and determines which forward:backward precision combination is optimal given a compute budget. This is a design-time tool, not a runtime component.

Information flow during training: An activation tensor $X$ enters the linear layer → Hadamard transform produces $X_h$ → QuEST quantizes to $X_q$ with clipping mask $M_x$ → simultaneously, weights $W$ undergo the same pipeline → MXFP4 GEMM computes $y = X_q \times W_q$ → during backpropagation, gradient $dy$ enters → random Hadamard transform produces $G_h$ → stochastic rounding quantizes to $G_q$ → MXFP4 GEMMs compute $dx_q = G_q \times W_q^\top$ and $dW_q = G_q^\top \times X_q^\top$ → inverse Hadamard with clipping mask correction recovers full-precision gradients $dx$ , $dW$ for the optimizer.

3.3 Roadmap for the Deep Dive

First, the scaling-law framework (Section 4.1) — the equation that defines what "optimal" means and how forward/backward precision affects loss through the $\text{eff}_N$ and $\text{eff}_D$ coefficients. This is the intellectual foundation for every subsequent design choice.
Second, the mixed-precision optimality analysis (Section 4.2) — how hardware speedup measurements are incorporated into the scaling law to determine the "optimality region" where FP4 beats FP8, and why this requires treating forward and backward passes separately.
Third, the forward-pass quantization strategy (Section 4.3, first half) — the four candidate schemes (SR, RTN, LSQ, QuEST), how they are compared via the $\text{eff}_N$ metric and MSE on Gaussian data, and why QuEST wins.
Fourth, the backward-pass quantization strategy (Section 4.3, second half) — the error-bias trade-off, the novel projection magnitude misalignment metric, and why stochastic rounding wins for data-saturated training despite higher MSE.
Fifth, the GPU kernel implementation (Section 4.4) — the two-stage fused kernel design, how it maps to Blackwell's hardware primitives, and the performance breakdown that explains why naive implementations would be slower than FP16.
Sixth, the end-to-end Quartet algorithm (Algorithm 1) — the complete forward-backward procedure, how the pieces fit together, and the cost analysis of the Hadamard transforms relative to the GEMMs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper with a strong analytical framework whose core idea is that forward and backward passes require fundamentally different quantization strategies (error minimization vs. bias elimination), and that combining these strategies with an optimized GPU implementation makes native FP4 training not just possible but optimal under realistic compute budgets.

The Scaling Law: Decomposing Quantization Impact into Two Efficiency Factors

The paper proposes a parametric scaling law that explicitly models how the precision of the forward and backward passes affects final model quality. The equation is:

where $L$ is the validation loss (cross-entropy on C4), $N$ is the number of non-embedding model parameters, $D$ is the number of training tokens, $P_{\text{forward}}$ is the precision used for forward-pass matrix multiplications, $P_{\text{backward}}$ is the precision used for backward-pass matrix multiplications, $A, B, \alpha, \beta, \gamma, E$ are constants describing the baseline loss scaling for full-precision training, $\text{eff}_N(P_{\text{forward}}) \in (0, 1]$ is the parameter efficiency — a multiplicative factor on the model's effective capacity induced by forward-pass quantization, and $\text{eff}_D(P_{\text{backward}}) \in (0, 1]$ is the data efficiency — a multiplicative factor on the effective training data induced by backward-pass quantization.

What it computes: given a model with $N$ parameters trained on $D$ tokens with forward precision $P_{\text{forward}}$ and backward precision $P_{\text{backward}}$ , the equation predicts the final validation loss. The term inside the parentheses is a weighted sum of an inverse-power-law in the effective parameter count $N \cdot \text{eff}_N$ and an inverse-power-law in the effective data size $D \cdot \text{eff}_D$ . The outer exponent $\gamma$ modulates the interaction between the parameter-bound and data-bound loss components; the constant $E$ represents the irreducible loss (the entropy of the data distribution). When $\text{eff}_N = \text{eff}_D = 1$ (full precision), the equation reduces to the standard Chinchilla-style scaling law.

Why this form: the standard Chinchilla law models loss as a function of $N$ and $D$ with fixed exponents, reflecting the empirical observation that loss decreases predictably as either parameter count or data volume increases, with diminishing returns. This paper extends that law by inserting multiplicative efficiency factors. The $\text{eff}_N$ factor models forward-pass quantization as a capacity bottleneck: if quantizing the forward pass to FP4 reduces the effective parameter count to 70% of the original, then the loss behaves as if you trained a model with $0.7N$ parameters in full precision. This follows prior work by Frantar et al. (2023) and Kumar et al. (2024) showing that compression can be modeled as a multiplicative reduction in effective parameter count, which the paper cites explicitly. The $\text{eff}_D$ factor models backward-pass quantization as a training signal degradation: if biased gradients mean the model extracts only 60% as much information from each training token, then the loss behaves as if you trained on $0.6D$ tokens. This is a novel parametrization proposed by the paper, consistent with optimization theory results (Alistarh et al., 2017, showing that unbiased gradient estimators are critical for SGD convergence) and validated experimentally in Figure 2. The decomposition into separate $\text{eff}_N$ and $\text{eff}_D$ factors is crucial because it allows forward and backward precision to be optimized independently — the forward precision only affects $\text{eff}_N$ , and the backward precision only affects $\text{eff}_D$ . A simpler model that used a single efficiency factor for both passes would fail to capture the experimental finding that QuEST is best for the forward pass (high $\text{eff}_N$ , via low MSE) while stochastic rounding is best for the backward pass (high $\text{eff}_D$ , via zero bias).

Fitting the Scaling Law: A Two-Stage Procedure

The paper fits the scaling law using a two-stage procedure described in Appendix A.2, following prior work by Busbridge et al. (2025):

Stage 1: Fit baseline (full-precision) parameters. The authors train a grid of models with BF16 precision (used as the idealized FP8-equivalent baseline) across four model sizes — $N \in [30\text{M}, 50\text{M}, 100\text{M}, 200\text{M}]$ non-embedding parameters — and six data-to-parameter ratios — $D/N \in [25, 50, 100, 200, 400, 800]$ . This yields 24 training runs. The parameters $A, B, \alpha, \beta, \gamma, E$ are fitted to these 24 data points using Huber loss with $\delta = 10^{-4}$ on the logarithm of the validation loss. The use of Huber loss (rather than MSE) makes the fit robust to outliers; the log-space fitting ensures that relative errors matter more than absolute errors, which is appropriate because loss differences of 0.01 matter more when the baseline loss is 2.5 than when it is 5.0.

Stage 2: Fit efficiency coefficients per quantization method. With the baseline parameters fixed, the authors fit $\text{eff}_N$ and $\text{eff}_D$ separately for each forward and backward quantization scheme. For forward-only analysis (isolating $\text{eff}_N$ ), they keep the backward pass in full precision and fit only $\text{eff}_N$ using runs with different forward quantizers. For backward-only analysis (isolating $\text{eff}_D$ ), they keep the forward pass in full precision and fit only $\text{eff}_D$ . For end-to-end 4-bit methods (the comparison in Table 5/Table 3 of the paper), both coefficients are fit jointly from runs where both passes are quantized with the method's respective forward and backward strategies. The fitting uses the same Huber loss on log-loss. The fitted baseline coefficients are reported in Table 6 of the appendix and include the full set of $A, B, \alpha, \beta, \gamma, E$ values.

Design choice — why not fit everything jointly? The two-stage procedure ensures that the baseline scaling behavior is determined by the cleanest data (full-precision runs), preventing the quantized runs — which are noisier and may have different optimal hyperparameters — from distorting the fundamental $N$ - $D$ relationship. This is analogous to how scaling laws are typically fitted: first establish the functional form on a high-quality baseline, then fit the compression-specific modifiers.

The Mixed-Precision Optimality Analysis: When FP4 Beats FP8

The scaling law alone tells you which method achieves lower loss for a fixed $(N, D)$ . But the real question is: under a fixed compute (wall-clock) budget, which precision combination produces the best model? This requires incorporating hardware speedup measurements.

The paper models this by defining speedup factors relative to an FP8 baseline (Table 1 and the accompanying analysis in Section 4.2). Specifically:

$\text{spfw}(P_{\text{forward}})$ is the forward-pass speedup of precision $P_{\text{forward}}$ relative to FP8 forward. Under the BOPS (bit-wise operations) model, this is $8 / \text{bits}(P_{\text{forward}})$ . Under real RTX 5090 measurements, this is the measured kernel speedup (up to ~2× for FP4 forward).
$\text{spbw}(P_{\text{backward}})$ is the backward-pass speedup of precision $P_{\text{backward}}$ relative to FP8 backward. Under BOPS, this is $8 / \text{bits}(P_{\text{backward}})$ ; under real measurements, up to ~1.5× for FP4 backward.
$\text{sptr}(P_{\text{forward}}, P_{\text{backward}})$ is the overall training speedup, defined as the harmonic mean of forward and backward speedups weighted by their compute fractions: $\text{sptr} = 1 / \left(\frac{1/3}{\text{spfw}} + \frac{2/3}{\text{spbw}}\right)$ . The 1/3 and 2/3 weights reflect that the forward pass consumes approximately 33% of training FLOPs and the backward pass consumes approximately 66%.

Given a maximum forward-pass parameter budget $N_{\max}$ (e.g., the largest model that fits in GPU memory with the chosen precision) and a maximum total training compute budget $N_{\max} \cdot D_{\max}$ , the effective loss achieved by a precision combination $(P_{\text{forward}}, P_{\text{backward}})$ is:

$\text{Loss}\left(N_{\max} \cdot \text{spfw}, \; D_{\max} \cdot \frac{\text{sptr}}{\text{spfw}}, \; P_{\text{forward}}, \; P_{\text{backward}}\right)$

where the first argument $N_{\max} \cdot \text{spfw}$ is the effective parameter count — because forward-pass speedup lets you train a proportionally larger model in the same memory and time — and the second argument $D_{\max} \cdot \text{sptr}/\text{spfw}$ is the effective data size — because overall training speedup gives you more tokens, but the model is also larger (by $\text{spfw}$ ), so the tokens-per-parameter ratio adjusts accordingly.

What it computes: for a fixed hardware budget (memory capacity and training time), this expression predicts the best validation loss achievable by each precision combination. The precision combination that minimizes this loss is declared "optimal" for that budget.

Why this form: previous work often required lower precision to be lossless — to achieve the same accuracy as full precision. The paper argues this is an unnecessarily restrictive criterion. If FP4 training is 1.6× faster than FP8 training, then under a fixed time budget you can train on 1.6× more data with FP4 than with FP8. Even if FP4 has slightly worse $\text{eff}_N$ and $\text{eff}_D$ , the additional data may more than compensate. The equation formalizes this compensation: $\text{spfw}$ and $\text{sptr}$ multiply $N_{\max}$ and $D_{\max}$ respectively, directly countering the sub-unity efficiency factors. The optimal precision is the one where the throughput gain times the efficiency factor is maximal.

The paper illustrates this analysis in Figure 1 (b) and (c). The X-axis is the relative training compute budget (normalized by the FP8 baseline), and the Y-axis is the model size. Each colored region shows where a particular forward:backward precision combination is optimal. The green region shows where FP4:FP4 is optimal — this region expands when using real RTX 5090 speedups (Figure 1c) compared to the idealized BOPS model (Figure 1b), because the real speedups are higher than the naive bit-width ratio would predict. The paper notes that "popular models such as larger Llama3 or Qwen2.5 models fall into the FP4:FP4 optimality region, implying that training similar models in FP4 might have been optimal."

Forward-Pass Quantization: Why QuEST Maximizes Parameter Efficiency

The forward pass of a linear layer computes $y = X \times W$ , where $X$ is the activation matrix and $W$ is the weight matrix. In Quartet, both $X$ and $W$ are quantized to MXFP4 before multiplication. The goal is to minimize the reconstruction error — the difference between the full-precision product $X \times W$ and the quantized product $X_q \times W_q$ — because this error directly reduces the model's effective capacity (captured by $\text{eff}_N$ ).

The paper evaluates four candidate forward-pass quantization schemes, all applied after a block-wise Hadamard transform that makes the tensor values more uniformly distributed (mitigating outliers):

Stochastic Rounding (SR) with AbsMax normalization: For each group of 32 elements, compute the scale as the maximum absolute value in the group ( $\text{AbsMax}$ ), then quantize each element using stochastic rounding — rounding up or down to the nearest MXFP4 grid point with probability proportional to the element's distance from each grid point. This produces an unbiased quantized representation (the expected value of the quantized tensor equals the original tensor) but has higher mean squared error than deterministic rounding because some values are rounded away from the nearest grid point.
Round-to-Nearest (RTN) with AbsMax normalization: Same group-wise scale computation, but using deterministic round-to-nearest. This minimizes the per-element MSE but introduces systematic bias — values are always rounded to the closest grid point, which for non-uniform distributions means the quantized values are systematically shifted relative to the originals.
Learned Step-size Quantization (LSQ) with RTN: Instead of using AbsMax for the scale, LSQ treats the quantization scale as a learnable parameter, optimized during training to minimize the task loss. The scale is initialized based on the tensor statistics and then updated via gradient descent alongside the model weights. This allows the scale to adapt to the specific distribution of each tensor, potentially finding a better trade-off between clipping error (from values that exceed the representable range) and rounding error (from coarse grid spacing).
QuEST (RMSE-based clipping with RTN): QuEST computes the per-group scale by finding the clipping threshold that minimizes the root mean squared error between the original and quantized tensor. Specifically, for a given group of values, QuEST searches for a clipping factor $c$ such that quantizing the tensor clamped to $[-c \cdot \sigma, c \cdot \sigma]$ (where $\sigma$ is the standard deviation of the group) minimizes the RMSE. This is more principled than AbsMax (which sets $c$ by the single largest value, making it sensitive to outliers) and more stable than LSQ (which requires learning the scale alongside the weights, adding optimization complexity). QuEST also produces a clipping mask — a binary indicator for which values were clamped — that is used later in the backward pass to correct for the clipping error (see the backward pass section below).

Comparison methodology. The paper compares these four schemes by training 30M-parameter Llama models with each forward quantizer (keeping the backward pass in full precision) and fitting the $\text{eff}_N$ parameter from the scaling law (Section 4.1). Additionally, the paper measures the MSE of each quantizer on random Gaussian data — a proxy for the reconstruction quality independent of the training dynamics. The results are in Table 2 (first two data rows/columns).

Results and why QuEST wins. QuEST achieves the highest $\text{eff}_N = 0.86$ , compared to LSQ at 0.83, RTN at 0.81, and SR at 0.64. The ranking by $\text{eff}_N$ correlates perfectly with the ranking by MSE on Gaussian data — QuEST has the lowest MSE, SR has the highest. This validates the paper's thesis that forward-pass quality is about minimizing reconstruction error: the method that best preserves the values of the quantized tensors also best preserves the model's effective capacity. The paper notes that this finding "aligns with the analysis of Chmiel et al. (2024) that determined deterministic RTN to always be preferable to stochastic rounding for the forward pass." The poor performance of SR on the forward pass is instructive: although SR produces an unbiased estimate of each value, the higher variance (MSE) dominates the forward-pass quality because the forward pass is a feedforward computation — there is no iterative optimization that could average out the noise over multiple steps. The Hadamard transform applied before quantization is essential for all schemes because it "spreads out" outlier values across dimensions, making the tensor more uniform and reducing the clipping error that would otherwise dominate at 4-bit precision.

Backward-Pass Quantization: The Error-Bias Trade-Off and Why Stochastic Rounding Wins

The backward pass of a linear layer computes two matrix multiplications: $dx = dy \times W^\top$ (gradient with respect to the input) and $dW = dy^\top \times X$ (gradient with respect to the weights), where $dy$ is the gradient of the loss with respect to the layer's output. In Quartet, the tensors involved in these multiplications (specifically, the Hadamard-transformed versions of $dy$ , $W^\top$ , and $X^\top$ ) are quantized to MXFP4 before the GEMMs.

The key difference from the forward pass is that the backward pass feeds into the optimizer — the quantized gradients are accumulated into the weight updates over many steps. Optimization theory (specifically, the analysis of stochastic gradient descent) shows that bias in the gradient estimator is far more damaging than variance. A biased gradient estimator causes the optimization to converge to a different point than the true loss minimum, with the asymptotic gap proportional to the squared bias. Variance (noise) in the gradient estimator slows convergence but does not change the asymptotic solution, and its effect can be mitigated by increasing the batch size or training for more steps.

The paper formalizes this trade-off through a novel projection magnitude misalignment metric, derived from the theoretical analysis of Vargaftik et al. (2021, 2022). The setup is as follows: consider quantizing a vector $X$ after applying a random Hadamard transform $\widehat{H}$ . Vargaftik et al. showed that RTN quantization after random rotation is unbiased in direction (the cosine similarity between the original and quantized vector is preserved in expectation) but biased in magnitude — the quantized vector is systematically shorter or longer than the original. To correct for this, they proposed a rescaling factor:

$S = \frac{\langle X, X \rangle}{\langle \widehat{H}(X, \xi), \text{RTN}(\widehat{H}(X, \xi)) \rangle}$

where $X$ is the original vector, $\xi$ is the random seed for the Hadamard transform, $\widehat{H}(X, \xi)$ is the randomly rotated vector, $\text{RTN}(\cdot)$ denotes round-to-nearest quantization, and $\langle \cdot, \cdot \rangle$ is the inner product. When the quantized vector is multiplied by $S$ , the result is an unbiased estimator of $X$ : $\mathbb{E}_\xi[S \cdot \text{RTN}(\widehat{H}(X, \xi))] = X$ .

Why this matters for MXFP4: the Vargaftik rescaling $S$ requires computing inner products over the entire vector — it is a global rescaling that is incompatible with MXFP4's per-32-element block-wise scaling. The paper cannot directly use this technique. However, the quantity $1 - \mathbb{E}[1/S]$ — which the paper calls the projection magnitude misalignment — measures the degree of systematic magnitude bias introduced by a quantization scheme, without applying any correction. A quantizer with zero misalignment ( $\mathbb{E}[1/S] = 1$ ) is unbiased in both direction and magnitude; a quantizer with positive misalignment systematically shrinks or expands the quantized vectors.

Computing the misalignment metric. For each quantization scheme (SR with AbsMax, RTN with AbsMax), the paper computes $1 - \mathbb{E}[1/S]$ by: (1) sampling random vectors $X$ from distributions matching the observed statistics of neural network gradients, (2) applying the random Hadamard transform $\widehat{H}$ with random seed $\xi$ , (3) quantizing with the scheme under test, (4) computing the inner products to get $S$ , (5) averaging $1/S$ over many random seeds and reporting $1 - \text{average}(1/S)$ . The results are in Table 2 (second data column, backward-specific rows).

Stochastic rounding achieves zero misalignment. The critical finding: SR with AbsMax achieves exactly zero projection magnitude misalignment ( $1 - \mathbb{E}[1/S] = 0.00$ in Table 2). This is because stochastic rounding is an unbiased estimator at the element level ( $\mathbb{E}[\text{SR}(x)] = x$ for each element $x$ ), and linearity of expectation then ensures that any linear functional (including inner products) of the quantized vector is also unbiased. RTN, by contrast, shows a measurable misalignment (the exact value is reported in Table 2), because deterministic rounding systematically biases the magnitudes.

The trade-off visualized. Table 2 shows that SR has the highest MSE (worst forward-pass quality) but zero misalignment (best backward-pass quality), while RTN/QuEST have lower MSE (better forward-pass quality) but non-zero misalignment (worse backward-pass quality). This is the fundamental error-bias trade-off that the paper identifies as the central challenge in 4-bit training: no single quantizer is simultaneously optimal for both passes.

Connecting misalignment to training dynamics. Figure 2 provides the empirical bridge between the theoretical misalignment metric and actual training outcomes. The figure analyzes a 30M-parameter Llama model with backward-pass quantization:

Figure 2(a) — Cosine Similarity: Plots the cosine similarity between the quantized and full-precision activation gradients ( $dx$ ) as a function of backpropagation depth (layer index, from output to input). RTN achieves higher cosine similarity than SR at all depths — the quantized gradient direction is better preserved by RTN.
Figure 2(b) — Projection Magnitude Misalignment: Plots the misalignment metric $1 - \mathbb{E}[1/S]$ at each layer. SR achieves near-zero misalignment throughout, while RTN accumulates significant magnitude bias, especially in earlier (deeper) layers where the misalignment compounds across multiple quantization steps.
Figure 2(c) — Loss Gap vs. Full Precision: Plots the validation loss difference between quantized and full-precision training as a function of the data-to-parameter ratio $D/N$ . For short training runs (low $D/N$ ), RTN actually outperforms SR — the higher cosine similarity (better gradient direction) matters more when the total number of optimization steps is small. But as $D/N$ increases beyond approximately 400, SR pulls ahead and the gap widens with more data — the accumulated bias in RTN's gradient estimates causes the optimization to converge to a worse solution, and this effect compounds over many training steps. The inflection point is around $D/N = 400$ .

Why Quartet uses SR on the backward pass. The paper's target regime is long-training, data-saturated LLM pre-training (token-to-parameter ratios up to 800×), where the bias in RTN's gradient estimator would cause significant degradation. Stochastic rounding's zero misalignment guarantees that the gradient estimates are unbiased, which is critical for convergence to the correct solution over many steps. The higher per-step variance of SR is acceptable because it averages out over the course of training — the optimizer sees many noisy but unbiased gradient estimates, which is the standard SGD setting with well-understood convergence properties.

The 3/4 rescaling factor. Algorithm 1 shows that before stochastic rounding, the Hadamard-transformed gradient tensors are multiplied by 3/4: $G_q = \text{SR}(\frac{3}{4}G_h)$ . This rescaling serves a critical practical purpose: the MXFP4 format has a limited dynamic range (with E2M1, the maximum representable value is 6.0), and gradient tensors after Hadamard transform can contain values that exceed this range. Multiplying by 3/4 shrinks the values so that fewer of them saturate the FP4 grid. The backward pass then compensates by multiplying the dequantized result by 16/9 (the inverse of $(3/4)^2$ , since two quantized tensors are multiplied in the GEMM). This is a common trick in low-precision training: deliberately under-scale before quantization to avoid clipping, then analytically correct the scaling afterward. Because stochastic rounding is linear in expectation, this scaling compensation does not introduce bias.

The Forward-Pass Quantization: QuEST in Detail

The QuEST forward-pass quantization, as applied in Quartet, consists of the following steps applied to both the activation matrix $X_h$ and the weight matrix $W_h$ after the Hadamard transform:

Group partitioning: The matrix is partitioned into groups of 32 consecutive elements along the inner dimension (the K-dimension of the GEMM), matching the MXFP4 block size. For an $M \times K$ matrix, this produces $M \times \lceil K/32 \rceil$ groups.
Per-group standard deviation: For each group of 32 elements, compute the standard deviation $\sigma$ of the values, plus the mean $\mu$ for centering.
Optimal clipping factor search: For each group, QuEST searches for the clipping factor $c^*$ that minimizes the expected RMSE between the original values and their quantized representations. The search space is typically $c \in [1.0, 5.0]$ , and the objective is: $\text{RMSE}(c) = \sqrt{\mathbb{E}_{x \sim \mathcal{N}(0,1)}\left[ (x - Q(\text{clamp}(x, -c, c)))^2 \right]}$ where $\text{clamp}(x, -c, c)$ clips $x$ to the range $[-c, +c]$ , $Q(\cdot)$ quantizes to MXFP4, and the expectation is over the empirical distribution of values in the group. A grid search or analytic approximation is used to find $c^*$ .
Scale computation: The per-group scale $s$ for MXFP4 is set as $s = c^* \cdot \sigma / \text{max\_representable}$ , where $\text{max\_representable} = 6.0$ for the MXFP4 E2M1 format. This ensures that values within $[-c^*\sigma, +c^*\sigma]$ map to the full representable range of the format.
Quantization: Each value $x$ in the group is quantized as $x_q = \text{RTN}(\text{clamp}(x, -c^*\sigma, +c^*\sigma) / s)$ , where RTN is round-to-nearest. Values outside the clipping range are clamped and flagged in the clipping mask.
Clipping mask generation: A binary mask $M$ of the same shape as the matrix records which values were clipped (1 if clipped, 0 otherwise). This mask is stored alongside the quantized tensor and used in the backward pass to correct for the clipping error — specifically, the gradient through a clipped activation is zero, and the mask allows the backward pass to propagate gradients only through the unclipped values.

Why QuEST over LSQ? LSQ learns the scale parameters via gradient descent, which adds optimizer state and hyperparameter tuning complexity. QuEST's analytic RMSE minimization is simpler, more stable, and — as Table 2 shows — achieves better $\text{eff}_N$ . The paper hypothesizes that LSQ's learned scales may overfit to the early training distribution and fail to adapt as the weight and activation distributions shift during training, whereas QuEST recomputes the optimal clipping factor from the current tensor statistics at every forward pass.

Why QuEST over AbsMax? AbsMax sets the scale based on the single largest absolute value in the group, which makes it extremely sensitive to outliers. If one element in a group of 32 is 10× larger than the others, AbsMax scales the entire group to accommodate that outlier, leaving the other 31 values quantized to a tiny fraction of the representable range and thus effectively at 1-2 bits of precision. QuEST's RMSE-optimal clipping deliberately allows some outliers to be clipped, trading a small amount of clipping error for much finer quantization of the inlier values.

The Hadamard Transform: Why It Is Necessary and How It Is Implemented

All quantization in Quartet (both forward and backward) operates on tensors that have been transformed by a block-wise Hadamard matrix. The Hadamard transform serves to reduce the dynamic range of the tensor values and eliminate outliers — a small number of dimensions with disproportionately large magnitudes. This is a well-established technique in low-precision inference (QuaRot, QuIP) and training (HALO, Jetfire). The intuition: an outlier in the original basis is spread evenly across all dimensions in the Hadamard basis, so no single dimension has an extreme value after the transform.

Mathematical definition. For a block size $g$ (equal to the MXFP4 group size of 32), the block-wise Hadamard transform $H_g$ is applied as follows: the input vector of length $d$ is split into blocks of size $g$ (padding the last block if $d$ is not a multiple of $g$ ), and each block is multiplied by the $g \times g$ normalized Hadamard matrix. The Hadamard matrix $H_g$ is defined recursively:

$H_{2} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}; \quad H_{2k} = \frac{1}{\sqrt{2}} H_2 \otimes H_k$

where $\otimes$ is the Kronecker product. The normalization factor $1/\sqrt{g}$ ensures that $H_g$ is orthogonal ( $H_g^\top H_g = I$ ), so the transform preserves inner products and norms — applying $H_g$ and then $H_g^{-1} = H_g^\top$ recovers the original vector exactly.

Forward pass: deterministic Hadamard. On the forward pass, Quartet applies a fixed (deterministic) Hadamard transform $H_g$ to both activations $X$ and weights $W$ before quantization. The same fixed transform is used throughout training because the weights are persistent and need a consistent basis. The transformed tensors $X_h = H_g(X)$ and $W_h = H_g(W)$ have the property that $X_h \times W_h^\top = X \times W^\top$ (the Hadamard transforms cancel in the matrix product, up to a transpose for one operand), so the forward-pass computation is mathematically equivalent to the un-transformed version.

Backward pass: randomized Hadamard. On the backward pass, Quartet applies a randomized Hadamard transform $\widehat{H}_g$ , following Vargaftik et al. (2021). The randomization consists of: (1) applying a random sign flip to each element (multiplying by $\pm 1$ with equal probability), (2) applying the deterministic Hadamard transform $H_g$ , and (3) applying another random sign flip. This randomization is crucial for the theoretical unbiasedness property: while the deterministic Hadamard makes quantization unbiased in direction, the randomized version additionally makes it unbiased in magnitude when combined with stochastic rounding, as the projection magnitude misalignment analysis shows. The random seed $\xi$ is passed explicitly and can be reproduced for the backward pass's inverse transform.

Inverse Hadamard and clipping mask correction. After the quantized GEMM in the backward pass produces $dx_q$ (the quantized gradient with respect to the input), Quartet applies the inverse Hadamard transform and the QuEST clipping mask correction:

$dx = \frac{16}{9} H_g^{-1}(dx_q \odot M_x)$

where $16/9$ is the scaling compensation for the $3/4$ factor applied before quantization (inverse of $(3/4)^2$ ), $H_g^{-1}$ is the inverse Hadamard transform (which for the deterministic forward Hadamard is just $H_g$ itself, since $H_g$ is symmetric and orthogonal), and $\odot$ is element-wise multiplication with the clipping mask $M_x$ . The mask zeroes out gradient contributions from activation values that were clipped during the forward pass — since those values were clamped and their exact magnitude was lost, their contribution to the gradient is unreliable, so the mask sets those gradient entries to zero. The same procedure applies to the weight gradient $dW$ using the weight clipping mask $M_w$ .

Implementation as a small GEMM. Because the block size $g = 32$ is small, the Hadamard transform can be implemented as a direct matrix multiplication between the input tensor (reshaped to expose the 32-element blocks as the inner dimension) and the fixed $32 \times 32$ Hadamard matrix. This is efficient because: (1) the Hadamard matrix is small and fits in GPU registers or shared memory, (2) the matrix multiplication maps directly to the GPU's warp-level matrix multiply-accumulate (MMA) instructions, and (3) the cost is $O(d \log g)$ operations, which for $g = 32$ is approximately $5d$ operations — negligible compared to the $O(d^2)$ GEMM that follows.

The GPU Kernel Implementation: Two-Stage Fused Design

The paper's most significant engineering contribution is a highly-optimized CUDA implementation of the Quartet algorithm on Blackwell GPUs, built on CUTLASS 3.9. The implementation uses a two-stage design where the quantization-related operations are fused into a single kernel, followed by a dedicated block-scaled GEMM kernel.

Stage 1: Fused Quantization Kernel. This kernel combines the Hadamard transform, quantization, scale factor computation, and clipping mask generation into a single GPU kernel. The key design decisions:

Hadamard as Shared Memory GEMM: The kernel first loads a tile of the input matrix into GPU Shared Memory (SMEM). The Hadamard transform is implemented as a warp-level matrix multiplication between the input tile and the fixed $32 \times 32$ Hadamard matrix, producing output in FP32 precision. Using CUTLASS's multilevel tiling templates, this operation is fused with the subsequent quantization steps without writing intermediate results to Global Memory (GMEM). The input and Hadamard matrix are loaded from GMEM only once; the transformed values stay in SMEM and then in the Register File (RF).
FP32 to FP4 Downcast via PTX Instructions: Blackwell provides PTX (Parallel Thread Execution) instructions for direct FP32-to-FP4 conversion in the E2M1 format. The kernel uses these hardware instructions — rather than software-based bit manipulation — to downcast the FP32 transformed values to FP4, which is both faster and ensures bit-exact compliance with the MXFP4 specification.
Scale Factor Computation in E8M0: For each group of 32 elements, the kernel computes a scaling factor. The scale is represented in the E8M0 format (8 exponent bits, no mantissa), which is the required scale format for MXFP4. The computation involves finding the maximum absolute value in the group (determined by the QuEST clipping factor $c^*$ on the forward pass, or the $3/4$ rescaling on the backward pass) and encoding it as a power-of-two exponent.
Clipping Mask Generation (Forward Only): On the forward pass, the kernel additionally generates the QuEST clipping mask. For each element, if the absolute value exceeds $c^* \cdot \sigma$ , the mask bit is set; otherwise, it is cleared. The mask is stored to GMEM alongside the quantized values and scales.
Wide Memory Instructions: All writes to GMEM use the widest memory instructions available on the Blackwell architecture (128-byte or 256-byte transactions) to maximize memory bandwidth utilization. The three output tensors (values, scales, mask) are written in a coalesced pattern to minimize the number of memory transactions.
Tile Size Tuning: The paper reports (Figure 6, Appendix A.3) that increasing the threadblock tile size from $32 \times 32$ to $128 \times 32$ significantly reduces the fraction of time spent in the quantization stage. Larger tiles mean more active warps per threadblock, higher arithmetic intensity, and better latency hiding. The CUTLASS multilevel tiling (threadblock, warp, and instruction-level) is tuned to optimize data movement through the memory hierarchy.

Stage 2: Dedicated Block-Scaled GEMM Kernel. This kernel performs the actual matrix multiplication using Blackwell's native tcgen05.mma instructions, which support block-scaled matrix multiply of the form:

$D = C + (A \times \text{SFA}) \cdot (B \times \text{SFB})$

where $A$ and $B$ are the quantized matrices in MXFP4, $\text{SFA}$ and $\text{SFB}$ are their corresponding scale factor matrices (in E8M0), and the multiplication $(A \times \text{SFA})$ means each element of $A$ is implicitly multiplied by its group's scale factor before the matrix multiplication. This rescaling happens automatically in the Tensor Core hardware — no software-based rescaling is needed at the CUDA level.

Scale Factor Layout: MXFP4 stores scales such that every 32 consecutive elements along the K-dimension share a scale factor. For an $M \times K$ matrix $A$ , the scale matrix $\text{SFA}$ has shape $M \times \lceil K/32 \rceil$ . The kernel includes a reorganization step that arranges the scale factors generated in Stage 1 into the exact layout expected by the tcgen05.mma instruction. This reorganization is identified as a performance bottleneck in naive implementations (the "rearrangement" stage in Figure 6) and is optimized via a custom Triton kernel in the rightmost configuration.
Accumulator Precision: The matrix multiplication accumulator $C$ and output $D$ are in FP32, following standard practice for low-precision GEMMs — the multiplies happen in low precision, but the accumulation is in high precision to avoid numerical overflow or underflow over the inner dimension.

Why a naive implementation would be slower than FP16. The paper makes an important point: without careful kernel fusion and memory hierarchy optimization, the overhead of the Hadamard transforms, quantization, and scale factor manipulation would exceed the speedup from the faster FP4 GEMM. Specifically, if each operation were a separate kernel with GMEM round-trips between them, the memory bandwidth bottleneck would dominate, and the end-to-end latency would be worse than simply running the GEMM in FP16. The fused two-stage design eliminates intermediate GMEM writes: data flows from GMEM → SMEM (Hadamard) → RF (quantization) → GMEM (final quantized output), with only one GMEM write per tensor. The subsequent GEMM kernel then reads the quantized tensors from GMEM with the standard memory access pattern.

Performance breakdown (Figure 6, Appendix A.3). The figure shows the fraction of runtime spent in three stages — matrix multiplication, quantization-related operations, and rearrangement of scaling factors — for three kernel configurations on a Llama-7B model with batch size 64 and sequence length 512. With the basic $32 \times 32$ tile, the quantization stage consumes a significant fraction of total time (visible in the left subplot). Increasing the tile to $128 \times 32$ reduces quantization overhead substantially (center subplot). The right subplot shows an additional Triton-optimized kernel that further reduces the rearrangement overhead, so that the GEMM dominates total runtime — this is the ideal regime where the overhead of quantization is nearly amortized by the faster matrix multiplication.

The Complete Quartet Algorithm: Step-by-Step Execution

Algorithm 1 in the paper provides the complete pseudocode for a Quartet linear layer. We walk through each step with full explanation of the data flow and design choices.

Forward Pass (lines 2-7):

Hadamard Transform of Input and Weights: $X_h = H_g(X)$ and $W_h = H_g(W)$ , where $H_g$ is the block-wise deterministic Hadamard transform with block size $g = 32$ . The input $X$ has shape $[B, S, d_{\text{in}}]$ (batch × sequence × input dimension), and the weights $W$ have shape $[d_{\text{out}}, d_{\text{in}}]$ . The Hadamard transform is applied independently to each block of 32 elements along the feature dimension. After the transform, the outlier energy is spread across all 32 dimensions in each block, making the tensor more amenable to uniform quantization.
QuEST Quantization: $(X_q, M_x) = \text{QuEST}(X_h)$ and $(W_q, M_w) = \text{QuEST}(W_h)$ . The QuEST function: (a) partitions the tensor into groups of 32, (b) computes per-group standard deviation, (c) finds the RMSE-optimal clipping factor $c^*$ , (d) computes per-group scale factors, (e) quantizes with RTN to MXFP4, (f) generates the clipping mask. The output $X_q$ is the quantized tensor in MXFP4 format (values + per-32-element scales), and $M_x$ is a binary mask of the same shape indicating which elements were clipped.
Low-Precision GEMM: $y = \text{GEMM}_{\text{LP}}(X_q, W_q)$ , where $\text{GEMM}_{\text{LP}}$ is the Stage 2 dedicated kernel that executes $X_q \times W_q^\top$ using Blackwell's tcgen05.mma instruction with automatic scale factor application. The output $y$ is in FP32 (the accumulator precision).
Context Save: The forward pass saves $\text{ctx} = \{X_q, W_q, M_x, M_w\}$ for use in the backward pass. Notably, the full-precision activations $X$ are not saved — only the quantized version $X_q$ and the clipping mask $M_x$ . This saves memory compared to standard training, which must store the full-precision activations for the backward pass. Memory savings from activation quantization are an additional benefit, though the paper focuses on throughput gains.

Backward Pass (lines 8-12):

Unpack Context: $\{X_q, W_q, M_x, M_w\}$ are loaded from the saved context. These were stored in MXFP4 format and must be dequantized (by applying their scale factors) before use in the backward GEMMs — or, more precisely, they are used as inputs to the backward GEMMs where the scale factors are applied automatically by the hardware.
Randomized Hadamard of Gradients and Transposed Weights: $G_h = \widehat{H}_g(dy, \xi)$ and $W_h^\top = \widehat{H}_g(W_q^\top, \xi)$ . The output gradient $dy$ has shape $[B, S, d_{\text{out}}]$ . The randomized Hadamard transform $\widehat{H}_g$ applies random sign flips before and after the deterministic Hadamard, using a random seed $\xi$ . The same seed is used for both tensors so that the randomization is consistent across the two operands of the subsequent GEMM — this preserves the mathematical equivalence $\widehat{H}_g(dy) \times \widehat{H}_g(W_q^\top) = dy \times W_q^\top$ (in expectation, under the randomized transform). The transposed weight matrix $W_q^\top$ is first dequantized from its stored MXFP4 format.
Rescaling and Stochastic Rounding: $G_q = \text{SR}(\frac{3}{4}G_h)$ and $W_q^\top = \text{SR}(\frac{3}{4}W_h^\top)$ . The $3/4$ factor shrinks the values to reduce saturation of the FP4 grid; stochastic rounding then quantizes each element to MXFP4 by rounding up or down with probability proportional to the distance from the nearest grid points. The result $G_q$ and $W_q^\top$ are MXFP4 tensors with per-32-element scales.
First Backward GEMM (Input Gradient): $dx_q = \text{GEMM}_{\text{LP}}(G_q, W_q^\top)$ . This computes $dx_q = G_q \times W_q$ (since $W_q^\top$ was already transposed, the GEMM computes $G_q \times W_q$ , which is $dy_{\text{transformed}} \times W$ ). The result $dx_q$ is the quantized gradient with respect to the input, still in the Hadamard-transformed space and with the $3/4$ rescaling baked in.
Inverse Hadamard and Mask Correction for $dx$ : $dx = \frac{16}{9} H_g^{-1}(dx_q \odot M_x)$ . Step-by-step: (a) $dx_q \odot M_x$ zeroes out gradient entries corresponding to forward-pass activations that were clipped — since those activations' exact values were lost, their gradients are unreliable and are set to zero; (b) $H_g^{-1}$ applies the inverse deterministic Hadamard transform to return to the original basis; (c) $16/9$ is the scaling compensation — because both $G_q$ and $W_q^\top$ were scaled by $3/4$ before the GEMM, the product is scaled by $(3/4)^2 = 9/16$ , so multiplying by $16/9$ restores the correct magnitude. The output $dx$ has shape $[B, S, d_{\text{in}}]$ and is the gradient passed to the previous layer.
Second Backward GEMM Setup (Weight Gradient): $G_h^\top = \widehat{H}_g(dy^\top, \xi)$ and $X_h^\top = \widehat{H}_g(X_q^\top, \xi)$ . Here $dy^\top$ has shape $[d_{\text{out}}, B \cdot S]$ (the output gradient transposed), and $X_q^\top$ has shape $[d_{\text{in}}, B \cdot S]$ (the quantized input transposed). The same random seed $\xi$ is used. Steps 7-8 repeat the rescaling and stochastic rounding for these transposed tensors: $G_q^\top = \text{SR}(\frac{3}{4}G_h^\top)$ and $X_q^\top = \text{SR}(\frac{3}{4}X_h^\top)$ .
Second Backward GEMM (Weight Gradient): $dW_q = \text{GEMM}_{\text{LP}}(G_q^\top, X_q^\top)$ . This computes $dW_q = G_q^\top \times X_q$ (since $X_q^\top$ was transposed), which is the quantized gradient with respect to the weights.
Inverse Hadamard and Mask Correction for $dW$ : $dW = \frac{16}{9} H_g^{-1}(dW_q \odot M_w)$ . Identical logic to step 5, but using the weight clipping mask $M_w$ to zero out gradients for clipped weight values. The output $dW$ has shape $[d_{\text{out}}, d_{\text{in}}]$ and is the gradient used by the optimizer to update the weights.

Cost analysis of the Hadamard transforms. The Quartet algorithm adds two Hadamard transforms (forward) and their inverses (backward) compared to standard training. With block size $g = 32$ , each transform costs $O(g \log g) = 5$ operations per element using the fast Walsh-Hadamard transform (FWHT), or equivalently a small $32 \times 32$ GEMM. For a typical linear layer with hidden dimension $d = 4096$ and batch size $B = 64$ , the GEMM costs $O(B \cdot d^2) = O(64 \cdot 4096^2) \approx 10^9$ operations, while the Hadamard transform costs $O(B \cdot d \cdot g \log g) = O(64 \cdot 4096 \cdot 160) \approx 4 \times 10^7$ operations — roughly 4% overhead. The paper notes that for $g \leq 256$ , the Hadamard cost is "negligible compared with the GEMMs." The critical engineering contribution is ensuring that this overhead is actually negligible in practice by fusing the transform with the quantization, avoiding separate kernel launches and GMEM round-trips that would inflate the effective cost.

Memory footprint. By storing quantized activations ( $X_q$ in MXFP4 at 4 bits per element, plus scales at 8 bits per 32 elements) instead of full-precision activations (16 bits per element), Quartet reduces the activation memory footprint by approximately 3-4×. This is significant because activation memory is often the bottleneck for large-batch or long-sequence training. However, the paper does not report detailed memory measurements, focusing instead on throughput speedups.

The Scaling Law as a Design Tool: How the Ingredients Fit Together

The scaling-law framework (Ingredient 1) provides the analytical glue that connects the individual design choices. Here is how the pieces fit together:

Forward-pass design (QuEST): The $\text{eff}_N$ metric quantifies the capacity loss from forward-pass quantization. The paper's comparison of four forward quantizers (Table 2) shows that QuEST achieves the highest $\text{eff}_N = 0.86$ , meaning a model trained with QuEST-forward FP4 behaves like a model with 86% of the parameters trained in full precision. The correlation between $\text{eff}_N$ and MSE validates the design principle: minimize reconstruction error on the forward pass.
Backward-pass design (stochastic rounding): The $\text{eff}_D$ metric quantifies the training signal degradation from backward-pass quantization. The paper's analysis shows that stochastic rounding achieves $\text{eff}_D \approx 0.60$ (inferred from Table 5/Table 3 data — the paper does not explicitly state this number, but the fitted values imply it), and that this is the best achievable with MXFP4. The projection magnitude misalignment analysis explains why: zero bias in the gradient estimator means the optimizer converges to the correct solution, and the higher variance is amortized over many steps.
Combined optimality (Ingredient 2): With the efficiency factors fitted and the hardware speedups measured, the optimality analysis (Figure 1) determines the specific regimes where FP4:FP4 (Quartet) is better than FP8:FP8 (baseline). The result depends on model size and relative training compute: for larger models (comparable to Llama3-8B or Qwen2.5-7B) and realistic compute budgets, FP4 is optimal because the 1.6× training speedup provides enough additional data (or parameters) to more than compensate for the $\text{eff}_N = 0.86$ and $\text{eff}_D \approx 0.60$ penalties. For very small models or very short training runs, FP8 remains preferable because the speedup cannot offset the efficiency losses.

This is the paper's central architectural insight: Quartet is not claimed to be universally better than FP8 — it is claimed to be optimal in a specific, practically relevant regime identified by the scaling-law analysis. This is a nuanced claim that distinguishes Quartet from prior 4-bit training methods, which typically aimed for "lossless" quantization and failed. Quartet succeeds precisely because it embraces the trade-off — it accepts some accuracy degradation and compensates for it with higher throughput, verified via the scaling-law framework to produce a net improvement under a fixed compute budget.

4. Key Insights and Innovations

Innovation 1: Separating Forward and Backward Precision Requirements as a Design Principle

The paper's most conceptually distinctive contribution is the argument—validated through both theory and experiment—that forward-pass and backward-pass quantization impose fundamentally different, and often opposing, requirements, and that acknowledging this separation is the key to unlocking accurate 4-bit training.

Prior work in low-precision training largely treated quantization as a uniform problem: pick a format, a rounding strategy, and an outlier mitigation method, and apply that same recipe everywhere. Methods like LUQ, LSS, Jetfire, and HALO all operate under this implicit assumption—they design a single quantizer and deploy it symmetrically on the forward and backward passes. The paper's Tables 2 and Figure 2 reveal why this fails below 8 bits: the quantizer that minimizes reconstruction error on the forward pass (QuEST, via RMSE-optimal clipping) maximizes gradient bias on the backward pass, while the quantizer that eliminates gradient bias (stochastic rounding) maximizes reconstruction error. No single quantizer can simultaneously optimize both.

This is not merely a practical observation—it is a diagnostic reframing of the low-precision training problem. The paper introduces a specific metric for what matters on each pass: eff_N (parameter efficiency, linked to forward-pass MSE) for the forward pass, and eff_D (data efficiency, linked to a novel projection magnitude misalignment metric measuring gradient bias) for the backward pass. By showing that these two metrics are not just different but anti-correlated across quantization strategies (Table 2: QuEST has best eff_N but worst misalignment; SR has worst eff_N but zero misalignment), the paper demonstrates that the search for a "universal" 4-bit quantizer is misguided. The correct approach is to compose the best forward quantizer with the best backward quantizer, accepting that they will be different algorithms.

The significance of this framing extends beyond Quartet itself. It provides an analytical language—parameter efficiency vs. data efficiency, MSE vs. misalignment—for reasoning about any future low-precision training method. A researcher proposing a new quantization scheme can now ask: does it improve eff_N, eff_D, or both? Does it shift the error-bias trade-off curve, or merely move along it? The paper's scaling-law framework (Section 4.1) operationalizes these concepts into a quantitative comparison tool that replaces the ad-hoc "train one model and compare loss" evaluations that dominated prior work.

The evidence for this claim is anchored in Figure 2(c), which shows the performance gap between RTN and SR backward quantization as a function of data-to-parameter ratio. For short training runs (low D/N), RTN—which has better cosine similarity with the true gradient (Figure 2a)—outperforms SR. But the relationship crosses over around D/N ≈ 400, and SR pulls increasingly far ahead with more data. This is precisely what the bias-variance decomposition predicts: variance hurts convergence speed (matters more for short runs), while bias hurts asymptotic solution quality (matters more for long runs). Prior work missed this crossover because it tested at a single, typically low D/N regime.

Innovation 2: A Scaling-Law Framework That Makes Quantization Strategy a First-Class Variable

Before this paper, scaling laws for language models (Chinchilla, Kaplan et al.) modeled loss as a function of parameter count N and training tokens D. Quantization was an afterthought—a technique applied post-hoc to reduce inference cost, not something that participated in the scaling relationship itself. This paper elevates quantization precision to the status of a first-class scaling variable by introducing multiplicative efficiency factors eff_N(P_forward) and eff_D(P_backward) into the standard Chinchilla-style law.

This is not a minor parametric extension. It is a conceptual reframing of what quantization does to a model during training. Rather than treating quantized training as a "degraded" version of full-precision training, the framework models it as training a different model—one with an effective parameter count of N · eff_N and an effective data budget of D · eff_D. A model trained in FP4 with 100M parameters on 10B tokens is, in this view, equivalent to a full-precision model with 100M · 0.86 ≈ 86M parameters trained on 10B · 0.60 ≈ 6B tokens. This equivalence is predictive, not merely descriptive: it allows extrapolation from small-scale experiments to larger models and budgets, just as standard scaling laws do.

The framework's power is demonstrated in how it enables the paper's central claim about optimality. By incorporating hardware speedup factors (spfw and sptr) into the scaling law, the paper can compute the net effect of switching precision: the efficiency losses (eff_N, eff_D) are countered by throughput gains that let you train a larger model on more data in the same wall-clock time. Figure 1(b)–(c) visualizes the resulting optimality regions, showing where FP4:FP4 (Quartet) beats FP8:FP8 as a function of model size and compute budget. This is a fundamentally different type of claim than prior work's "our method is within X% of full precision accuracy." It says: under a fixed compute budget, FP4 training produces a better final model than FP8 training, even though FP4 has lower per-step accuracy, because the speedup lets you process more data. This is an economic argument about resource allocation, not an accuracy argument about precision preservation.

The significance of this reframing is amplified by the paper's observation that prior work set an unnecessarily high bar by requiring lower precision to be "lossless"—to match full-precision accuracy point-for-point. The scaling-law analysis reveals that this criterion is too strict: a method can be "lossy" relative to FP8 at a fixed (N, D) and still be "optimal" under a fixed compute budget. By shifting the evaluation criterion from accuracy-matched to compute-matched, the paper opens the door for methods—like Quartet—that embrace a trade-off between per-step quality and throughput, rather than trying to eliminate it.

The evidence for this framework's validity is in the quality of the scaling-law fits themselves (Table 5 / Table 3, and Figure 1a). The fitted efficiency factors predict validation loss across model sizes (30M–200M) and data budgets (25×–800× tokens/parameter) with sufficient accuracy to support the optimality analysis. The paper also tests alternative scaling-law forms (fixed γ=1, fixed β=1, Appendix A.2, Figure 5) and shows that the chosen parametrization provides the best fit, ruling out the possibility that the efficiency factors are merely absorbing model misspecification.

Innovation 3: Projection Magnitude Misalignment as a Diagnostic for Gradient Estimator Quality

The paper introduces a novel diagnostic metric for evaluating backward-pass quantizers: the projection magnitude misalignment, derived from the theoretical analysis of Vargaftik et al. on distributed mean estimation with randomized rotations. This metric quantifies the systematic bias in gradient magnitude introduced by a quantization scheme, separate from the more commonly measured cosine similarity (which captures directional bias).

What makes this conceptually distinctive is that it operationalizes a theoretical concept—gradient unbiasedness—into a measurable quantity that predicts training outcomes. Prior work knew theoretically that unbiased gradient estimators are important for SGD convergence (Alistarh et al., 2017), but there was no practical way to compare the "degree of bias" across different quantizers on real neural network gradients. The projection magnitude misalignment closes this gap: it measures 1 − E[1/S], where S is the rescaling factor needed to make a post-Hadamard RTN-quantized vector unbiased. A quantizer with zero misalignment (E[1/S] = 1) produces gradients that are unbiased in both direction and magnitude; a quantizer with positive misalignment systematically shrinks or expands the gradient magnitudes.

The metric's power is demonstrated in Table 2, where stochastic rounding achieves exactly zero misalignment (0.00 is explicitly stated), while RTN shows measurable bias. This clean separation—SR has zero bias but high MSE, RTN has low MSE but non-zero bias—is what the paper terms the "error-bias trade-off." Figure 2(c) then validates that this trade-off has real training consequences: the misalignment metric predicts which quantizer will perform better for long training runs, and the crossover point (D/N ≈ 400) is where the accumulated bias in RTN's gradients overtakes the per-step variance of SR.

This contribution is fundamental rather than incremental because it gives the field a new tool for designing and evaluating backward-pass quantizers. Instead of running expensive full-training experiments to compare methods, a researcher can compute the projection magnitude misalignment on representative gradient distributions and predict which method will have better data efficiency. The paper implicitly validates this by showing that the ranking of backward-pass methods by eff_D (fitted from full training runs) matches their ranking by misalignment (computed from gradient statistics alone).

The counterintuitive finding that cosine similarity is a poor predictor of long-training performance—RTN has better cosine similarity than SR at all layers (Figure 2a) but worse final loss (Figure 2c)—is a specific example of why this new metric matters. Cosine similarity captures whether the gradient points in the right direction on average; misalignment captures whether the gradient magnitudes are systematically distorted. The paper shows that for data-saturated training, magnitude bias matters more than directional noise, because directional noise averages out over many steps while magnitude bias compounds. This is a non-obvious result that changes how one should evaluate gradient quantizers.

Innovation 4: The "Optimality" Criterion—Compute-Matched Rather Than Accuracy-Matched Evaluation

A significant conceptual move in this paper is the rejection of "lossless" quantization as the evaluation standard in favor of a compute-matched optimality criterion. The paper explicitly contrasts its approach with prior work that "often requires lower precision to suffer no accuracy loss" (Section 4.2, emphasis in original), arguing that this "unnecessarily reduces these trade-offs to simple selection of the fastest lossless precision."

This is a meta-methodological innovation rather than an algorithmic one. It changes how the field should evaluate low-precision training methods, not just what method to use. The core argument is: if FP4 training is 1.6× faster than FP8 training, then under a fixed time budget, you can train on 1.6× more data. Even if FP4 has worse per-token learning efficiency (eff_D < 1), the additional data may produce a net better model. The right comparison is therefore not "FP4 vs. FP8 at the same (N, D)" but "FP4 with 1.6× more D vs. FP8 at the baseline D, under equal wall-clock time."

This is conceptually analogous to the shift that Chinchilla (Hoffmann et al., 2022) brought to pretraining: before Chinchilla, the default was to scale model size aggressively and train on a fixed data budget. Chinchilla showed that, under a fixed compute budget, you get a better model by training a smaller model on more data. Similarly, this paper shows that, under a fixed compute budget, you may get a better model by training in lower precision on more data. Both results are counterintuitive under the previous paradigm (where "bigger model" or "higher precision" was assumed to be better) but obvious once the evaluation criterion shifts from "per-step quality" to "total-budget quality."

The evidence for this claim's practical significance is in Figure 1(c), which uses real RTX 5090 speedup measurements to plot the optimality regions. The green region—where FP4:FP4 is optimal—covers models at the scale of "larger Llama3 or Qwen2.5," which are exactly the kinds of models being deployed and trained today. The paper is making a concrete, falsifiable prediction: if you were to train a Llama3-8B-scale model from scratch with a fixed GPU budget, you would get a better model using Quartet (FP4) than using FP8. This prediction follows directly from the compute-matched evaluation criterion and would not be reachable under the traditional accuracy-matched criterion, since Quartet has strictly lower accuracy than FP8 at a fixed (N, D).

The implications of this shift extend beyond this paper. If the field adopts compute-matched evaluation, then the goal of low-precision training research changes: instead of trying to make 4-bit training match 8-bit accuracy (which may be impossible due to fundamental information-theoretic limits), researchers should try to make the 4-bit speedup × 4-bit efficiency factor product exceed the 8-bit speedup × 8-bit efficiency factor product. This reframes precision selection as a resource allocation problem—analogous to choosing model size or data quantity—rather than a pure accuracy problem.

5. Experimental Analysis

Evaluation Methodology

Dataset. All pre-training experiments use the C4 dataset (Raffel et al., 2020), specifically the train split, with validation loss on the C4 validation set as the primary metric. The choice of C4—a large, diverse web-text corpus—is standard for LLM scaling-law studies and ensures the results reflect general-purpose language modeling rather than domain-specific behavior. The paper does not report using any additional curated or synthetic datasets for training.
Base model(s). All experiments use Llama-2-style Transformer models (Touvron et al., 2023) with non-embedding parameter counts ranging from 30M, 50M, 100M, and 200M for the main scaling-law grid, plus selective scaling up to ~7B parameters to verify training stability at larger scale. The models are trained from scratch (random initialization) on C4—there is no fine-tuning from a pretrained checkpoint. The 30M–200M range is chosen to densely sample the scaling-law parameter space (4 model sizes × 6 data ratios = 24 baseline runs) while keeping the total compute budget tractable (~6,000 GPU-hours on H100s, per Appendix A.6). The 7B run is a stability check, not part of the scaling-law fit, and was conducted specifically because "we found the 7B pre-training experiment specifically to be very DRAM-demanding" (Appendix A.6).
Metrics. The primary metric is validation loss (cross-entropy) on the C4 validation set, reported after training on a fixed token budget. This is the standard metric for scaling-law studies because it is continuous, well-behaved, and correlates with downstream task performance (validated in Figure 4 for HellaSwag, WinoGrande, and ARC-easy). The scaling-law framework additionally produces two derived metrics: parameter efficiency eff_N (fitted from the scaling law, representing the multiplicative reduction in effective model capacity due to forward-pass quantization) and data efficiency eff_D (fitted from the scaling law, representing the multiplicative reduction in effective training data due to backward-pass quantization). For the speedup analysis, the metric is kernel execution time measured on an NVIDIA RTX 5090 GPU, reported as speedup factors relative to FP8 and BF16 baselines.
Baselines. The paper compares Quartet against four prior end-to-end 4-bit training methods, all evaluated in the same Llama/C4 pre-training setup:
- LUQ (Chmiel et al., 2023): Logarithmic Unbiased Quantization, evaluated in both INT4 and FP4 variants. Uses unbiased quantization pairing 4-bit weights/activations with stochastic underflow and logarithmic stochastic rounding.
- HALO (Ashkboos et al., 2025): Hadamard-assisted low-precision optimization, evaluated at the most accurate "HALO-2" setting in FP4. Uses Hadamard rotations for outlier mitigation.
- Jetfire (Xi et al., 2024): Originally designed for INT8, adapted to FP4 by the paper's authors. Uses per-block quantization with 32×32 blocks.
- LSS / INT4-Transformers (Xi et al., 2023): Combines Hadamard-based forward pass with leverage-score sampled INT4 gradients.
All baselines reuse identical optimizer settings, learning rate schedules, and hyperparameters as Quartet (Appendix A.1). The baselines are evaluated at 30M parameters across multiple D/N ratios. For the speedup benchmarks, the baselines are CUTLASS MXFP8 kernels (for the FP8 comparison) and PyTorch BF16 kernels (for the BF16 comparison), both using Blackwell-optimized implementations.
Generation budget / compute accounting. Compute for accuracy experiments is measured in training tokens (the total number of tokens processed during pre-training, equal to D in the scaling law). Runs are conducted across a grid of model sizes N ∈ [30M, 50M, 100M, 200M] and data-to-parameter ratios D/N ∈ [25, 50, 100, 200, 400, 800]. The compute budget in the scaling-law analysis is N · D (total FLOPs proportional to parameters × tokens, following standard practice). For the speedup experiments, compute is measured as wall-clock kernel execution time on an RTX 5090, using a batch size of 64 and sequence length of 512, with speedups computed as the ratio of baseline kernel time to Quartet kernel time for the same tensor shapes. For the end-to-end prefill speedup (Appendix A.4), a 7B model is evaluated at sequence length 256 across varying batch sizes.
Cross-validation / statistical protocol. The scaling law is fitted using a two-stage procedure (Appendix A.2). Stage 1 fits the baseline parameters A, B, α, β, γ, E on BF16 (full-precision) runs across all 24 (N, D) grid points using Huber loss with δ = 10^{-4} on log-loss. Stage 2 fits eff_N and eff_D per quantization method with the baseline parameters held fixed, using the same Huber loss. The paper tests three alternative scaling-law forms (fixed γ = 1, fixed β = 1, and the chosen "Busbridge et al., 2025" form) and reports fits for all three in Figure 5 (Appendix A.2), confirming that the chosen parametrization yields the best fit. There is no explicit cross-validation over data splits for the scaling-law fits—all runs contribute to the fit. For the post-training quantization comparison (Appendix A.5), 128 samples from the C4 training set serve as the calibration set for GPTQ in the QuaRot baseline. Hardware speedup measurements do not involve statistical protocols beyond standard benchmarking (single GPU, repeated kernel executions with reported averages).

Main Quantitative Results

Scaling-Law Fit and Efficiency Coefficients

The paper's foundational result is the scaling-law fit itself, which establishes the quantitative framework for all subsequent comparisons. The baseline (BF16) fit produces the coefficients in Table 6 (Appendix A.2), from which eff_N and eff_D are derived for each quantization method. The key numbers are in Table 2 of the main text:

Forward-pass only (isolating eff_N):

QuEST (Quartet's forward quantizer): eff_N = 0.86 — the highest among all forward quantizers tested.
LSQ: eff_N = 0.83
RTN with AbsMax: eff_N = 0.81
SR with AbsMax: eff_N = 0.64 — substantially worse, supporting the principle that stochastic rounding is inappropriate for the forward pass.

The ranking by eff_N correlates perfectly with the ranking by MSE on Gaussian data (Table 2, first data column): QuEST achieves the lowest reconstruction error and the highest parameter efficiency. This validates the paper's central thesis that forward-pass quality is about minimizing MSE.

Backward-pass only (isolating eff_D):

SR with AbsMax (Quartet's backward quantizer): achieves the highest eff_D among backward quantizers, correlated with zero projection magnitude misalignment (0.00 in Table 2).
RTN with AbsMax: shows measurable misalignment (non-zero entry in Table 2) and correspondingly lower eff_D.

The crossover behavior in Figure 2(c) shows that RTN (better cosine similarity) actually outperforms SR for D/N < ~400, but SR pulls ahead for longer training runs — precisely the regime targeted by LLM pre-training. The paper does not state the exact numeric eff_D values for these isolated backward-only fits, but the joint end-to-end fits in Table 5/Table 3 imply them.

End-to-end 4-bit comparisons (both passes quantized, Table 5/Table 3): Table 3 (labeled as "Table 5" in the text but appearing as Table 3 in the rendering) reports validation loss on C4 for 30M-parameter Llama models across D/N ratios from 100× to 800×:

Method	D/N = 100	D/N = 200	D/N = 400	D/N = 800
Quartet (FP4)	Lowest loss	Lowest loss	Lowest loss	Lowest loss
LUQ-INT4	+10% relative loss vs. Quartet	larger gap	larger gap	larger gap
LUQ-FP4	worse than LUQ-INT4	—	—	—
LSS-INT4	competitive	diverges	NaN (crashed)	NaN (crashed)
Jetfire-FP4	"large degradation"	unstable	—	—
HALO-FP4	"large degradation"	unstable	—	—

At D/N = 100×, Quartet achieves approximately 10% relative loss reduction over LUQ-INT4 (the strongest prior method). The gap widens as D/N increases, consistent with Quartet's higher eff_D (better data efficiency via unbiased gradients) becoming more important in the data-saturated regime. LSS is competitive only up to D/N ≈ 50×, beyond which it diverges — a critical failure mode that the paper attributes to compatibility with prior observations (Fishman et al., 2024) about LSS's instability in long training runs. Jetfire and HALO show "large degradation and are unstable when ported to FP4," which the paper attributes to the narrower dynamic range of FP4 being insufficient for their block-wise quantization without the additional safeguards (QuEST clipping, stochastic rounding) that Quartet provides.

Scaling-law efficiency coefficients for end-to-end methods (inferred from the fits underlying Table 3): Quartet achieves both higher eff_N and higher eff_D than all prior methods, making it strictly dominant under the paper's comparison framework — no method beats Quartet on either metric. The paper states that Quartet's fitted coefficients suggest it "requires, roughly, 15% fewer parameters and 5x less data to reach the same loss" as LUQ-INT4 (Section 5, Accuracy Discussion paragraph). This is a composite statement derived from both efficiency factors: the 15% fewer parameters follows from eff_N(Quartet) / eff_N(LUQ-INT4) ≈ 1.15, and the 5× less data follows from eff_D(Quartet) / eff_D(LUQ-INT4) ≈ 5.0.

Optimality Region Analysis (Figure 1)

Figure 1 translates the scaling-law efficiency coefficients into actionable guidance about when FP4 training is "optimal":

Figure 1(a): Shows the scaling-law fit across four forward:backward precision combinations (FP8:FP8, FP8:FP4, FP4:FP8, FP4:FP4) for a range of model sizes and training budgets. The fit quality is visually tight, with the data points clustering closely around the fitted curves. FP8:FP8 (orange) achieves the lowest loss at any fixed (N, D), as expected since it has no efficiency penalties. FP4:FP4 (red) is consistently higher at fixed (N, D) but the gap narrows for larger models and larger data budgets — an important observation that motivates the optimality analysis.
Figure 1(b): Using the BOPS (bit-wise operations) speedup model, the plot shows optimality regions as a function of model size (Y-axis) and relative training compute budget (X-axis, normalized to FP8 baseline). The red region is where FP8:FP8 is optimal; the green region is where FP4:FP4 (Quartet) is optimal. Under BOPS assumptions (FP4 is ideally 2× faster than FP8), the green region covers moderate-to-large models at moderate-to-large compute budgets.
Figure 1(c): Replacing BOPS speedups with actual RTX 5090 measurements (which show FP4 forward speedups up to ~2× and overall training speedups of ~1.6×), the optimality regions shift. The paper notes that "popular models such as larger Llama3 or Qwen2.5 models fall into the FP4:FP4 optimality region" — these are models in the 7B–70B parameter range, which are representative of current production-scale LLMs. This is the paper's headline practical claim: if you were training a Llama3-8B-scale model today on Blackwell hardware, Quartet FP4 would produce a better model than FP8 under a fixed compute budget.

The mechanism is straightforward from the scaling law: the FP4 speedup (sptr ≈ 1.6×) multiplies the effective data budget, and the additional data compensates for the eff_D < 1 penalty. For large enough models and budgets, the compensation is net positive. The crossover point depends on the specific eff_N and eff_D values — Quartet's high efficiency factors (especially eff_N = 0.86) mean the crossover happens at smaller model sizes and budgets than it would for a less efficient 4-bit method.

Accuracy on Downstream Benchmarks (Figure 4)

To validate that C4 validation loss correlates with practically meaningful performance, the paper trains Llama models at 100M, 200M, 430M, 800M, and 1.6B parameters with both Quartet and FP8 at D/N = 100×, then evaluates on few-shot benchmarks:

HellaSwag (Zellers et al., 2019): commonsense reasoning
WinoGrande (Sakaguchi et al., 2019): pronoun resolution
ARC-easy (Clark et al., 2018): science question answering

Figure 4 shows that downstream performance tracks C4 validation loss closely across model sizes for both Quartet and FP8. The paper states these evaluations "are consistent with C4 validation loss for larger models." The implication is that the scaling-law optimality conclusions — which are derived entirely from validation loss — translate to downstream task performance. No statistical tests are reported for the correlation.

Training Stability at Scale (Figure 3c)

Figure 3(c) shows the training loss curve for a 7B-parameter Llama model trained with Quartet over 200B tokens, compared to an FP8 baseline. The curves track each other closely throughout training, with no signs of divergence, instability, or loss spikes. This is a critical result because:

It validates that Quartet's design principles (QuEST forward, stochastic rounding backward, Hadamard transforms) scale two orders of magnitude beyond the 30M–200M models used for the scaling-law grid.
It addresses a common failure mode of low-precision training methods: instability at scale that does not appear in small-model experiments (as seen with LSS, Jetfire, and HALO collapsing at FP4 even at 30M).
The fact that the 7B Quartet and FP8 loss curves remain visually parallel suggests that the eff_N and eff_D efficiency factors are approximately constant across scale — a necessary condition for the scaling-law extrapolation to be valid.

The 7B experiment is a single run at one data scale, not a full scaling-law grid, so it does not independently validate the efficiency factors at 7B. However, it provides strong evidence against the hypothesis that Quartet's good results at 30M–200M are an artifact of small scale.

GPU Kernel Speedup Results (Figure 3a, 3b)

Figures 3(a) and 3(b) present the measured kernel speedups of Quartet's MXFP4 implementation relative to FP8 and BF16 baselines on an NVIDIA RTX 5090. The measurements are taken across different linear layer shapes (corresponding to different model sizes) at batch size 64 and sequence length 512:

Forward pass speedups (Figure 3a, left):

Ranges from approximately 1.2× to 2.0× over FP8 across model sizes, with the speedup increasing with arithmetic intensity (larger models saturate the GPU better) and plateauing near 2.0× for the largest layers.
Ranges from approximately 2.5× to 4.0× over BF16, again increasing with model size and plateauing near 4.0×.
The speedup is more pronounced on the forward pass because the forward GEMMs are less complex than backward GEMMs, and the quantization overhead (Hadamard + scaling) represents a smaller fraction of total forward-pass time.

Backward pass speedups (Figure 3b, middle):

Up to ~1.5× over FP8 for the largest layers.
Up to ~2.6× over BF16.
The backward speedup is lower than forward because: (1) the backward pass involves two GEMMs per linear layer (activation gradient and weight gradient) versus one on the forward pass, increasing the relative overhead of the Hadamard transforms; (2) the stochastic rounding and rescaling steps add additional operations; (3) the backward GEMMs have different memory access patterns that may not benefit as much from FP4's reduced memory bandwidth.

Overall training speedup (Figure 3 text):

The paper reports end-to-end training speedup of approximately 1.6× over FP8 and 2.9× over BF16, computed as the harmonic mean of forward and backward speedups weighted by their compute fractions (1/3 forward, 2/3 backward). Specifically: 1 / ((1/3)/2.0 + (2/3)/1.5) ≈ 1.6× for FP8, and similarly for BF16.

These speedups are measured at the kernel level (individual linear layer operations), not end-to-end training throughput including optimizer steps, embedding lookups, attention mechanisms, or communication overhead. The paper acknowledges this scope limitation implicitly by reporting "kernel speedups" rather than full training speedups. The end-to-end prefill speedup in Appendix A.4 (Figure 7) provides a closer-to-realistic measurement: for inference prefill on a 7B model at sequence length 256, MXFP4 achieves up to 1.41× over FP8 at batch size 128, where it plateaus.

Performance Breakdown of the Kernel Implementation (Figure 6, Appendix A.3)

Figure 6 breaks down the runtime composition of the Quartet forward pass for three representative linear layer shapes from a Llama-7B model (batch size 64, sequence length 512). The breakdown divides time into three components: matrix multiplication (the GEMM itself), quantization-related operations (Hadamard transform, clipping factor search, FP32-to-FP4 conversion, scale factor computation, mask generation), and rearrangement of scaling factors (organizing scales into the layout required by the tcgen05.mma instruction).

Three kernel configurations are compared:

Basic 32×32 threadblock tile (left subplot): Quantization operations consume a significant visible fraction of total runtime — the overhead is non-negligible and would erode the speedup from faster GEMMs.
128×32 threadblock tile (center subplot): Quantization fraction shrinks substantially. Larger tiles mean more active warps per block, higher arithmetic intensity, and better latency hiding for the quantization operations.
Custom Triton kernel (right subplot): Further reduces the rearrangement overhead, so that matrix multiplication dominates total runtime. This is the target regime where quantization overhead is nearly fully amortized.

The paper states that "increasing the threadblock tile size leads to more active warps per block, enhancing arithmetic intensity and enabling better latency hiding" (Appendix A.3). The Triton backend "exhibits similar trends, with rearrangement overheads further reduced and matrix multiplication dominating the total runtime." This breakdown validates a key claim from Section 4.4: that a naive implementation of the Quartet pipeline would be slower than FP16, and that the specific engineering choices (fused kernel, large tile sizes, optimized rearrangement) are necessary to realize the theoretical speedup.

Post-Training Quantization Comparison (Appendix A.5)

Table 7 compares Quartet against post-training quantization (PTQ) using QuaRot with GPTQ weight optimization, both in MXFP4, on the 7B model. The metric is perplexity on the C4 validation set:

Quartet: Lower perplexity than PTQ by 0.42 points (the exact numbers are in Table 7 — the paper does not restate them in the main text, only the difference is quoted: "Quartet achieves a 0.42-point lower perplexity compared to QuaRot when applied to the same model").
QuaRot + GPTQ (PTQ): The baseline uses 128 C4 training samples as the calibration set for GPTQ.

This comparison establishes that Quartet's quantization-aware training produces a model that is not only trainable in low precision but also achieves better inference-time quality than applying the best available PTQ technique to an FP8-trained model. The paper notes that "Quartet is also more efficient than standard QAT methods, as it quantizes both forward and backward passes." This is a secondary benefit — the model trained with Quartet is already in MXFP4 format at the end of training, with no additional PTQ step needed for deployment.

End-to-End Prefill Speedups (Figure 7, Appendix A.4)

Figure 7 shows inference prefill speedup of MXFP4 over FP8 as a function of batch size for the 7B model at sequence length 256 on an RTX 5090. The speedup increases progressively from approximately 1.05× at batch size 1 to approximately 1.41× at batch size 128, where it plateaus. This is the only end-to-end (rather than kernel-level) speedup measurement in the paper, and it is inference-only, not training. The paper does not provide end-to-end training throughput measurements including optimizer overhead, gradient communication, or data loading — a notable gap given that the paper's primary claim is about training efficiency.

Ablation Studies and Robustness Checks

Forward-pass quantizer comparison (Table 2, first data rows): Four quantizers (SR, RTN, LSQ, QuEST) are compared via the fitted eff_N parameter and MSE on Gaussian data. QuEST achieves the highest eff_N = 0.86, LSQ achieves 0.83, RTN achieves 0.81, and SR achieves 0.64. The ranking by eff_N perfectly mirrors the ranking by MSE — lower reconstruction error directly translates to higher effective parameter count. This validates the design choice of QuEST over alternatives and empirically confirms the principle that forward-pass quality is about error minimization.

Backward-pass quantizer comparison via misalignment (Table 2, second data rows): Stochastic rounding achieves exactly zero projection magnitude misalignment (0.00), while RTN shows measurable bias. The paper connects this to eff_D through Figure 2(c): the quantizer with zero bias (SR) outperforms the low-MSE quantizer (RTN) for D/N > 400, establishing that the misalignment metric predicts which method will have better data efficiency. The crossover at D/N ≈ 400 is an empirical finding that defines the regime where unbiasedness matters more than per-step gradient fidelity.

Scaling-law form alternatives (Figure 5, Appendix A.2): Three scaling-law parametrizations are compared: the chosen form (from Busbridge et al., 2025, with free γ), the Chinchilla form (fixed γ = 1), and the Kaplan et al. form (fixed β = 1). Figure 5 shows the fits and errors for all three. The chosen parametrization provides the best fit across the (N, D) grid, ruling out the possibility that the efficiency factors are artifacts of model misspecification. This is important because if the base scaling law form were wrong, the fitted eff_N and eff_D would be absorbing structural error rather than measuring real quantization effects.

Training stability at scale (Figure 3c): A 7B-parameter Llama model trained with Quartet over 200B tokens shows loss curves that track the FP8 baseline closely, with no divergence or instability. This is an ablation against the hypothesis that Quartet's small-scale results are fragile or that the method develops instability at larger model sizes — a genuine concern given that LSS, Jetfire, and HALO all degrade even at 30M in FP4. The single 7B run is not a full scaling-law validation, but it provides strong evidence of robustness across two orders of magnitude in parameter count.

Correspondence between validation loss and downstream benchmarks (Figure 4): Models trained with Quartet and FP8 at 100M–1.6B parameters (D/N = 100) are evaluated on HellaSwag, WinoGrande, and ARC-easy. Downstream performance tracks C4 validation loss closely for both methods, supporting the claim that validation loss improvements from Quartet translate to practical task improvements. This is not an ablation of Quartet itself but a validation of the evaluation methodology — if validation loss and downstream accuracy diverged, the scaling-law analysis would be measuring a proxy that does not reflect real model quality.

Post-training quantization vs. Quartet (Table 7, Appendix A.5): The 7B model trained with Quartet achieves 0.42-point lower perplexity than applying QuaRot + GPTQ (the strongest available PTQ method) to the same architecture in MXFP4. This establishes that Quartet's QAT approach is not just competitive with PTQ but superior for deployment-time quality, while also providing the training-time speedup that PTQ cannot offer.

Kernel configuration comparison (Figure 6): Three configurations of the quantization kernel (32×32 tile, 128×32 tile, Triton-optimized) show progressively lower quantization overhead, validating the engineering claim that careful kernel design is necessary to amortize the cost of the Hadamard transforms and scale factor manipulation. The rightmost configuration achieves the ideal regime where GEMM dominates runtime.

What is NOT ablated: Several potentially informative ablations are absent:

No ablation of the 3/4 rescaling factor: The paper does not test different rescaling values or ablate the rescaling entirely on the backward pass. The choice of 3/4 is stated without justification or sensitivity analysis.
No ablation of the Hadamard block size: The block size is fixed at 32 (matching the MXFP4 group size). The paper does not test alternative block sizes or evaluate how sensitive the results are to this choice.
No ablation of stochastic rounding without Hadamard on the backward pass: The paper always applies stochastic rounding after randomized Hadamard. The individual contributions of SR alone vs. Hadamard alone vs. the combination are not separated for the backward pass.
No alternative optimizer or learning rate sensitivity: All experiments use AdamW with fixed hyperparameters inherited from the FP8 baseline. The paper does not test whether FP4 training benefits from different optimization settings (e.g., higher learning rate to compensate for gradient variance, different weight decay, different warmup).
No alternative FP4 formats tested: Only MXFP4 (E2M1) is evaluated. NVFP4, which uses different block sizes (16 vs. 32) and scale formats (E4M3 vs. E8M0), is mentioned as supported by Blackwell hardware but is not benchmarked — the paper notes that "MXFP4 is the only microscaling format with support for all required layouts for both forward and backward multiplications in low precision on Blackwell."

Critical Assessment

Do the Experiments Support the Claim That Quartet Achieves "Optimal" FP4 Training?

The paper's central quantitative claim — that Quartet enables FP4 to be "optimal" on the accuracy-efficiency trade-off — is supported specifically for the Llama/C4/RTX 5090 combination tested, but the scope of the evidence is narrower than the abstract's framing suggests.

What IS demonstrated: The scaling-law analysis (Figure 1, Table 3) shows that Quartet outperforms all prior 4-bit training methods in terms of both eff_N and eff_D on C4 validation loss for Llama models at 30M parameters, with increasing margins as the data-to-parameter ratio grows. The 7B training run (Figure 3c) demonstrates that the method does not collapse at larger scale, and the downstream benchmark correlation (Figure 4) shows that validation loss improvements carry over to practical metrics. The kernel speedup measurements (Figure 3a–b) show that Quartet's implementation achieves real throughput gains on Blackwell hardware.

What is NOT demonstrated: The "optimality" claim — that FP4 training with Quartet produces a better model than FP8 under equal compute — is a prediction of the scaling-law model, not a directly measured outcome. The scaling law is fitted on 30M–200M models and extrapolated to the 7B–70B regime where the optimality regions are claimed. No experiment directly compares a Quartet-trained model against an FP8-trained model under equal wall-clock time at the 7B+ scale — the 7B run in Figure 3(c) shows loss curves at equal tokens, not equal time, so it does not test the throughput-compensation hypothesis. The optimality regions in Figure 1(b)–(c) are derived by plugging the fitted efficiency factors and measured speedups into the scaling law; they are not validated by training a large model with Quartet and confirming it outperforms an FP8 baseline under a fixed time budget.

This is a significant gap between the evidence and the claim. The scaling law may not extrapolate perfectly to 7B+ models — the 7B run provides one data point suggesting stability, but stability ≠ identical eff_N and eff_D at scale. Efficiency factors could change with model size (e.g., if activation distributions shift, affecting QuEST's clipping accuracy), and the paper provides no evidence ruling this out.

Do the Experiments Genuinely Show Quartet Beats Prior Methods, or Just That Prior Methods Are Poorly Adapted to FP4?

A careful reading of the baselines reveals that several of them were not designed for FP4:

Jetfire was designed for INT8 (Xi et al., 2024). The paper "adapted to FP4 for our setup" (Section 5, Accuracy Comparisons), but the adaptation details are not described. The reported "large degradation and instability" may reflect the adaptation methodology rather than Jetfire's core approach.
HALO was designed for INT8 (Ashkboos et al., 2025). Again, "evaluated in FP4 at their most accurate HALO-2 setting," but HALO-2 was not designed or tuned for 4-bit precision — its outlier mitigation strategy may not be sufficient for FP4's extreme dynamic range constraints.
LSS / INT4-Transformers (Xi et al., 2023) was designed for INT4, which has different numerical properties than FP4 (uniform vs. non-uniform grid spacing, different dynamic range). The adaptation is more direct than for Jetfire/HALO, but still not native.
LUQ (Chmiel et al., 2023) was developed for ResNets on ImageNet, not Transformers on text. Its evaluation on LLM pre-training is a new application, and the paper is the first to test it in this setting.

The only method that was originally designed for FP4 in Transformers is Quartet itself. This means the comparison is partly between a method purpose-built for the target format and hardware and methods that were retrofitted. The paper acknowledges this implicitly by noting that Jetfire and HALO were designed for INT8, but it does not discuss how much of Quartet's advantage comes from format-specific design vs. genuinely superior principles. A fairer comparison would include FP4-native adaptations of Jetfire and HALO, where the block sizes, clipping strategies, and rounding methods are re-optimized for FP4's specific grid and dynamic range.

The Single Model Family and Dataset Limitation

All accuracy experiments use Llama-2 architecture on C4, evaluated on a single NVIDIA RTX 5090 GPU. The paper's claims about optimality (Figure 1c) reference "Llama3 or Qwen2.5 models," but neither Llama3 nor Qwen2.5 is actually trained in the experiments. Llama-2 and Llama-3 have different architectural details (e.g., grouped-query attention in Llama-3, different normalization placement) that could affect quantization error propagation. The 7B stability run uses the Llama-2 architecture, so extrapolation to Llama-3 is architectural extrapolation as well as scale extrapolation.

C4 is a single dataset with specific statistical properties. Different pre-training corpora (The Pile, RefinedWeb, multilingual data) may have different token distributions, outlier characteristics, or gradient statistics that affect quantization behavior. The paper provides no evidence that Quartet's efficiency factors generalize across datasets.

What Is Missing: End-to-End Training Throughput Measurements

The paper reports kernel-level speedups (individual linear layer GEMMs) and inference prefill speedups (Appendix A.4, Figure 7), but does not report end-to-end training throughput including all components: attention mechanisms (which may or may not be quantized — the paper focuses on linear layers), embedding lookups, layer normalization, residual connections, optimizer steps (AdamW updates in FP32), gradient communication (for multi-GPU training), and data loading. The 1.6× overall training speedup is a derived number (harmonic mean of forward and backward kernel speedups), not a measured end-to-end training speedup.

In practice, non-GEMM operations can consume 20–40% of training time depending on model size, batch size, and hardware configuration. If Quartet only accelerates the GEMM portions, the end-to-end speedup would be lower than 1.6× — potentially significantly lower for small models or large cluster configurations where communication dominates. The paper's decision to only report kernel-level speedups is understandable given the focus on the algorithmic contribution, but it means the headline "optimal training" claim overstates what has been demonstrated.

Statistical Significance and Reproducibility Concerns

The scaling-law fits are based on single training runs per grid point — there are no repeated runs with different random seeds to estimate variance in the final validation loss. This matters because LLM pre-training loss can vary by 0.01–0.03 depending on random seed, data order, and initialization, which is comparable to the loss differences between some quantization methods at lower D/N ratios. The fitted eff_N and eff_D values are point estimates without confidence intervals; small perturbations in the validation loss of a few grid points could shift the fitted efficiency factors. The paper does not address this source of uncertainty.

The 7B training run is a single experiment — there is no replicate, no sweep over learning rates, and no comparison of multiple random seeds. The claim that Quartet is "stable" at 7B is based on one training trajectory. While a divergent or unstable trajectory would be conclusive evidence against Quartet, a single non-divergent trajectory is weaker evidence for stability than multiple independent runs would be.

The Missing Combination: Quartet with Higher-Precision Attention

The paper quantizes "all three matrix multiplications of a linear layer" in FP4, but Transformers also include attention score computations (Q × K^T and attention_weights × V). The paper does not specify whether attention MatMuls are quantized or left in higher precision. If attention is left in FP8 or FP16, the effective speedup of the full training pipeline would be lower than the linear-layer kernel speedup reported, because attention operations are a non-trivial fraction of total FLOPs (especially for long sequences). This is a specification gap that affects the real-world applicability of the reported speedups.

The Absent Ablation: How Much Does Each Ingredient Contribute?

The paper presents Quartet as a package of four ingredients (scaling-law framework, mixed-precision optimality analysis, QuEST forward + SR backward, optimized GPU kernels), but only Ingredient 3 (forward/backward quantizer choice) and Ingredient 4 (kernel implementation) are directly ablated. The contribution of Ingredient 1 (the scaling-law framework itself) to the quality of Quartet is never tested — would a simpler evaluation methodology (e.g., grid search over hyperparameters at a single model size) have arrived at the same design? The contribution of Ingredient 2 (the mixed-precision optimality analysis) to the design of Quartet is also not isolated — did the optimality analysis guide design choices that would not have been made otherwise, or is it a post-hoc justification for empirically good choices?

The most informative missing ablation is: What happens if you use QuEST on both forward and backward passes? The paper's theoretical framework predicts this would have high eff_N but low eff_D due to gradient bias. Demonstrating this degradation experimentally would close the loop between the misalignment theory and observed training outcomes. Similarly, what happens with SR on both passes? This would test whether the error-bias trade-off is truly necessary or whether the forward pass could also tolerate unbiased high-variance quantization if trained long enough.

Summary of Evidence-to-Claim Gaps

"Optimal" in the abstract: The optimality regions in Figure 1 are model-derived, not measured. No experiment shows a Quartet-trained model beating an FP8-trained model at equal wall-clock time at production scale (7B+). The scaling law extrapolation to 7B–70B is plausible but unvalidated.
"Up to 1.6× overall training speedup": This is a derived number from kernel-level measurements. End-to-end training throughput including all operations is not reported. The real end-to-end speedup may be lower.
Superiority over prior methods: Quartet's advantage is clear against FP4-adapted versions of INT8 methods, but the adaptation quality is unknown. The gap may narrow against FP4-native implementations of competing approaches.
Generalizability: The results are from one model family (Llama-2), one dataset (C4), one GPU architecture (Blackwell RTX 5090), and one FP4 format (MXFP4 with E2M1). The paper's framework provides a methodology for testing generalizability but does not actually perform the generalization experiments.

These gaps do not invalidate the paper's contributions — the scaling-law framework is genuinely novel, the decomposition of forward/backward requirements is well-supported, and the kernel engineering is impressive. But they do mean that the paper's strongest framing ("Native FP4 Training Can Be Optimal") should be understood as a conditional statement — optimal under the assumptions of the scaling-law extrapolation, for the specific architecture/format/hardware tested, and pending end-to-end training throughput validation — rather than a demonstrated fact.

6. Limitations and Trade-offs

The Scaling-Law Optimality Claim Is Extrapolated, Not Measured, at Production Scale

The assumption or constraint. The paper's central claim — that FP4 training with Quartet can be "optimal" relative to FP8 under a fixed compute budget — is a prediction of the fitted scaling law, not a directly measured outcome at the model sizes where optimality matters. The scaling law (Equation 1) is fitted on models with 30M–200M non-embedding parameters across D/N ratios from 25× to 800×. The optimality regions in Figure 1(b)–(c) — which show where FP4:FP4 beats FP8:FP8 — are then extrapolated into the 7B–70B range by plugging fitted efficiency factors and measured speedups into the scaling-law formula. The authors are transparent about the framework's logic: the loss is evaluated as Loss(N_max · spfw, D_max · sptr/spfw, P_fwd, P_bwd) using the scaling law. But no experiment trains a 7B+ model with Quartet under a wall-clock-matched compute budget and compares it directly against an FP8-trained model of the same scale to verify the prediction.

The consequence. If the efficiency factors eff_N and eff_D change with model scale — for instance, because activation or gradient distributions at 7B differ from those at 200M, affecting QuEST's clipping accuracy or stochastic rounding's bias properties — then the extrapolated optimality regions are incorrect. The scaling law could systematically overestimate or underestimate the crossover point where FP4 becomes optimal. A practitioner deciding whether to adopt Quartet for training a production-scale model (e.g., Llama3-8B or Qwen2.5-7B) must rely on an unvalidated extrapolation, not on direct evidence that Quartet produces a better model than FP8 at equal wall-clock time at that scale.

What evidence exists in the paper. A single 7B-parameter Llama model is trained with Quartet over 200B tokens, and its loss curve is plotted alongside an FP8 baseline in Figure 3(c). The curves track closely, demonstrating stability — the model does not diverge or exhibit loss spikes. But stability is not the same as validating the efficiency factors at scale. The 7B run compares Quartet and FP8 at equal tokens, not equal wall-clock time. To test the optimality claim, one would need: (1) train Quartet at 7B with a 1.6× larger token budget (mirroring the throughput advantage) under the same wall-clock time as FP8 at 7B; (2) confirm that the Quartet model achieves lower validation loss. This experiment is not performed. The paper does not even report the final validation loss of the 7B Quartet run — only the loss curve shape is shown. No eff_N or eff_D values are fitted at 7B.

Mitigation status. The paper does not address this gap directly. The 7B run is presented as a stability demonstration, not an optimality validation. The authors do not claim to have verified the optimality prediction at scale; the framing in Section 4.2 is explicitly conditional ("implying that training similar models in FP4 might have been optimal," emphasis added). But this conditionality is easily lost in the paper's stronger statements (the title, the abstract's claim that Quartet "establishes that MXFP4 can indeed provide 'optimal' training in practice"). The limitation is partially structural: fitting scaling laws at 7B+ would require training multiple large models at multiple data budgets — a compute expense (~10,000+ GPU-hours per run) that may exceed the paper's resources. The authors flag this implicitly by noting the 7B run was "very DRAM-demanding" (Appendix A.6) and required specific hardware. A full validation at scale remains future work.

End-to-End Training Throughput Is Not Measured; Reported Speedups Are Kernel-Level Only

The assumption or constraint. All reported speedup numbers in the paper's main claims — "up to ~1.6× overall training speedup and ~2× forward-pass speedup over highly-optimized FP8 kernels" — are measured at the individual linear layer kernel level on an NVIDIA RTX 5090, not on an end-to-end training pipeline. The measurements (Figures 3a–b, Appendix A.3) benchmark the Quartet quantized GEMM kernels (forward, backward for activations, backward for weights) against CUTLASS MXFP8 and PyTorch BF16 kernels for identically shaped matrix multiplications. The "overall training speedup of up to around 1.6×" is a derived number — the harmonic mean of forward and backward kernel speedups weighted by their compute fractions (1/3 forward, 2/3 backward) — not a measurement of an actual training loop.

The consequence. In a real training pipeline, non-GEMM operations consume a substantial fraction of wall-clock time. These include: the attention mechanism (Q×K^T and attention_weights×V matrix multiplications, which may or may not be quantized — the paper does not specify); softmax and layer normalization (typically in FP32); residual connections and activation functions; the AdamW optimizer step (FP32 weight updates, momentum and variance buffers); embedding lookups and the final projection to vocabulary; gradient synchronization across GPUs in distributed training (AllReduce operations); and data loading and preprocessing. If these non-GEMM components consume, say, 30% of total training time and are not accelerated by FP4 quantization, then a 1.6× GEMM speedup translates to at best a ~1.35× end-to-end training speedup (Amdahl's Law: 1 / (0.7/1.6 + 0.3/1.0) ≈ 1.35). The gap between kernel-level and end-to-end speedup grows as the fraction of non-GEMM time increases — which is more likely at smaller model sizes, shorter sequence lengths, or in distributed settings where communication dominates.

Additionally, the kernel benchmarks use a single batch size (64) and sequence length (512). Throughput characteristics may differ at the much larger batch sizes and longer sequences typical of production LLM training.

What evidence exists in the paper. The only end-to-end measurement provided is an inference prefill speedup (Figure 7, Appendix A.4), showing MXFP4 achieves up to 1.41× over FP8 at batch size 128 on a 7B model at sequence length 256. This is inference-only (no backward pass, no optimizer), and the speedup is lower than the kernel-level forward-pass speedup of ~2.0×, confirming that non-GEMM overhead erodes the kernel advantage in a more realistic setting. No end-to-end training throughput measurement (including backward pass, optimizer, and communication) is reported anywhere in the paper.

Mitigation status. The paper does not acknowledge this gap explicitly. The speedup results section (Section 5) describes the measurements as "kernel speedups" but then reports a derived "overall training speedup" that could mislead readers into believing it is measured end-to-end. The paper does provide the performance breakdown in Figure 6 (Appendix A.3), which shows the fraction of kernel time spent on quantization overhead — but this still operates within a single GEMM kernel, not a full training step. The open-source code release ("QuTLASS") would allow practitioners to measure end-to-end throughput themselves, but the paper does not claim to have done so. A rigorous training throughput comparison on a multi-GPU node (e.g., 8×H100 or 8×B200) remains necessary to validate the headline speedup claims.

The Method Is Designed for a Single Hardware Architecture, Format, and Model Family

The assumption or constraint. Quartet's design and implementation are tightly coupled to NVIDIA's Blackwell architecture and specifically to the MXFP4 (E2M1) microscaling format. The GPU kernel relies on Blackwell-specific PTX instructions (tcgen05.mma for block-scaled matrix multiply, FP32-to-FP4 conversion intrinsics), shared memory sizes, and memory hierarchy characteristics. The algorithm choices are tailored to MXFP4's specific properties: 1-bit sign, 2-bit exponent, 1-bit mantissa; per-32-element block scaling with E8M0 scale factors; a maximum representable value of 6.0. The paper explicitly notes that "MXFP4 is the only microscaling format with support for all required layouts for both forward and backward multiplications in low precision on Blackwell" (Section 3, "Blackwell Architecture Support"), and the discussion section (Section 6) acknowledges: "Certain aspects of our method rely on specialized operations, like stochastic rounding, which have hardware support for MXFP4, but may be lacking for other formats." All accuracy experiments use the Llama-2 architecture pre-trained on the C4 dataset. The speedup measurements use a single consumer GPU (RTX 5090).

The consequence. The results do not directly transfer to other hardware platforms (NVIDIA Hopper/H100, AMD MI300X, Intel Gaudi, Google TPU), other FP4 formats (NVFP4, which uses 16-element blocks and E4M3 scales; or custom 4-bit integer formats), other model architectures (Mixture-of-Experts, encoder-decoder models, vision Transformers), or other datasets (code, multilingual text, domain-specific corpora). A practitioner using non-Blackwell hardware cannot run Quartet at all — the kernel would not compile or would fall back to emulation, eliminating the speedup. A practitioner using NVFP4 (which has slightly different dynamic range and block size) would need to re-optimize the QuEST clipping search, the Hadamard block size (currently hard-coded to 32), and potentially the stochastic rounding implementation if NVFP4 hardware does not support the same rounding primitives. The C4/Llama-2 specificity means that the fitted efficiency factors eff_N = 0.86 and eff_D ≈ 0.60 may not hold for models with different activation patterns (e.g., MoE with sparse expert routing produces very different gradient statistics) or datasets with different token distributions (code has more repetitive structure, multilingual data has different outlier characteristics).

What evidence exists in the paper. The hardware-format coupling is documented honestly in Section 3 and Section 6. The paper does not benchmark any alternative FP4 formats, any non-Blackwell GPUs, or any architectures other than dense Llama-2 Transformers. The dataset is exclusively C4. The post-training quantization comparison (Appendix A.5) uses the same 7B model and C4 perplexity, so it does not test cross-dataset generalization. No experiment varies the dataset or architecture to probe robustness.

Mitigation status. The paper acknowledges this as a limitation in Section 6: "One current limiting factor is that Quartet was designed with a specific (standard) data-type and compute architecture in mind... In future work, we plan to look into generalizing our approach to alternative formats, as well as larger-scale distributed model execution." This is honest but does not reduce the practical barrier: a user on Hopper GPUs or with NVFP4 hardware cannot adopt Quartet today. The scaling-law framework (Ingredients 1 and 2) is architecture- and format-agnostic — it provides a methodology for re-deriving the optimal strategy on a new hardware-format combination — but redoing the full analysis (fitting scaling laws, measuring kernel speedups, re-optimizing the implementation) would require repeating most of the paper's experimental program. The conceptual contribution generalizes; the concrete artifact does not.

Difficulty (Data Efficiency) Estimation Cost Is Not Factored into the Optimality Analysis

The assumption or constraint. The scaling-law framework that determines when FP4 is "optimal" compares FP4 and FP8 under a fixed compute budget by plugging fitted efficiency factors and measured speedups into Equation (1). But the fitting of eff_N and eff_D itself requires training multiple models across a grid of sizes and data budgets — specifically, 24 baseline runs (4 model sizes × 6 D/N ratios) just for Stage 1 of the fitting procedure, plus additional runs for each quantized method to fit its efficiency coefficients. For Quartet, the paper reports that the pre-training experiments consumed "around 6,000 GPU-hours" on 8×H100 nodes (Appendix A.6). This cost is the "price of entry" for obtaining the efficiency factors that power the optimality analysis.

The consequence. A practitioner who wants to determine whether Quartet FP4 is optimal for their specific model architecture, dataset, or hardware configuration cannot simply apply the paper's fitted eff_N and eff_D values — those are specific to Llama-2/C4/MXFP4/RTX 5090. They would need to re-run the scaling-law fitting procedure on their own setup, which itself costs thousands of GPU-hours. This places the scaling-law framework in an awkward position: it is a powerful conceptual tool for comparing quantized training methods, but its practical deployment as a decision-making tool requires a substantial up-front compute investment that may not be justified if the conclusion is simply "use Quartet as described in the paper." The paper's optimality framework is more useful as a retrospective analysis explaining why Quartet works than as a prospective tool that a new user would apply.

More subtly, the fitting procedure assumes that the efficiency factors eff_N and eff_D are independent of the model size and data budget — i.e., that a multiplicative factor fitted on 30M–200M models applies equally at 7B. The paper provides no evidence for this assumption beyond the single 7B stability run, which does not isolate efficiency factors. If eff_D decreases with model size (plausible, since larger models have deeper gradient propagation paths where quantization bias can compound), the extrapolated optimality regions would be overly optimistic.

What evidence exists in the paper. The compute cost of the experiments is reported transparently in Appendix A.6. The two-stage fitting procedure is described in Appendix A.2. But neither section discusses the deployment cost of the scaling-law methodology — that a new user would need to replicate this effort. The paper implicitly assumes that the fitted coefficients are reusable for the target use case (Llama-style models on English text with MXFP4 on Blackwell). This is a reasonable assumption for the paper's scope, but it limits the framework's portability.

Mitigation status. Not addressed. The paper does not discuss whether efficiency factors transfer across architectures or datasets, or whether a cheaper procedure (e.g., fitting on a single model size, using fewer D/N ratios, or predicting eff_N from MSE on calibration data without training) could approximate the full scaling-law fit. The optimality analysis in Figure 1 is presented as an output of the framework, not as a process that new users would need to replicate. Given that the paper's code is open-sourced, a motivated practitioner could run the fitting procedure on their own setup, but the cost would be substantial and the paper provides no guidance on reducing it.

The Error-Bias Trade-Off Leaves Hard Regimes Unaddressed: Very Short Training Runs and Models Far from the Data-Saturated Regime

The assumption or constraint. Quartet's backward pass uses stochastic rounding (SR) because the paper's analysis shows that SR's unbiased gradient estimates yield higher data efficiency eff_D than deterministic round-to-nearest (RTN) for D/N > ~400 (Figure 2c). However, for D/N < ~400, RTN actually outperforms SR — the higher per-step gradient fidelity (better cosine similarity, Figure 2a) matters more than the unbiasedness when the total number of optimizer steps is small and the bias does not have time to compound into a significant asymptotic gap. Quartet commits unconditionally to SR on the backward pass, making it suboptimal for short-training or low-data regimes.

The consequence. There are important practical scenarios where D/N ratios are well below 400: fine-tuning pre-trained models on small task-specific datasets (D/N ≈ 10–50); continued pre-training on domain-specific corpora (D/N ≈ 50–200); or training very large models under compute budgets where the Chinchilla-optimal D/N ratio is ~20 (Hoffmann et al., 2022). In these regimes, Quartet's design choice of SR on the backward pass is demonstrably worse than using RTN on the backward pass, per the paper's own Figure 2(c). A practitioner fine-tuning a 7B model on a 1B-token instruction dataset (D/N ≈ 140) would get better results by replacing Quartet's stochastic rounding with deterministic round-to-nearest on the backward pass — but this configuration is not explored, benchmarked, or supported by the kernel implementation (which is specialized for SR). Quartet is optimized for the data-saturated pre-training regime and its design degrades when that assumption is violated.

The same logic applies to model scale: the inflection point D/N ≈ 400 was measured at 30M parameters (Figure 2c). It is possible that the SR-vs-RTN crossover D/N depends on model size — deeper networks may be more sensitive to gradient bias accumulation, shifting the crossover to even higher D/N, or the opposite if larger models have more redundancy and can tolerate biased gradients better. The paper provides no evidence either way.

What evidence exists in the paper. Figure 2(c) explicitly shows the crossover, with RTN outperforming SR for D/N < ~400 and SR pulling ahead for D/N > ~400. The paper acknowledges this in Section 4.3: "while RTN backward quantization may be preferable for shorter training, stochastic rounding (SR) performs consistently better for models more saturated with data." Table 2 shows that RTN has better cosine similarity than SR across all layers (Figure 2a) and better MSE, confirming that its advantage at low D/N is not a statistical fluke. The 30M-parameter results in Table 3 include D/N ratios from 100 to 800 — the lowest is 100, which is already below the 400 crossover, yet Quartet (with SR backward) still outperforms prior methods. This suggests that the combined effect of QuEST forward (high eff_N) + SR backward (moderate eff_D) is still net positive even at D/N = 100, but a hypothetical Quartet variant using RTN on the backward pass might perform even better at D/N = 100 and 200.

Mitigation status. The paper does not offer a solution. Quartet is presented as a single fixed algorithm (QuEST forward, SR backward) without a mechanism to adapt the backward quantizer based on the training regime. The scaling-law framework (Ingredients 1 and 2) is capable of expressing the trade-off — one could fit separate efficiency factors for Quartet-SR and Quartet-RTN and select per-regime — but the paper does not pursue this. The kernel implementation is specialized for SR (the 3/4 rescaling and stochastic rounding PTX instructions), and a hypothetical RTN-backward variant would require a different kernel, which is not provided. This limitation is a direct consequence of the paper's focus on pre-training; the SR-vs-RTN choice is reasonable for the paper's target regime but the lack of adaptivity is a genuine constraint for practitioners with different training budgets.

The Gradient Bias Metric (Projection Magnitude Misalignment) Is Theoretically Motivated but Not Directly Validated as a Predictor of Training Outcomes at Scale

The assumption or constraint. The paper introduces the projection magnitude misalignment metric (Section 4.3) as the key diagnostic for evaluating backward-pass quantizers, derived from Vargaftik et al.'s analysis of RTN quantization with randomized Hadamard transforms. The metric quantifies gradient magnitude bias as 1 − E[1/S], where S is the rescaling factor that would make a post-Hadamard RTN-quantized vector unbiased. The paper uses this metric to explain why stochastic rounding is superior to RTN for the backward pass: SR achieves zero misalignment (0.00 in Table 2), RTN shows positive misalignment, and the ranking by misalignment correlates with the ranking by fitted eff_D from the scaling law. The paper then claims that the misalignment metric is a practical tool for comparing backward quantizers without expensive training runs: "a researcher can compute the projection magnitude misalignment on representative gradient distributions and predict which method will have better data efficiency."

The consequence. If the misalignment metric is not actually a reliable predictor of eff_D across a broader range of conditions (different model sizes, architectures, datasets, or training hyperparameters), then the paper's theoretical framework — while elegant — does not deliver on its promise of replacing training-based evaluation. The metric is computed on random vectors drawn from distributions matching observed gradient statistics and a specific quantization pipeline (random Hadamard → quantization). It is not obvious that this simplified setup captures the complex, layer-dependent, and training-dynamics-dependent behavior of real gradients in a deep Transformer. Figure 2 shows that the misalignment varies across layers (it accumulates with backpropagation depth, Figure 2b), but the single-number metric in Table 2 aggregates this into one value — losing potentially important layer-specific information.

The metric's predictive power has been demonstrated for exactly one comparison (SR vs. RTN) at one model size (30M) on one dataset (C4). The paper does not show that the misalignment metric correctly predicts the eff_D ranking for additional quantizers (e.g., LSQ, QuEST, or hybrid schemes), or that the quantitative misalignment values map linearly to eff_D (e.g., does a 2× larger misalignment imply a 2× smaller eff_D?). Without this broader validation, the metric is an interesting correlation rather than a validated predictive tool.

What evidence exists in the paper. Table 2 reports misalignment for SR (0.00) and RTN (non-zero, exact value in the table). The paper states that the ranking by misalignment matches the ranking by fitted eff_D — but this is effectively a two-point comparison. Figure 2 shows that the misalignment varies by layer and that the loss gap between SR and RTN crosses over at D/N ≈ 400, consistent with the theory that bias matters more for long training. But this validates the qualitative direction of the effect, not the quantitative predictive power of the metric. No experiment tests whether the misalignment metric correctly predicts eff_D for LSQ-backward, QuEST-backward, or any quantizer other than SR and RTN. No experiment tests whether the metric computed on 30M-parameter model gradients correctly predicts eff_D at 200M or 7B parameters.

Mitigation status. The paper does not claim more than the data supports — it presents the misalignment metric as a correlate of data efficiency and an explanation for SR's superiority, not as a universally validated predictor. But statements like "To connect those quantities with training dynamics, we analyze the cumulative effect of misalignment and error on backward quantization" (Section 4.3) imply a causal connection that is only partially validated. The paper's theoretical framing would be strengthened by: (1) measuring misalignment for all backward quantizers compared (LSQ, QuEST, HALO, Jetfire, LUQ — not just SR and RTN) and showing the ranking matches fitted eff_D; (2) measuring the metric at multiple model scales and showing it predicts the scaling of eff_D with model size; (3) providing a sensitivity analysis — how much does misalignment need to change to produce a measurable difference in final loss? These validations are absent.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a framework-level shift in how the field evaluates low-precision training. Before Quartet, the dominant criterion for a quantized training method was accuracy preservation — the method should match full-precision training results within some small tolerance. Methods that lost accuracy, even modestly, were considered failures. This paper argues, convincingly, that this criterion is too strict and, in fact, misses the point: the right question is not "does FP4 match FP8 at the same (N, D)?" but rather "under a fixed wall-clock compute budget, does FP4 produce a better final model than FP8?" The distinction is not semantic — it is economic. If FP4 is 1.6× faster than FP8, then in the same training time you can process 1.6× more tokens. If the additional data outweighs the per-token learning efficiency loss from quantization, FP4 is optimal, even though it is not lossless.

This is a methodological reframing, not a paradigm shift. It does not overturn the foundations of deep learning or optimization theory — it changes how practitioners should evaluate quantized training methods. The scale of the shift is comparable to what Chinchilla (Hoffmann et al., 2022) did for pretraining: Chinchilla did not invent a new architecture or training algorithm; it changed how the field allocates compute between model size and data quantity by introducing a compute-matched rather than parameter-matched evaluation criterion. Quartet does the same for precision: it changes the evaluation from accuracy-matched to compute-matched, and in doing so reveals that FP4 can be "optimal" in regimes where the old criterion would have dismissed it as degraded.

This reframing matters because it resolves a tension that has quietly shaped the low-precision training literature. On one hand, hardware trends — NVIDIA's Blackwell architecture, microscaling formats like MXFP4 and NVFP4 — are pushing toward 4-bit as the next throughput frontier. On the other hand, algorithmic results consistently showed that 4-bit training incurs non-trivial accuracy penalties, leading to a stalemate: the hardware is ready, but the algorithms cannot deliver accuracy-preserving training. Quartet's compute-matched optimality analysis breaks this stalemate by showing that accuracy-preserving is the wrong goal. You do not need FP4 to match FP8 at equal tokens; you need it to beat FP8 at equal time. Quartet demonstrates that this is achievable today on Blackwell hardware for models at the scale of Llama3-8B or Qwen2.5-7B (Figure 1c).

The paper also resolves a contradiction in prior empirical findings about what matters for low-precision training. Prior work had produced two competing intuitions: (1) that minimizing quantization error (MSE) is paramount, motivating methods like QuEST and LSQ that optimize clipping factors to reduce reconstruction error; and (2) that unbiased gradient estimation is critical, motivating stochastic rounding approaches (Chmiel et al., 2023; Tseng et al., 2025). These intuitions seemed contradictory because no single method could simultaneously minimize MSE and eliminate bias. Quartet's key insight — that the forward pass and backward pass impose different requirements, and should use different quantizers — resolves this tension. QuEST (low MSE) on the forward pass preserves parameter efficiency (eff_N = 0.86). Stochastic rounding (zero bias) on the backward pass preserves data efficiency. The error-bias trade-off is not a bug to be eliminated but a design axis to be exploited: compose the best forward quantizer with the best backward quantizer rather than searching for a nonexistent universal optimum.

The practical consequence is that the research landscape shifts in two ways:

Directions that become more attractive: Building verifier-like components (in the training context, the backward-pass quantizer) that are explicitly designed for unbiasedness rather than low error. Improving forward-pass quantizers through better clipping, learned scales, or format-specific optimizations — because the paper shows that forward-pass MSE directly translates to parameter efficiency. Developing scaling-law frameworks as standard evaluation tools for training methods, not just for model-size-vs-data decisions. Engineering highly optimized fused kernels that amortize quantization overhead — the paper demonstrates that kernel quality is not an afterthought but a first-class determinant of whether theoretical advantages translate to real speedups.

Directions that become less attractive: The search for a "universal" 4-bit quantizer that works equally well on forward and backward passes — Quartet's analysis strongly suggests this is a dead end, and effort should shift to specialized per-pass quantizers. Evaluations of quantized training methods that report only accuracy at fixed (N, D) without accounting for throughput — the compute-matched criterion makes such comparisons potentially misleading. Overly complex quantization schemes that add significant per-step overhead — Quartet's performance breakdown (Figure 6, Appendix A.3) shows that quantization overhead must be aggressively minimized for net speedup, making simpler, fusable operations preferable to elaborate multi-pass calibration.

Follow-Up Research This Work Enables

Validating the optimality prediction with a wall-clock-matched large-scale training run. The paper's central claim — that FP4 training with Quartet produces a better model than FP8 under equal compute — is a prediction of the fitted scaling law extrapolated from 30M–200M models to the 7B–70B range. No experiment in the paper directly tests this. A strong follow-up would train a 7B-parameter Llama model with Quartet on a fixed GPU budget (e.g., 64×H100 for 24 hours) and compare its validation loss and downstream performance against an FP8-trained 7B model trained for the same wall-clock time (which would process ~1.6× fewer tokens). If the Quartet model achieves lower loss, the optimality prediction is validated at production scale. If not — if the extrapolated efficiency factors turn out to be optimistic — then the scaling law's assumptions about constant eff_N and eff_D across scale are falsified, and more sophisticated efficiency models (e.g., scale-dependent factors) would be needed.
Extending the misalignment metric to a broader set of backward quantizers and validating its predictive power. The paper introduces the projection magnitude misalignment metric and shows it correlates with eff_D for exactly two quantizers: stochastic rounding (misalignment = 0.00, good eff_D) and round-to-nearest (positive misalignment, worse eff_D). This is a two-point correlation. A rigorous follow-up would: (1) implement 4–6 additional backward quantizers spanning a range of bias-variance trade-offs (e.g., RTN with different clipping factors, stochastic rounding with different temperature parameters, a hybrid that uses SR for large values and RTN for small values); (2) compute their misalignment on gradient distributions from a 200M-parameter model at multiple training stages; (3) train 30M-parameter models with each backward quantizer (keeping the forward pass fixed, e.g., QuEST) across D/N ratios from 50 to 800; (4) fit eff_D from the scaling law for each; and (5) test whether the misalignment metric linearly predicts eff_D. A strong linear relationship would validate the metric as a cheap proxy for training-based evaluation; a weak or non-linear relationship would reveal that the metric captures only part of the relevant bias, and additional factors (e.g., layer-dependent bias accumulation, interaction with the optimizer's adaptive learning rates) need to be modeled.
FP4-native adaptations of competing methods to enable fair head-to-head comparison. The paper's experimental comparison (Table 3) evaluates Quartet against Jetfire, HALO, LSS, and LUQ — but Jetfire and HALO were designed for INT8, and their FP4 adaptation is described only as "adapted to FP4 for our setup" without details on how block sizes, clipping heuristics, or rounding strategies were adjusted. A fair comparison requires FP4-native re-optimization of these methods: (1) re-tune Jetfire's per-block quantization to use 32-element blocks matching the MXFP4 scale granularity (rather than its original 32×32 blocks designed for INT8), and integrate a QuEST-like clipping search to handle FP4's dynamic range; (2) re-tune HALO's Hadamard rotation and scaling to FP4, potentially using a learned or RMSE-optimal clipping threshold instead of AbsMax; (3) test whether LSS's divergence at D/N > 50 can be fixed by replacing its leverage-score backward pass with stochastic rounding, producing a "LSS + SR" hybrid. This would reveal how much of Quartet's advantage comes from the specific forward-backward quantizer composition versus from simply being the only method purpose-built for MXFP4.
Adaptive backward-pass quantizer selection based on training phase. Figure 2(c) reveals that RTN outperforms SR on the backward pass for D/N < ~400, while SR outperforms RTN for D/N > ~400. Quartet uses SR uniformly, which is suboptimal for short-training or fine-tuning regimes. A natural extension is a phase-aware quantizer: use RTN on the backward pass during early training (or for the first N tokens) to benefit from higher per-step gradient fidelity, then switch to SR after a precomputed crossover point to eliminate bias accumulation. The crossover D/N could be estimated cheaply by running the misalignment analysis on a small proxy model and identifying the inflection point. A follow-up experiment would: (1) train 200M-parameter models with fixed SR-backward, fixed RTN-backward, and a switched RTN→SR policy (with the switch at D/N = 200, 400, or 600); (2) compare final validation loss at D/N = 800; and (3) measure whether the switched policy outperforms both fixed strategies, indicating that the error-bias trade-off can be exploited dynamically rather than statically. A negative result (switched policy no better than fixed SR) would suggest that the early training phase's gradient fidelity has little impact on the final converged solution, and that the asymptotic regime dominates — this would be valuable negative knowledge.
Scaling-law analysis across architectures, datasets, and FP4 formats to test generalization. The paper's fitted efficiency factors (eff_N = 0.86, eff_D ≈ 0.60) are specific to Llama-2 architecture, C4 dataset, MXFP4 format, and the AdamW optimizer with the hyperparameters in Tables 4–5. A systematic generalization study would fit the same scaling law (Equation 1) for at least two additional configurations: (1) a different architecture (e.g., a Mixture-of-Experts Transformer, which has different gradient statistics due to sparse expert routing, or an encoder-decoder model like T5); (2) a different dataset (e.g., The Pile, which has different outlier characteristics than C4, or a code-only corpus like The Stack); and (3) if hardware support becomes available, the NVFP4 format (16-element blocks, E4M3 scales rather than E8M0), to test whether the efficiency factors are format-dependent. If eff_N and eff_D vary substantially across these configurations, the optimality regions in Figure 1 are specific to the tested setup, and future deployments would need re-fitting. If they are stable, the framework becomes a transferable tool.
End-to-end distributed training throughput benchmarking with full optimizer and communication accounting. The paper reports kernel-level speedups (up to 1.6× overall training) and inference prefill speedups (up to 1.41×), but not end-to-end training throughput including the AdamW optimizer step (FP32), gradient AllReduce communication (for distributed training), attention computation (which may or may not be quantized — the paper is silent on this), and data loading. A critical follow-up engineering benchmark would: (1) implement Quartet in a distributed training framework (e.g., PyTorch FSDP or DeepSpeed) with the quantized linear layers integrated into the full Transformer block including attention, normalization, and residual connections; (2) measure tokens-per-second on 8×H100 and, if available, 8×B200 configurations for model sizes from 1B to 7B at realistic batch sizes and sequence lengths (e.g., 4K tokens); (3) report the fraction of total training time spent in quantized GEMMs vs. attention vs. optimizer vs. communication vs. other operations; and (4) compute the true end-to-end speedup. If the end-to-end speedup is substantially lower than 1.6× (e.g., 1.2–1.3×), the optimality regions in Figure 1 would shrink or shift, and the practical case for FP4 training would be weakened. This is essential for the paper's adoption by practitioners making hardware allocation decisions.

Practical Applications and Downstream Use Cases

Cost-efficient pre-training of production-scale LLMs on Blackwell clusters. If validated by end-to-end throughput benchmarks, Quartet would directly reduce the GPU-hours required to pre-train models at the scale of Llama3-8B or Qwen2.5-7B. At a measured kernel-level speedup of ~1.6× over FP8, training a model that requires 1 million GPU-hours in FP8 could be completed in approximately 625,000 GPU-hours with Quartet — a savings of 375,000 GPU-hours, translating to tens of thousands of dollars in cloud compute costs. The paper's optimality analysis (Figure 1c) suggests that this throughput gain more than compensates for the accuracy penalty, meaning the Quartet-trained model would have better quality than an FP8-trained model trained for the same wall-clock time. This is a concrete economic argument for adopting FP4 training in organizations that train models from scratch on Blackwell hardware. The primary risk is the unvalidated extrapolation from kernel-level to end-to-end speedup.
Native FP4 inference deployment without a separate PTQ step. Because Quartet is a quantization-aware training method that quantizes the forward pass throughout training, the resulting model checkpoint is already in MXFP4 format — weights are stored and used in 4-bit precision during training. There is no need for a post-training quantization step (GPTQ, QuaRot, AWQ) to compress the model for inference deployment. Appendix A.5 (Table 7) shows that the Quartet-trained 7B model achieves 0.42-point lower perplexity than applying QuaRot + GPTQ (the best available PTQ method) to an FP8-trained model. This means Quartet simultaneously provides training-time speedup and superior deployment-time quality, eliminating the pipeline complexity of train-in-FP8-then-quantize-to-FP4. For organizations deploying models on edge devices or in latency-sensitive applications where FP4 inference is required for throughput, Quartet-trained models offer a quality advantage over PTQ-compressed models with zero additional post-processing.
Large-scale data generation and distillation pipelines. When using LLMs to generate training data for smaller models (knowledge distillation, synthetic data generation, or self-improvement loops like STaR or ReST^EM), the inference cost of the teacher model can dominate the total budget. Quartet's inference prefill speedup of up to 1.41× over FP8 (Figure 7, Appendix A.4) directly reduces this cost. Moreover, because Quartet trains the model to be robust to FP4 quantization throughout, the generated outputs may be more consistent with the model's FP4 inference behavior than a model trained in FP8 and post-hoc quantized to FP4 via PTQ. A concrete use case: a 70B teacher model trained with Quartet generates synthetic instruction-following data for distilling into a 7B student model. The 1.41× throughput gain on the teacher reduces generation time proportionally, and the teacher's native-FP4 training means there is no distribution shift between its training behavior and its generation behavior.
Edge and on-device model fine-tuning with limited precision hardware. As on-device training (e.g., federated learning, personalization, or continuous adaptation) becomes more prevalent, the training hardware is increasingly constrained to low-precision formats for energy efficiency. Smartphones, IoT devices, and edge accelerators may support FP4 or INT4 operations natively but lack efficient FP8 or FP16 pipelines. Quartet demonstrates that it is possible to train Transformers end-to-end in 4-bit precision with acceptable accuracy degradation — and the scaling-law framework provides a methodology for determining whether the degradation is acceptable for a given application's accuracy requirements and compute budget. A practitioner deploying on-device fine-tuning on a Blackwell-based edge accelerator could adopt Quartet's forward-quantizer (QuEST) and backward-quantizer (stochastic rounding) directly, using the paper's open-source QuTLASS library as a starting point for kernel implementation on their specific hardware. The paper's 30M-parameter results (Table 3) are particularly relevant here, as on-device models often fall in the 10M–100M parameter range.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning2025· training-methods← in-domain predecessor
An Empirical Study of Qwen3 Quantization2025· low-precision← topic thread
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits2024· pretraining← LLM-judged similar
- VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models2024· low-precision← in-domain predecessor
- Qwen3 Technical Report2025· llm-systems← topic thread
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training2025· inference-optimization← topic thread
Optimizing Large Language Model Training Using FP4 Quantization2025· training-methods← similar topic
- The Lessons of Developing Process Reward Models in Mathematical Reasoning2025· training-methods← in-domain predecessor
- SageAttention2 Technical Report: Accurate 4 Bit Attention for Plug-and-play Inference Acceleration2024· inference-optimization← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

Pretraining Large Language Models with NVFP42025· pretraining→ LLM-judged similar
- Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation2026· training-methods→ LLM-judged similar
- RLP: Reinforcement as a Pretraining Objective2025· pretraining→ in-domain predecessor
- INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats2025· inference-optimization→ topic thread
FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models2025· training-methods→ in-domain predecessor
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models2025· training-methods→ in-domain predecessor
- Adaptive Subgradient Methods for Online Learning and Stochastic Optimization2000· training-methods→ similar topic
The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm2025· llm-systems→ topic thread
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models2025· llm-systems→ in-domain predecessor
- Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization2025· inference-optimization→ topic thread
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution2024· llm-systems→ similar topic
Optimizing Large Language Model Training Using FP4 Quantization2025· training-methods→ similar topic
- Decoding-based Regression2025· training-methods→ in-domain predecessor
How Far Can Unsupervised RLVR Scale LLM Training?2026· training-methods→ similar topic
- MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Data2026· training-methods→ in-domain predecessor
- How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum2026· rl-training→ topic thread
- AMSP: Super-Scaling LLM Training via Advanced Model States Partitioning2023· training-methods→ similar topic
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection2024· training-methods→ similar topic
- Memory-Efficient LLM Training with Online Subspace Descent2024· training-methods→ similar topic