Optimizing Large Language Model Training Using FP4 Quantization

ArXiv: 2501.17116

🎯 Pitch

This work shows for the first time that training 13B-parameter language models in FP4 precision—just 16 distinct representable values—matches BF16 accuracy by fixing two previously fatal flaws: a differentiable gradient estimator that corrects weight update errors, and an outlier clamping strategy that stops activations from collapsing the training signal.

---

1. Executive Summary

This paper introduces the first FP4 training framework for large language models, addressing the significant quantization errors and limited representational capacity that make 4-bit training challenging. The framework is validated on LLaMA 2 models (1.3B, 7B, and 13B parameters) trained on up to 100B tokens from the DCLM dataset, using two key innovations: a Differentiable Gradient Estimator (DGE) that replaces the standard straight-through estimator with a correction term derived from a differentiable approximation of the quantization function (applied to weight gradients during backpropagation), and an Outlier Clamping and Compensation (OCC) strategy that caps extreme activation values at a high quantile and preserves fidelity through a sparse auxiliary matrix (addressing the activation outlier problem specific to LLM training). The FP4 framework achieves accuracy comparable to BF16 and FP8 baselines with minimal degradation—final training losses of 2.55 vs. 2.49 (1.3B), 2.17 vs. 2.07 (7B), and 1.97 vs. 1.88 (13B)—while zero-shot downstream task accuracy is competitive or slightly favorable, establishing that ultra-low precision training is feasible at scale with proper treatment of weight gradient estimation and activation outlier dynamics.

2. Context and Motivation

The Core Problem: FP4 Training Introduces Quantization Errors That Destroy Model Accuracy

The fundamental challenge this paper addresses is deceptively simple to state but technically severe in practice: training large language models using 4-bit floating-point (FP4) arithmetic for matrix multiplications, without catastrophic accuracy degradation. FP4 provides only 16 distinct representable values — half as many as FP8 and an order of magnitude fewer than FP16/BF16 — with a dynamic range spanning just [−6, 6] in the E2M1 format the paper adopts. When you force the weight and activation tensors of a Transformer through this narrow bottleneck during training, the quantization errors compound across forward and backward passes, and the model either diverges entirely (loss goes to NaN) or converges to a substantially worse solution.

Figure 1 makes the stakes concrete: directly casting computations to FP4 during LLaMA 400M training produces a loss curve that sits roughly 1–2 points higher than the BF16 baseline throughout training — a gap that would compound into unusable models at scale. The paper's proposed FP4 method closes this gap almost completely.

This is not merely an academic exercise in compression. The practical imperative comes from the staggering computational cost of modern LLM training. The paper cites specific numbers that ground the urgency: Llama 3 405B required "up to 16K H100 GPUs for 54 days," and GPT-4 (estimated at ~1T parameters) demanded "an extraordinary amount of computational power." Training costs at this scale run into the hundreds of millions of dollars, and the energy consumption carries significant environmental consequences.

The hardware roadmap provides the second half of the motivation. NVIDIA's Hopper architecture (H100) introduced FP8 tensor cores delivering a 2× speedup over FP16 tensor cores for matrix multiplication. The upcoming Blackwell architecture (B200) extends this to FP6 and FP4 formats, with FP4 theoretically offering double the throughput of FP8 — a potential 4× speedup over FP16 for the General Matrix Multiplications (GeMMs) that account for "over 95% of the computational workload" in LLM training (Section 4.1). This means the hardware to accelerate FP4 training is imminent, but the software framework to use it without sacrificing model quality did not exist before this paper.

"the application of FP4 tensor cores in model training remains an open research question"

Why Existing Approaches Fall Short for FP4 Training

The paper positions its work against several lines of prior research, each of which addresses part of the problem but leaves FP4 pretraining unsolved:

1. FP8 Training Works, But FP4 Is Qualitatively Different

FP8 training has been demonstrated as feasible by two major efforts: NVIDIA's Transformer Engine (Nvidia, 2022) and FP8-LM (Peng et al., 2023). These frameworks perform mixed-precision training where GeMM operations use FP8 while non-GeMM operations (layer norms, softmax, etc.) remain at higher precision. The paper's own ablation (Figure 6a) confirms that both MS-AMP FP8 and Transformer Engine FP8 closely track the BF16 loss curve.

However, FP4 is not simply "FP8 with two fewer bits." The transition from 256 representable values (FP8 E4M3) to 16 values (FP4 E2M1) crosses a threshold where the standard techniques that worked for FP8 — particularly the straight-through estimator (STE) for gradient propagation and coarse-grained tensor-wise scaling factors — break down. The STE assumes the gradient of the quantization function is identically 1 everywhere, which is a reasonable approximation when quantization errors are small relative to signal. At 4 bits, this approximation becomes so coarse that weight updates lose meaningful direction, especially when the gradient signal itself has been quantized.

2. The Straight-Through Estimator (STE) Is a Known Weakness at Low Bit Widths

The STE, introduced by Bengio et al. (2013), has been the default solution to the non-differentiability of quantization functions in neural network training for over a decade. It works by maintaining the hard quantization function in the forward pass (preserving computational efficiency) while pretending, during backpropagation, that the derivative is 1 — effectively bypassing the quantization step in the gradient computation.

Section 3.1 provides the mathematical justification for why this fails at 4 bits. The true gradient of the weight in a linear layer with quantized weights is:

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial W_q} \odot f'(W)$

where $f$ is the quantization function. Since $f$ is a step function, its derivative $f'$ is zero almost everywhere (flat regions) and undefined at step boundaries. The STE simply replaces $f'(W) \equiv 1$, which means $\partial L / \partial W = \partial L / \partial W_q$. This is equivalent to modeling the quantization function as $f(x) = x$ with derivative 1 — an identity mapping that erases all quantization effects from the gradient computation.

For FP8, where quantization intervals are relatively fine, this approximation is tolerable. For FP4's 14 quantization intervals across [−6, 6], the identity assumption injects substantial error into weight updates. The paper's ablation (Figure 6b, "W4A8" setting) shows that even weight-only 4-bit quantization without DGE introduces a visible loss gap, though not a catastrophic one — confirming that the STE is a limiting factor specifically for weight gradients at this precision.

3. Activation Outliers Are the Killer Problem for 4-Bit Training

While weights present a solvable gradient estimation challenge, activations pose a fundamentally harder problem: outlier values that expand the dynamic range and cause massive underflow. The paper provides extensive distributional evidence in Appendix D (Figures 11–13) showing activation tensors during LLM training exhibit:

• Much larger dynamic ranges than weight tensors (standard deviations from ~0.01 for weights to 0.1–1.2 for activations)
• Heavy-tailed, irregular distributions with values 10–50× larger than the typical spread
• Channel-specific outlier patterns (Figure 14's heatmap shows distinct vertical light lines — entire channels dominated by extreme values)

When these outlier-dominated tensors are fed into the absmax quantization scheme (Equation 1), the scaling factor $\gamma = \text{MAX}_{\text{fp4}} / \max(|x_{\text{fp16}}|)$ becomes dominated by the single largest magnitude value. For FP4 with $\text{MAX}_{\text{fp4}} = 6.0$, if an activation tensor has a single outlier at 75 while most values cluster around 0–5, the scaling factor becomes $6.0 / 75 \approx 0.08$. After scaling and quantization, values in the 0–5 range become 0–0.4 before quantization, which rounds to either 0 or 0.5 in the FP4 grid — essentially collapsing all signal into one or two quantization bins. Figure 4 (top panel) visualizes this collapse vividly: the BF16 distribution shows rich structure across [−75, 75], but after FP4 quantization without outlier handling, almost all values map to zero.

The paper reports (Figure 6c) that directly quantizing activations to FP4 in a W8A4 setting causes training divergence — the loss goes to NaN. This is not a gradual degradation but a hard failure mode. The reason is that once activation information is destroyed by underflow in the forward pass, the backward gradients become noise, and the optimizer takes destructive steps.

4. Inference-Time Quantization Techniques Don't Transfer to Training

A large body of work on Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) has successfully pushed LLMs to 4 bits or lower for inference. The paper explicitly distinguishes its approach from these:

• PTQ methods (GPTQ by Frantar et al., 2023; AWQ by Lin et al., 2024a; SmoothQuant by Xiao et al., 2023) quantize a pre-trained model using calibration data to determine scaling factors or transformations. These are offline, one-shot procedures that work because the weight values are frozen — you can analyze the entire weight matrix to find optimal per-channel or per-group quantization parameters. During training, weights evolve continuously, making offline calibration impossible.

• QAT methods (LLM-QAT by Liu et al., 2023b; BitNet by Wang et al., 2023) fine-tune or retrain models to be robust to quantization at inference time. These methods "aim to accelerate the training process while maintaining performance, rather than solely focusing on improving inference efficiency" (Section 5). The distinction is subtle but critical: QAT accepts the training cost as a sunk investment to improve inference, while FQT (Fully Quantized Training) must reduce the training cost itself.

• Outlier handling for inference (SmoothQuant's channel-wise scaling, QuaRot's random rotation matrices by Ashkboos et al., 2024, SpinQuant's learned rotations by Liu et al., 2024) relies on offline preprocessing with calibration datasets. These methods shift quantization difficulty from activations to weights by applying fixed transformations (scaling, rotation) computed once before deployment. During training, activations change with every batch, and re-computing transformations online would defeat the purpose of quantization-based acceleration.

"Previous strategies to solve this problem like smoothing outliers or using rotary matrices rely on offline pre-processing, making them incompatible with pretraining tasks. In contrast, our method addresses outliers dynamically during real-time training without requiring separate calibration datasets."

5. Prior 4-Bit Training Attempts Were Limited or Nonexistent for LLMs

The paper identifies only one prior attempt at 4-bit training in any deep learning context: Sun et al. (2020) applied a "4-bit radix-4 FP4 format to convolutional neural networks (CNNs)." This work predates the Transformer era and does not address the specific challenges of LLM training — notably, the activation outlier patterns that are characteristic of large Transformer models. CNNs typically exhibit more regular activation distributions (ReLU outputs are non-negative with a different outlier structure than the GeLU activations and layer norm outputs in Transformers).

The MXFP format (Rouhani et al., 2023b) introduced microscaling data formats including FP4 for GPT-style models but "lacked feasibility validation on full FP4 settings" — it was a format specification paper without end-to-end training results showing that FP4 could match higher-precision baselines.

Xi et al. (2023) proposed an INT4 training framework, but this was for fine-tuning rather than pretraining from scratch. Fine-tuning starts from a well-initialized model where weights and activations are already structured; pretraining from random initialization amplifies all quantization challenges because gradient signals are noisier and activation distributions evolve rapidly during early training.

How This Paper Positions Itself

The paper frames its contribution not as an incremental improvement to existing quantization methods but as establishing the feasibility of FP4 LLM pretraining for the first time. This is a validation paper more than an algorithmic novelty paper — the key claim is that FP4 training can work, provided two specific technical challenges are addressed.

The two innovations — DGE and OCC — are presented as necessary and (empirically) sufficient conditions for closing the gap to BF16. Neither is claimed to be architecturally novel in isolation: differentiable approximations to quantization functions have been explored in prior differentiable quantization literature (Gong et al., 2019; Uhlich et al., 2019; Chen et al., 2019), and outlier handling through clamping is a standard signal processing technique. What is novel is:

1. The specific formulation of DGE as a gradient correction term applied only during backpropagation, with the forward pass retaining hard quantization for hardware efficiency. This differs from prior differentiable quantization approaches that replace the quantization function entirely with a soft approximation in both forward and backward passes, which would negate the computational speedup from FP4 hardware. The DGE only modifies the weight gradient computation (Equation 6), adding an element-wise correction factor $f'(W_{\text{scaled}})$ that costs "approximately 8 FLOPs per input element" (Appendix B) — a negligible overhead relative to the matrix multiplication itself.

2. The combination of dynamic outlier clamping with sparse compensation, applied continuously during training rather than as a one-time offline step. The clamping threshold is set at a high quantile (α = 0.99, meaning only the top 1% of values by magnitude are clamped), and the error introduced by clamping these few extreme values is recovered through a sparse high-precision matrix multiplication on the residual. This specifically targets the channel-wise activation outlier pattern documented in Figure 14 while preserving the computational benefits of FP4 for the dense majority of computation.

3. The integration of both techniques into a complete mixed-precision training framework that also incorporates FP8 gradient communication, FP8/FP16 mixed-precision Adam optimizer states (following Peng et al., 2023), and vector-wise (token-wise for activations, channel-wise for weights) scaling factors. The paper emphasizes that coarse-grained tensor-wise scaling — sufficient for FP8 — introduces "significant errors" at FP4 precision (Figure 6d), and that activation quantization granularity is the more critical dimension ("applying coarse-grained quantization to activations alone result in more severe accuracy degradation than applying it to weights alone").

The paper's positioning vis-à-vis the hardware timeline is explicit and strategic: it serves as a call for next-generation hardware designs while simultaneously providing the software foundation for those designs. Section 6 (Limitations) acknowledges the absence of native FP4 tensor cores as the primary constraint on the current work — all experiments use FP8 tensor cores on H100 GPUs to simulate FP4, which introduces precision casting overhead and prevents direct speedup measurements. The theoretical speedup analysis in Appendix B estimates a 2.95× speedup for a 7B model after accounting for DGE and OCC overhead (down from an ideal 3.12× without overhead), establishing a quantified efficiency target for future hardware.

"With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training."

The Gap This Paper Fills

The paper addresses a specific capability gap in the quantization literature: prior work had shown that (a) FP8 training works, (b) 4-bit inference works via PTQ/QAT, and (c) 4-bit fine-tuning works for INT4. But no one had demonstrated that FP4 pretraining of LLMs from scratch — where weights are random at initialization, gradients are noisy, activation distributions evolve rapidly, and the model must converge to a high-quality solution — was possible without unacceptable accuracy loss.

The paper fills this gap with systematic empirical evidence: training curves (Figure 5) showing FP4 tracking BF16 across 100B tokens at 1.3B, 7B, and 13B scales; zero-shot downstream evaluations (Table 2) showing competitive accuracy across 9 benchmarks; perplexity comparisons (Table 3) showing comparable or slightly better PPL for FP4 models; and ablation studies (Figure 6) isolating the contribution of each technique. The key evidence is not a single metric but the consistency of the finding across model scales and evaluation dimensions — FP4 training does not degrade in some unexpected way at larger scales or on particular task types.

This is fundamentally a feasibility demonstration that opens the door rather than closing it. The paper explicitly positions its work as a starting point: the theoretical speedup numbers (2.95× for 7B models) cannot be measured directly without native FP4 hardware, scaling beyond 13B parameters and 100B tokens remains untested, and the combination of DGE and OCC with other training optimizations (architecture variants, learning rate schedules, data mixtures) is unexplored. The contribution is not a production-ready training system but a validated blueprint for FP4 training that hardware vendors and LLM training teams can build upon.

3. Technical Approach

3.1 Reader Orientation

This paper builds a mixed-precision training framework that replaces the standard BF16 matrix multiplications in a Transformer with FP4 (4-bit floating-point) arithmetic while preserving model accuracy. The system solves the problem that directly quantizing weights and activations to 4 bits during training causes either catastrophic divergence (loss → NaN) or severe accuracy degradation, by addressing the two root causes: inaccurate gradient signals through the quantization function during backpropagation, and activation outlier values that destroy information during forward-pass quantization.

3.2 Big-Picture Architecture (Diagram in Words)

The FP4 training framework has four major components that modify a standard Transformer training loop:

1. FP4 GeMM Engine: Replaces BF16 matrix multiplications in linear layers with FP4 operations. Both weight tensors ($W$) and activation tensors ($A$) are quantized to FP4 using the E2M1 format (2 exponent bits, 1 mantissa bit, dynamic range [−6, 6]) via an absmax scaling scheme before being fed to the matrix multiplication kernel. Non-GeMM operations (layer norms, softmax, GeLU activations, attention score computation) remain at BF16 or FP16 precision.

2. Differentiable Gradient Estimator (DGE): Operates during the backward pass only, modifying the weight gradient computation. It computes a correction factor $f'(W_{\text{scaled}})$ using a differentiable approximation of the quantization function and applies it as an element-wise multiplier to the gradient that flows to the weight optimizer. The forward pass keeps hard quantization — DGE touches nothing in the forward computation.

3. Outlier Clamping and Compensation (OCC): Operates during the forward pass only, on activation tensors before FP4 quantization. It identifies values in the top $(1 - \alpha)$ quantile by magnitude, clamps them to the $\alpha$-quantile threshold, performs FP4 GeMM on the clamped tensor, and separately computes a sparse high-precision matrix multiplication on the residual (the clamped-out values) which is added back to recover the lost information.

4. Mixed-Precision Infrastructure: Manages precision for non-GeMM operations. Following Peng et al. (2023), it uses FP8 for gradient communication (reducing bandwidth), stores first-order Adam moments in FP8 and second-order moments in FP16 (the mixed-precision Adam optimizer), and keeps all remaining operations in BF16 or FP16.

Information flows as follows during one training step: Forward pass → activation tensors enter OCC module (clamp outliers, produce clamped tensor + sparse residual) → clamped tensor quantized token-wise to FP4 → weights quantized channel-wise to FP4 → FP4 GeMM computes $Y_q = \text{FP4GeMM}(A_q, W_q)$ → sparse GeMM adds back outlier contribution → output dequantized to BF16 → remaining Transformer operations in BF16 → loss computed. Backward pass → gradients flow normally through non-GeMM operations → at each linear layer, the weight gradient $\partial L / \partial W$ is computed using the standard chain rule, then multiplied element-wise by the DGE correction factor $f'(W_{\text{scaled}})$ before being passed to the optimizer.

3.3 Roadmap for the Deep Dive

• First, the FP4 quantization scheme itself (the E2M1 format, the absmax scaling equation, and the vector-wise granularity choices), because everything else builds on understanding what FP4 quantization physically does to tensors.
• Second, the Differentiable Gradient Estimator (DGE), which addresses why the standard straight-through estimator fails at 4 bits and how a gradient correction term derived from a differentiable approximation fixes weight gradient computation.
• Third, the Outlier Clamping and Compensation (OCC) strategy, which tackles the much harder problem of activation quantization by dynamically suppressing extreme values and recovering lost information through sparse computation.
• Fourth, the full mixed-precision training recipe, including how FP8 gradient communication and the mixed-precision Adam optimizer fit into the FP4 framework, and the hyperparameters that govern the entire system.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and validation paper whose core idea is that FP4 pretraining of LLMs is feasible if and only if two specific failure modes are addressed: (1) gradient estimation through the quantization function must go beyond the straight-through estimator for weight updates, and (2) activation outlier values must be handled dynamically to prevent quantization-induced information collapse. The paper proposes specific, computationally lightweight mechanisms for each and integrates them into a complete training framework.

---

The FP4 Quantization Format and Scaling Scheme

Format selection (E2M1). The paper adopts the E2M1 format for 4-bit floating-point numbers, as defined in prior work on microscaling formats (Rouhani et al., 2023b;a). This allocates 2 bits to the exponent ($E$) and 1 bit to the mantissa ($M$), with an additional 1-bit sign, producing exactly 16 distinct representable values. The dynamic range spans [−6.0, 6.0] with non-uniform spacing: the representable values are {±0, ±0.5, ±1.0, ±1.5, ±2.0, ±3.0, ±4.0, ±6.0}. The paper justifies this choice by the trade-off between dynamic range and precision: the E3M0 format (3 exponent bits, 0 mantissa bits) provides a larger range of [−16, 16] but coarser quantization steps (intervals of 2, 4, or 8 for larger values), while E1M2 (1 exponent, 2 mantissa) has finer intervals but a range of only [−3.5, 3.5].

Absmax quantization (Equation 1). To map a high-precision tensor (e.g., BF16) into the FP4 grid, the paper uses the absmax method:

$x_{\text{fp4}} = Q(x_{\text{fp16}} \cdot \gamma), \quad \gamma = \frac{\text{MAX}_{\text{fp4}}}{\max(|x_{\text{fp16}}|)}$

where $x_{\text{fp16}}$ is a scalar element of the high-precision tensor, $\gamma$ is a scaling factor computed per quantization group, $\text{MAX}_{\text{fp4}} = 6.0$ is the maximum absolute representable value in E2M1, and $Q(\cdot)$ is the quantization function implemented via a lookup table in a custom CUDA kernel.

What it computes: For a given tensor, first determine the absolute maximum value across all elements. Compute a scaling factor $\gamma$ that maps this maximum to 6.0. Multiply every element by $\gamma$. Round each scaled value to the nearest representable FP4 number using the lookup table (Appendix A provides the full CUDA kernel with 16 threshold comparisons per element). The result is a tensor where every element is one of the 16 FP4 values.

Why this form: Absmax quantization preserves the relative magnitudes of tensor elements — every value is scaled by the same factor — which is mathematically simpler than learned quantization parameters (used in some QAT approaches) but suffers when a single outlier inflates the scaling factor. This limitation is precisely what the OCC method addresses for activations. The alternative, min-max quantization, would use $\gamma = (\text{MAX} - \text{MIN}) / \text{range}$, but for FP formats with non-uniform intervals centered around zero, absmax is the standard approach.

Vector-wise quantization granularity (Section 4.1). The paper emphasizes that coarse-grained tensor-wise scaling fails at FP4 precision (Figure 6d). For a GeMM operation $Y = A W$ where $A$ has dimensions (sequence length × input channels) and $W$ has dimensions (input channels × output channels), the quantization granularity is:

• Activation tensor $A$: quantized token-wise — a separate scaling factor is computed for each row (each token position along the sequence length dimension). This respects the independence of different tokens in the batch.
• Weight tensor $W$: quantized channel-wise — a separate scaling factor is computed for each column (each output channel). This aligns with the matrix multiplication logic where each output channel is a dot product between $A$ and one column of $W$.

The paper's ablation (Figure 6d) explicitly compares four granularity combinations: coarse/coarse, fine/coarse, coarse/fine, and fine/fine. Fine-grained quantization for both tensors achieves the lowest loss. Crucially, applying coarse quantization to activations alone causes more severe accuracy degradation than applying it to weights alone — confirming that activation quantization is the harder problem and explaining the vector-wise granularity as a necessary adaptation from FP8's tensor-wise approach.

Dequantization after GeMM. After the FP4 matrix multiplication kernel computes a scaled quantized output, two scaling factors ($1 / \gamma_A$ and $1 / \gamma_W$) are applied to restore the result to the high-precision range. The paper notes this is performed "after the GeMM kernel execution" (Appendix C.2), meaning the FP4 tensor core sees the scaled quantized tensors and produces a scaled integer output, which is then rescaled.

---

The Differentiable Gradient Estimator (DGE)

The fundamental gradient problem. In a linear layer $Y = A \cdot W_q$ where $W_q = f(W)$ is the result of quantizing the weight tensor through a non-differentiable step function $f$, the chain rule for backpropagation gives:

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial W_q} \cdot \frac{\partial W_q}{\partial W}$

Since $f$ is an element-wise function, its derivative is also element-wise, producing a diagonal Jacobian matrix. This means the full gradient computation collapses to an element-wise (Hadamard) product:

$\frac{\partial L}{\partial W}[i, j] = \frac{\partial L}{\partial W_q}[i, j] \cdot f'(W[i, j])$

or in compact notation:

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial W_q} \odot f'(W)$

where $\frac{\partial L}{\partial W_q} = A^T \frac{\partial L}{\partial Y}$ is the gradient with respect to the quantized weight (computed by standard backpropagation through the matrix multiplication), and $f'(W)$ is the element-wise derivative of the quantization function.

What this equation computes: The true gradient of the loss with respect to the original (pre-quantization) weight is the gradient with respect to the quantized weight, multiplied element-wise by the derivative of the quantization function at each weight value. If $f$ is a step function (hard quantization), $f'$ is zero almost everywhere (flat plateaus) and infinite/undefined at step boundaries. This means the gradient $\partial L / \partial W$ would be zero for nearly all weights, preventing any learning.

Why this matters: The quantized weight $W_q$ is a deterministic function of the original weight $W$. The optimizer updates $W$, not $W_q$. If the gradient flowing to $W$ is zero because $f' = 0$ everywhere, the optimizer cannot change $W$, and training stalls. This is the mathematical statement of the "non-differentiability of quantization" problem.

The Straight-Through Estimator (STE) and why it fails at 4 bits. The STE (Bengio et al., 2013) solves this by assuming $f'(W) \equiv 1$ for all elements, effectively replacing the true gradient with:

$\frac{\partial L}{\partial W} \approx \frac{\partial L}{\partial W_q}$

This is equivalent to pretending that the quantization function is the identity $f(x) = x$ during backpropagation. For FP8, where quantization intervals are narrow relative to typical weight magnitudes, this approximation is adequate — the difference between the true gradient (which would include narrow spikes at step boundaries and zeros elsewhere) and the STE gradient (which is smooth) averages out over many training steps. At FP4, the quantization intervals are so wide that the identity assumption introduces systematic bias: weight values that sit near the middle of a quantization interval get the same gradient treatment as those near the edge, even though a small weight change would produce different quantization outcomes.

The DGE solution: a differentiable approximation for the gradient only. The DGE does not modify the forward pass — weights are still hard-quantized using the lookup table $f(x) = Q(x)$. Instead, it modifies the gradient computation by replacing $f'(W)$ with the derivative of a carefully chosen differentiable function that approximates the quantization behavior.

The differentiable approximation function (Equation 7):

$f(x) = \frac{\delta}{2} \cdot \left(1 + \operatorname{sign}\left(\frac{2x}{\delta} - 1\right) \cdot \left|\frac{2x}{\delta} - 1\right|^{\frac{1}{k}}\right)$

where $\delta$ is the width of the current quantization interval, $k > 1$ is a hyperparameter controlling approximation sharpness, and $\operatorname{sign}(\cdot)$ is the sign function.

What this function does: For a single quantization interval [0, δ], the term $\frac{2x}{\delta} - 1$ maps the interval to [−1, 1], centered at the midpoint δ/2. The power function $|t|^{1/k}$ with $k > 1$ creates a smooth S-curve that transitions from 0 to δ as $x$ moves from 0 to δ. The sign function provides symmetry around the interval midpoint. Visualized in Figure 3(a) for $k=5$, the resulting curve approximates the hard step function but with a continuous, non-zero derivative everywhere.

Across all 14 quantization intervals of E2M1 (spanning [−6, 6]), the complete differentiable function (Figure 3b) closely tracks the hard quantization staircase but with rounded corners at interval boundaries. The STE, by contrast, produces a straight diagonal line $f(x) = x$ in Figure 3b — a poor approximation to the stair-step pattern.

The derivative used for gradient correction (Equation 8):

$f'(x) = \frac{1}{k} \cdot \left|\frac{2x}{\delta} - 1\right|^{\frac{1}{k} - 1}$

where $\frac{2x}{\delta} - 1$ is the normalized position within the quantization interval, and the exponent $\frac{1}{k} - 1$ is negative for all $k > 1$.

What this computes: For each weight value, determine which quantization interval it falls into (which determines $\delta$). Compute the normalized distance from the interval midpoint $|2x/\delta - 1|$. Raise this to the power $(1/k - 1)$. Since this exponent is negative (e.g., $-4/5$ for $k=5$), the result is a large value near the interval midpoint (where the normalized distance is small) and decays as the weight moves toward the interval edges.

Operational meaning: Weights close to interval boundaries (where a small change might cause them to jump to the next quantization level) receive large correction factors — amplifying their gradients and encouraging the optimizer to move them decisively to one side or the other. Weights in the middle of intervals (where quantization error is already minimized) receive smaller correction factors. This selective amplification respects the actual sensitivity of the quantization outcome to weight changes, unlike the STE which treats all positions identically.

Why the DGE gradient is used only for weight updates: The DGE correction factor $f'(W_{\text{scaled}})$ is applied only to $\partial L / \partial W$ (Equation 6). It does not affect the gradient that flows back to the activations ($\partial L / \partial A$). The paper's implicit rationale is that activation gradients are used to update earlier layers through additional chain rule steps that introduce their own approximations, while weight gradients feed directly into the optimizer for the current layer. The DGE targets the most direct and quantifiable source of gradient error.

Clipping the correction factor (Section 3.1, Appendix C.3). Near the midpoint of each quantization interval, the term $|2x/\delta - 1|$ approaches zero. With a negative exponent $(1/k - 1)$, the derivative $f'(x)$ would mathematically approach infinity — a singularity. To prevent unbounded gradient spikes, the paper clips $f'(x)$ to a maximum value of 3.0:

"In practice, the magnitude of f'(x) is capped at 3.0 to prevent infinite gradient spikes at δ/2 point, impacting only a very small subset of elements."

Appendix C.3 provides the mathematical justification: this clipping is equivalent to using a smoothed absolute value $|x| \approx \sqrt{x^2 + \epsilon^2}$ with a small $\epsilon$, which bounds the derivative at $1/(k \cdot \epsilon^{1 - 1/k})$. The clipping constant of 3.0 corresponds to a specific choice of $\epsilon$ and $k$, and makes the implementation simpler than modifying the functional form.

Hyperparameter $k = 5$. The paper sweeps $k = 3$, $k = 5$, and $k = 10$ in the weight-only quantization ablation (Figure 6b, W4A8 setting). Lower $k$ values produce smoother gradients with less correction; higher $k$ values produce sharper approximations that more closely mimic hard quantization but at the cost of more volatile correction factors. The paper selects $k = 5$ as giving the best final performance, noting that "a larger k can better model the quantization function, but it can also lead to a more unstable correction term for the gradient."

Integration with vector-wise scaling (Appendix C.2). When weights are quantized with channel-wise scaling factors, the full weight quantization pipeline is:

$W_{\text{scaled}} = W \odot s_f$ $W_q^{\text{scaled}} = Q(W_{\text{scaled}})$ $W_q = W_q^{\text{scaled}} \odot \frac{1}{s_f}$

where $s_f$ is a vector of scaling factors with dimensions (1 × output_channels), broadcast along the input channel dimension, and $\odot$ is element-wise multiplication.

The gradient computation must account for these scaling and unscaling operations. The paper shows (Appendix C.2, Equations 17–22) that the scaling factors cancel out in the final gradient expression, yielding the same simplified form as Equation 6, with the only modification being that the weights fed into $f'$ must be the scaled weights:

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial W_q} \odot f'(W_{\text{scaled}})$

This cancellation is a fortunate property — if the scaling factors did not cancel, the DGE correction would need to account for channel-specific scale differences, complicating the implementation.

Computational overhead (Appendix B). The DGE adds "approximately 8 FLOPs per input element" in the backward pass of GeMM operations. For a 7B model with hidden size 4096 and sequence length 2048, this overhead across all four GeMM operations per Transformer layer (QKV projection, attention output, MLP up, MLP down) totals:

$8 \times (3bsh + bsh + 4bsh + 4bsh) = 96bsh \text{ FLOPs}$

where $bsh$ is batch size × sequence length × hidden size. The paper calculates this as only 32 / (6h + 5s + 36) ≈ 0.1% of total computation — a negligible cost that preserves the speedup from FP4 GeMM.

---

Outlier Clamping and Compensation (OCC)

The activation outlier problem. During LLM training, activation tensors exhibit distributions that make them "significantly more challenging to quantize than weight tensors" (Section 3.2). Appendix D provides extensive distributional evidence:

• Weight tensors (Figures 8–10): Approximately normal distributions with standard deviations of 0.006–0.028, small dynamic ranges, and rare outliers. These can be quantized to FP4 with manageable error even without sophisticated handling.
• Activation tensors (Figures 11–13): Heavy-tailed, irregular distributions. Post-attention layer norm outputs have standard deviations of 0.67–1.28. MLP down-projection outputs show standard deviations from 0.08 to 0.75 with values spanning ±25 or more. The distributions are often multi-modal and asymmetric.
• Channel-wise outlier patterns (Figure 14): A heatmap of GeLU activation output reveals "vertical light lines" — specific channel dimensions where nearly all token positions have extreme values, consistent with the channel-specific outlier phenomenon documented in the LLM quantization literature (Xiao et al., 2023; Wei et al., 2022).

Why outliers break absmax quantization. In the absmax scheme (Equation 1), the scaling factor $\gamma$ is $\text{MAX}_{\text{fp4}} / \max(|x|) = 6.0 / \max(|x|)$. If max(|x|) = 75 due to an outlier while typical values are in [−5, 5], then $\gamma = 0.08$. After scaling, typical values of 5 become 0.4 — which rounds to 0 or 0.5 in the FP4 grid (the two representable values nearest 0.4). A value of 1 becomes 0.08, rounding to 0. Almost all information in the tensor collapses to zero or near-zero quantization levels. Figure 4 (top panel) demonstrates this collapse with real data: the rich structure of the BF16 distribution across [−75, 75] is flattened into a single bin after FP4 quantization.

Why channel-wise quantization is insufficient. The paper acknowledges that "activation outliers in LLMs are typically channel-specific," and that channel-wise quantization (different scaling factors per channel) would reduce quantization loss. However, it notes this "conflicts with the computation structure of matrix multiplication in linear layers" (Section 5). Specifically, in a GeMM operation $Y = A W$ where $A$ is (s × ci) and $W$ is (ci × co), the matrix multiplication requires that the same scaling factor be applied across the inner dimension (ci) for each outer dimension element. If activations are quantized per-channel along ci, each scaling factor would affect a different subset of the dot product computation, requiring a more complex quantization scheme incompatible with standard tensor core operations.

The OCC solution: dynamic clamping at a high quantile. The paper's approach handles outliers in three stages:

Stage 1: Quantile-based dynamic range clipping (Equation 9). For a given pre-defined quantile $\alpha$ (set to 0.99 in the main experiments), identify the threshold values at the $\alpha$ and $1 - \alpha$ quantiles of the absolute activation values. Clamp all values outside this range:

$Y_c = \text{clamp}(Y, \text{max} = \alpha\text{-quantile}, \text{min} = (1-\alpha)\text{-quantile})$

This is applied token-wise — for each row (token position) in the activation tensor, the quantile thresholds are computed independently over that row's elements. This respects the independence of different sequence positions while accounting for the varying outlier patterns across tokens.

With $\alpha = 0.99$, approximately the top 1% of values by magnitude are clamped, leaving the bottom 99% unchanged. The clamping is symmetric (upper and lower tails) to preserve the distribution's zero-mean structure where present.

Stage 2: FP4 GeMM on the clamped tensor. The clamped tensor $Y_c$ has a drastically reduced dynamic range — the maximum value is now the $\alpha$-quantile rather than the absolute maximum. This means the scaling factor $\gamma = 6.0 / \text{max}(|Y_c|)$ is much larger, and typical values are no longer scaled into the underflow region. The quantized version $Q(Y_c)$ preserves the structure of the majority of activation values, as shown in Figure 4 (bottom panel) where the quantized distribution retains the multi-modal shape of the original.

Stage 3: Sparse outlier compensation. Clamping introduces error proportional to the magnitude of the removed outlier values. The paper recovers this through a sparse residual computation. The residual matrix is:

$\Delta Y = Y - Y_c$

This matrix is highly sparse — with $\alpha = 0.99$, only about 2% of elements are non-zero (the 1% in each tail). The non-zero elements are exactly the portions of the outlier values exceeding the clamping threshold.

During the GeMM computation, $Y_c$ is processed using FP4 tensor cores, while $\Delta Y$ is handled with a high-precision sparse matrix multiplication (Appendix B mentions FP8 for this). The final output is:

$Y_{\text{output}} = \text{FP4GeMM}(Q(Y_c), W_q) + \text{SparseGeMM}(\Delta Y, W)$

The sparse GeMM uses the full-precision (unquantized) weight matrix $W$ for the compensation path, since outliers interact with weights to produce potentially large contributions that would lose precision if quantized.

What the compensation achieves quantitatively (Table 1). The paper evaluates cosine similarity (SIM), mean squared error (MSE), and signal-to-noise ratio (SNR) between original and quantized activation tensors under different configurations:

Clamping alone improves cosine similarity from 92.19% to 98.83% and SNR from 8.31 to 14.25 — a substantial reduction in quantization error. Adding sparse compensation further improves similarity to 99.61% and SNR to 15.31 at the 99.9 quantile. Lowering the quantile to 99 or 97 (clamping more values) further reduces error, achieving 100% cosine similarity and SNRs of 18.38 and 20.88 respectively. This demonstrates a clear trade-off: more aggressive clamping produces higher accuracy but increases the density of the sparse compensation matrix.

Hyperparameter $\alpha = 0.99$. The paper selects $\alpha = 0.99$ as the operating point, corresponding to clamping only the top and bottom 1% of values, with a sparse matrix density of approximately 2% non-zero elements. The ablation in Figure 6c tests three settings:

• $\alpha = 0.999$: Clamping top 0.1%, sparse matrix 0.2% dense. Training loss close to but slightly above BF16.
• $\alpha = 0.99$: Clamping top 1%, sparse matrix 2% dense. Training loss matches the other settings with better stability.
• $\alpha = 0.97$: Clamping top 3%, sparse matrix 6% dense. Slightly better accuracy but at increased computational cost.

The paper justifies $\alpha = 0.99$ as "a better choice for comprehensive computational performance considerations" — the accuracy gain from $\alpha = 0.97$ does not justify the 3× increase in sparse matrix density (2% → 6%).

Why direct activation quantization diverges (Figure 6c, W8A4 curve). The paper reports that directly quantizing activations to FP4 without OCC causes "curve divergence, where the loss values turn into NaN after certain training steps." This is not gradual degradation but a hard failure. The mechanism is: outlier-dominated scaling factors cause underflow → most activation information is lost in the forward pass → backpropagated gradients are inaccurate or zero → optimizer takes destructive steps → activations become even more extreme in subsequent steps → feedback loop leads to numerical overflow (NaN). OCC breaks this loop at the source by preventing the initial information collapse.

Computational overhead (Appendix B). The OCC overhead comes from the sparse high-precision GeMM on the residual matrix $\Delta Y$. For each of the four GeMM operations per Transformer block, the additional computation scales with the sparsity $2(1 - \alpha)$:

$\text{OCC overhead} = 2(1 - \alpha) \times (12bsh^2) \text{ FLOPs per iteration}$

With $\alpha = 0.99$, this is $0.02 \times 12bsh^2 = 0.24 bsh^2$ FLOPs. The paper calculates this as approximately 5.6% of total computation for a 7B model — the largest overhead component, but still modest relative to the 4× theoretical speedup from FP4.

---

Mixed-Precision Training Infrastructure

Scope of FP4 quantization. The paper quantizes only the General Matrix Multiplication (GeMM) operations to FP4, consistent with the design of FP4 tensor cores in next-generation hardware. GeMM operations account for "over 95% of the computational workload" in LLM training (Section 4.1), so accelerating these operations captures nearly all possible speedup. The specific operations quantized are:

1. Attention query, key, value projections: $A \cdot W_Q$, $A \cdot W_K$, $A \cdot W_V$ (three separate GeMMs)
2. Attention output projection: $\text{AttentionOutput} \cdot W_O$ (one GeMM)
3. MLP up-projection: $A \cdot W_{\text{up}}$ (one GeMM)
4. MLP down-projection: $\text{GeLU}(A \cdot W_{\text{up}}) \cdot W_{\text{down}}$ (one GeMM)

The total is six GeMMs per Transformer layer, though the FLOPs are dominated by the MLP projections ($8bsh^2$ each for up and down in FP32) and the attention QKV projections ($6bsh^2$ total in FP32).

Non-GeMM operations remain at higher precision. The following operations are not quantized to FP4 and run in BF16 or FP16:

• Layer normalization (computes mean and variance, applies affine transform)
• Softmax in attention (exponential and normalization)
• GeLU activation function
• Attention score computation ($Q \cdot K^T$, though technically a matrix multiplication, may not use FP4 due to different tensor shapes and the need for subsequent softmax)
• Embedding lookups and output projection to vocabulary
• Loss computation (cross-entropy requires full precision for numerical stability)

This mixed-precision strategy follows the precedent set by FP8 training frameworks (Micikevicius et al., 2017; Peng et al., 2023) and is necessary because these operations involve non-linearities (softmax, GeLU), small-scale computations with disproportionate impact on training dynamics (layer norm), or operations where the FP4 quantization error would be catastrophic (loss computation requiring precise probability values).

FP8 gradient communication. Following Peng et al. (2023), gradient tensors are compressed to FP8 before being communicated across GPUs during distributed training (data parallelism, model parallelism). This reduces inter-GPU bandwidth usage. The gradient communication format uses FP8 E4M3 (the standard FP8 format with 4 exponent and 3 mantissa bits), providing a larger dynamic range than FP4 and sufficient precision for gradient values which are typically small and zero-centered. The paper does not quantize gradients to FP4, recognizing that gradient distributions often have extreme dynamic range across layers and that gradient communication bandwidth is typically not the dominant bottleneck compared to GeMM compute.

Mixed-precision Adam optimizer (Appendix of Peng et al., 2023). The Adam optimizer maintains three state tensors per parameter: first-order moment ($m$, the exponential moving average of gradients), second-order moment ($v$, the exponential moving average of squared gradients), and the parameter itself. The paper's mixed-precision scheme stores:

• First-order moments ($m$) in FP8 E4M3: These are gradient averages that typically have moderate dynamic ranges and can tolerate reduced precision since they are used in a ratio $m / (\sqrt{v} + \epsilon)$.
• Second-order moments ($v$) in FP16: These are squared gradient averages that are always non-negative and can span a large dynamic range. FP16 provides sufficient range (max ~65,504) for the squared terms while preserving precision.
• Master parameters in BF16 or FP32: The master copy of weights is maintained at higher precision to accumulate small updates that might be lost in FP4. The FP4 quantized copy is a temporary view used only during the forward GeMM.

Training hyperparameters (Section 4.1). All experiments use consistent hyperparameters across BF16 and FP4 training:

• Model architecture: LLaMA 2 (Touvron et al., 2023), trained from scratch
• Dataset: DCLM (Li et al., 2024), a comprehensive pretraining dataset
• Peak learning rate: $3 \times 10^{-4}$
• Learning rate schedule: Warm-up for 5% of total steps, cosine decay to 10% of peak over remaining 90%
• Weight decay: 0.1
• Adam optimizer: $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 1 \times 10^{-8}$
• Sequence length: 2048 tokens
• Batch size: 2048 sequences (approximately 4M tokens) for main experiments; reduced to 256 for 1.3B ablation experiments to accelerate convergence
• FP4-specific: $k = 5$ (DGE sharpness), $\alpha = 0.99$ (OCC clamp quantile)

Validation infrastructure. Because native FP4 tensor cores do not exist in current hardware, the paper uses Nvidia H-series GPUs' FP8 tensor cores to emulate FP4 computations. The FP8 tensor core supports the FP8 E4M3 format with dynamic range up to ±448, which encompasses the FP4 E2M1 range of ±6.0 — meaning FP4 values can be represented within the FP8 format without overflow. The emulation involves: (1) perform the quantization to FP4 using the lookup table CUDA kernel, (2) promote the 4-bit values to 8-bit representation for the FP8 tensor core, (3) execute the GeMM using FP8 hardware, (4) dequantize the result. The paper acknowledges this introduces "additional computational overhead due to extra precision casting and significantly prolong runtime" (Section 6), making it impossible to measure the actual speedup from FP4 — only to validate accuracy.

Theoretical speedup analysis (Appendix B). The paper provides a detailed FLOP breakdown per Transformer layer (Table 5) to estimate the theoretical speedup. For a standard Transformer with hidden size $h$, batch size $b$, and sequence length $s$:

• FP32 GeMM FLOPs: $24bsh^2 + 5bs^2h + 36bsh$
• FP4 GeMM FLOPs: $6bsh^2 + 5bs^2h + 36bsh$ (the $24bsh^2$ comes from 6 projections at $4bsh^2$ each in FP32, reduced by 4×; the non-GeMM terms remain unchanged)
• Ideal speedup: $\frac{24h + 5s + 36}{6h + 5s + 36}$

For a 7B model ($h = 4096$, $s = 2048$), ideal speedup is $3.12\times$.

After accounting for DGE overhead (0.1% of total) and OCC overhead (5.6% of total), adjusted speedup is:

$\frac{24h + 5s + 36}{6h + 24(1-\alpha)h + 5s + 68} = 2.95\times$

This analysis makes explicit the trade-off the paper accepts: approximately 5.5% of the theoretical speedup is sacrificed for accuracy preservation, resulting in a still-substantial ~3× speedup over FP32 for GeMM-dominated workloads.

4. Key Insights and Innovations

Innovation 1: The Two-Horse Problem — Weight Quantization and Activation Quantization Require Fundamentally Different Solutions

The paper's most important conceptual contribution is not any single technique but the diagnostic separation of the FP4 training problem into two qualitatively different failure modes that demand independent solutions. Prior quantization work — both for inference (PTQ/QAT) and for FP8 training — treated weights and activations as symmetrically challenging, applying the same quantization scheme and gradient estimation to both. This paper demonstrates that at 4-bit precision, this symmetry breaks down completely.

The evidence is stark. The weight-only ablation (Figure 6b, W4A8 setting) shows that directly quantizing weights to FP4 produces a visible but non-catastrophic loss gap — the model trains, just suboptimally. The activation-only ablation (Figure 6c, W8A4 setting) shows that directly quantizing activations to FP4 causes training divergence with loss going to NaN. These are not two points on a spectrum of "quantization difficulty." They are categorically different failure regimes: weight quantization causes slow degradation from accumulated gradient estimation errors, while activation quantization causes hard collapse from instantaneous information destruction in the forward pass.

This diagnostic distinction reframes the FP4 training challenge. The field's default assumption — inherited from FP8 work like Transformer Engine and FP8-LM — was that a unified quantization framework with a single scaling strategy and STE-based gradient estimation would extend downward to 4 bits with appropriate tuning. The paper shows this assumption is wrong at a mechanistic level. Weights fail because the STE gradient approximation becomes too coarse — the quantization function's derivative matters. Activations fail because outliers inflate scaling factors and cause underflow — the quantization function's domain matters. These are different problems with different root causes (gradient estimation accuracy vs. representational capacity collapse), and they require different solution mechanisms (DGE for weights, OCC for activations).

This reframing is significant beyond the specific techniques the paper proposes. It establishes that 4-bit quantization is not simply 8-bit quantization with tighter constraints — it crosses a threshold where the dominant error source shifts from "quantization noise" (small, approximately Gaussian errors treatable with the STE) to "structural information loss" (systematic collapse of the representable space). Future work on sub-4-bit training (FP2, ternary, binary) would need to grapple with this distinction from the start, rather than attempting to extend FP8 methods incrementally.

The appendices reinforce this diagnostic with distributional evidence (Figures 8–13): weight tensors are approximately normal with standard deviations of 0.006–0.028, while activation tensors are heavy-tailed with standard deviations reaching 1.28 and values spanning ±75. A single quantization framework cannot handle both distributions well, and the paper's approach of separate mechanisms for each tensor type is a direct consequence of recognizing this asymmetry.

Innovation 2: Gradient Correction Without Forward-Pass Modification — Decoupling Quantization Approximation from the Compute Path

A long-standing tension in quantization research is the trade-off between gradient accuracy and hardware efficiency. Prior differentiable quantization methods (Gong et al., 2019; Uhlich et al., 2019; Chen et al., 2019; Huang et al., 2022) attempted to solve the non-differentiability problem by replacing the quantization function entirely with a smooth approximation in both forward and backward passes. This is mathematically clean — the training objective becomes fully differentiable — but it eliminates the computational speedup that motivates quantization in the first place. If the forward pass doesn't use actual FP4 operations, you can't leverage FP4 tensor cores.

The DGE method breaks this trade-off through a simple but non-obvious design choice: apply the differentiable approximation only to the gradient computation, not to the forward pass. The forward pass retains hard quantization via the lookup table, preserving compatibility with FP4 hardware and the associated speedup. The backward pass adds a correction factor $f'(W_{\text{scaled}})$ — cheap to compute (0.1% overhead) — that captures the sensitivity of the quantization outcome to weight changes. The two passes are decoupled: what the hardware computes (hard quantization) and what the optimizer sees (corrected gradients) are different mathematical objects, and that's fine because the gradient's only job is to provide a descent direction, not to exactly reflect forward-pass mechanics.

This is a conceptual reframing of what gradient estimation through quantization should achieve. The STE implicitly assumes the goal is identity — make the gradient behave as if quantization didn't happen. The DGE's goal is different: make the gradient sensitive to where the weight sits within its quantization interval, because weight changes have different effects depending on proximity to the step boundary. The correction factor $f'(x) = (1/k) \cdot |2x/\delta - 1|^{1/k - 1}$ is large near interval midpoints (small denominator → large value) and small near edges. This means the gradient is amplified for weights that are "on the fence" — a small push could move them to the next quantization level — and attenuated for weights that are safely in the middle of a bin where small changes don't affect the quantized value. This is not an approximation of the identity; it's an approximation of the quantization function's derivative, which the STE discards entirely.

The significance extends beyond this paper. The DGE pattern — maintain hard nonlinearity in the forward pass for hardware efficiency, apply a differentiable surrogate only in the gradient — is generalizable to other non-differentiable operations in neural network training (sparsification, discretization, routing decisions). The key insight is that the forward and backward passes serve fundamentally different purposes and don't need to use the same function. This is already implicit in the STE (which explicitly uses different functions), but the DGE makes it explicit and computes a principled rather than trivial backward function.

The integration with vector-wise scaling factors (Appendix C.2) shows additional sophistication: the scaling and unscaling operations in channel-wise quantization cancel out in the gradient expression, meaning the DGE correction doesn't need to account for channel-specific scale differences. This cancellation is not obvious a priori — if it didn't hold, the DGE would require per-channel correction factors, significantly complicating the implementation. The paper identifies and exploits this mathematical property rather than forcing a complex workaround.

Innovation 3: Sparse Compensation as a Bridge Between Dynamic Range and Precision — Making Clamping Lossless in Expectation

Clamping outlier values to restrict dynamic range is a standard technique in signal processing and has been applied to neural network quantization before (e.g., in PTQ calibration where activation ranges are determined offline). What makes the OCC method distinctive is the recognition that clamping during training is fundamentally lossy unless the clamped-out information is recovered, and that a sparse compensation mechanism can achieve this recovery at modest cost because the clamped values are, by construction, extreme and few.

The conceptual move is to treat the clamping operation not as a preprocessing heuristic (the PTQ approach) but as a decomposition of the activation tensor into dense and sparse components:

$Y = Y_c + \Delta Y$

where $Y_c$ is the clamped, well-behaved tensor suitable for FP4 quantization (dense, FP4 GeMM), and $\Delta Y$ is the sparse outlier residual requiring high-precision handling (sparse, FP8/FP16 GeMM). This decomposition is not an approximation — it's an identity. The FP4 GeMM on $Y_c$ handles the bulk computation efficiently; the sparse GeMM on $\Delta Y$ recovers the exact contribution of the outliers. Together, they produce the mathematically correct output (up to the precision of the sparse GeMM), not an approximation to it.

This framing shifts outlier handling from "reduce the damage" to "redirect the computation." Prior approaches like SmoothQuant (Xiao et al., 2023) or QuaRot (Ashkboos et al., 2024) mitigate outlier effects by applying fixed transformations (channel-wise scaling, random rotations) that smooth the activation distribution before quantization. These are effective for inference where the transformation can be computed once and baked into the weights, but they fundamentally alter the computation — the model is computing a different function (the smoothed equivalent) than the original. OCC, by contrast, computes exactly the same function, just with a heterogeneous precision strategy: low precision for the dense core, high precision for the sparse extremes.

The dynamic nature of the clamping — re-computing quantile thresholds per token per training step — is what makes this practical for training where activation distributions evolve. The paper's distributional analysis in Appendix D (Figure 14, heatmap showing channel-specific vertical lines of outliers) confirms that outliers are structured (not random) and persistent (not transient), but their exact magnitudes shift during training. A fixed threshold would be either too aggressive early in training (destroying signal) or too lenient late in training (failing to suppress outliers). Dynamic quantile-based thresholds adapt automatically.

The quantitative evidence in Table 1 validates this decomposition framing. Clamping alone improves cosine similarity from 92.19% to 98.83% — a large gain from simply narrowing the dynamic range. Adding sparse compensation pushes similarity to 99.61% (at the 99.9 quantile) or 100% (at the 97 and 99 quantiles), with corresponding SNR improvements from 8.31 (no clamp) to 15.31 (clamp + compensate at 99.9) to 20.88 (at 97). The compensation effectively converts clamping from a lossy heuristic into a (nearly) lossless transformation, with the remaining error coming only from the accuracy of the sparse GeMM itself.

The trade-off the paper makes explicit — $\alpha = 0.99$ as the operating point balancing compensation overhead (2% sparse matrix density, 5.6% computation overhead) against accuracy — is a practical engineering choice, but the conceptual framework it embodies (dense-sparse decomposition for heterogeneous precision) is the deeper contribution. This framework generalizes beyond FP4: any precision regime where a small fraction of values dominate quantization error could benefit from a similar decomposition strategy, with the threshold quantile and sparse precision chosen to match the hardware's efficiency characteristics.

Innovation 4: Empirical Validation as the Primary Contribution — Feasibility at Scale Trumps Algorithmic Novelty

This paper's most distinctive intellectual contribution may be what it doesn't claim: it does not claim to have invented either DGE or OCC as fundamentally new mathematical techniques. Differentiable approximations to quantization functions predate this work by years (the paper cites Gong et al., 2019; Uhlich et al., 2019; Chen et al., 2019). Clamping with sparse recovery is, at its core, a signal processing idea. What the paper claims — and what the evidence supports — is that combining these techniques into a complete training framework and validating it at scale (up to 13B parameters, 100B tokens) constitutes a meaningful scientific contribution.

This is a validation-as-contribution framing that is increasingly important in ML systems research. The field has a proliferation of proposed quantization methods that work on small models, on synthetic benchmarks, or in inference-only settings, but lack evidence that they transfer to the regime that matters: large-scale pretraining from scratch. The paper's training curves (Figure 5) showing FP4 tracking BF16 across three model scales (1.3B, 7B, 13B) and 100B tokens, with zero-shot downstream evaluations (Table 2) confirming competitive performance across nine benchmarks, constitute the first such evidence for FP4. Prior work on 4-bit training (Sun et al., 2020, for CNNs; Xi et al., 2023, for INT4 fine-tuning) never demonstrated LLM pretraining at this scale.

The negative results in the ablation studies strengthen this contribution. The paper shows that direct FP4 quantization fails (Figure 6a: large loss gap; Figure 6c: NaN divergence), that coarse-grained quantization fails (Figure 6d), and that the ReST$^{EM}$ training approach actually degrades revision model performance when applied to FP4 (no, wrong paper — ignore). Each negative result validates that FP4 training is genuinely hard and that the proposed techniques are necessary, not merely helpful. This transforms the paper from "here are some tricks that help" to "here are the minimum requirements for FP4 training to work."

The theoretical speedup analysis (Appendix B, 2.95× for 7B models) is also part of this validation contribution. By quantifying the expected gains after accounting for all overhead, the paper provides a concrete target for hardware designers and a benchmark against which future FP4 implementations can be measured. This is forward-looking validation — the paper can't measure speedup on current hardware, but it establishes what speedup is plausible given the computational structure.

This contribution type — "FP4 LLM pretraining is feasible, here is the evidence, here is what it costs" — is arguably more valuable to the field than a novel quantization algorithm with no scaling evidence. It de-risks the hardware investment in FP4 tensor cores by showing that the software exists to use them. It also establishes baseline expectations: FP4 training does not require exotic architectures or training procedures; it works with standard LLaMA models, Adam optimizer, and cosine learning rate schedules, using the same hyperparameters as BF16 training. This compatibility is itself a finding — it means FP4 training can be adopted incrementally rather than requiring a wholesale redesign of the training stack.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the DCLM dataset (Li et al., 2024) for pretraining — a comprehensive dataset designed for language model training from scratch. For downstream evaluation, the paper uses a suite of nine benchmarks accessed through the lm-evaluation-harness library (Gao et al., 2024): PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), ARC-Challenge and ARC-Easy (Clark et al., 2018), BoolQ (Clark et al., 2019), LogiQA (Liu et al., 2021), SciQ (Welbl et al., 2017), and Lambada (Paperno et al., 2016). Perplexity is evaluated on Lambada OpenAI, Lambada standard, Pile 10k (Gao et al., 2020), and Wikitext (Merity et al., 2017). The paper uses LLaMA 2 models trained on 100B tokens of DCLM for the main experiments, with a 10B-token subset used for ablation studies on the 1.3B model to accelerate convergence.

• Base model(s). The LLaMA 2 architecture (Touvron et al., 2023) is used across all experiments, trained from scratch at three scales: 1.3B, 7B, and 13B parameters. The paper states that model training is conducted "from scratch using the DCLM dataset." LLaMA 2 was chosen as the "widely recognized" architecture (Section 4.1), providing a standard baseline against which quantization methods can be compared. The consistent use of a single architecture family means the paper tests whether FP4 training generalizes across model scales within that family, but does not test across different architectural designs (e.g., non-LLaMA attention patterns, different activation functions).

• Metrics. Three categories of metrics are reported. Training loss is the primary in-training metric, plotted as a function of training tokens across all experiments (Figures 1, 5, 6). Final training losses at 100B tokens are reported for each model size. Zero-shot downstream accuracy (Table 2) is reported as the percentage of correct answers on each benchmark, with an average across all nine benchmarks. Perplexity (Table 3) is reported for four held-out text datasets, with lower values indicating better language modeling. All downstream evaluations are conducted in a zero-shot manner without task-specific fine-tuning, testing whether FP4 training preserves the model's general language capabilities rather than just training loss.

• Baselines. The paper uses four baseline configurations. BF16 mixed-precision is the primary reference point — the standard training scheme where GeMM operations use BF16 and non-GeMM operations use various higher precisions, representing the current state-of-the-art for LLM training without FP4. MS-AMP FP8 (Peng et al., 2023) and Transformer-Engine FP8 (Nvidia, 2022) serve as intermediate baselines in the precision ablation (Figure 6a), demonstrating that FP8 methods achieve accuracy comparable to BF16 and establishing the gap that FP4 must close. Direct-cast FP4 (labeled W4A4 in Figure 6a) quantizes both weights and activations to FP4 using standard absmax scaling and the STE, with no DGE or OCC — this is the "naive FP4" baseline that demonstrates the severity of the FP4 training problem. For the weight-only and activation-only ablations, additional baselines include W4A8 (FP4 weights, FP8 activations) and W8A4 (FP8 weights, FP4 activations) without the proposed techniques.

• Generation budget / compute accounting. Training is measured in tokens processed (not wall-clock time or FLOPs), with main experiments training on 100B tokens and ablations on 10B tokens. This is the natural unit for pretraining experiments where the goal is to compare convergence behavior across precision formats. The paper explicitly states that "Hyperparameters remain consistent across precision settings for fair comparison" — same learning rate schedule (warm-up for 5% of steps, cosine decay to 10% of peak), same peak learning rate (3 × 10⁻⁴), same weight decay (0.1), same Adam parameters (β₁ = 0.9, β₂ = 0.95, ε = 10⁻⁸), same sequence length (2048 tokens), and same batch size (2048 sequences, approximately 4M tokens for main experiments; reduced to 256 for 1.3B ablations). This "identical hyperparameters" approach means any loss differences can be attributed to the precision format rather than optimizer tuning. However, it also means the paper does not explore whether FP4 training might benefit from precision-specific hyperparameter adjustments — the comparison is strictly "BF16 hyperparameters applied to FP4."

• Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, or confidence intervals. Training curves (Figures 5, 6) show single-run trajectories, and the table results represent point estimates from a single trained model per configuration. This is standard practice for large-scale LLM training experiments where the computational cost of multiple training runs is prohibitive, but it means that small differences between configurations (e.g., the 0.06 loss gap between BF16 and FP4 at 1.3B, the 0.2–0.5% accuracy differences in Table 2) cannot be distinguished from run-to-run variance. The paper implicitly relies on the consistency of results across three model scales (1.3B, 7B, 13B) as a form of informal replication — if FP4 were genuinely worse than BF16 in a way that matters, the gap would likely grow with model scale rather than remaining small and stable. The FP4 simulations on H100 GPUs use the FP8 tensor cores, which the paper states "encompass FP4's dynamic range and enable accurate simulation," but no quantitative analysis of simulation fidelity (e.g., bit-exactness compared to hypothetical native FP4 hardware) is provided.

Main Quantitative Results

Training Loss: FP4 Tracks BF16 Across Model Scales

Figure 5 presents the central empirical claim of the paper: training loss curves for LLaMA models at 1.3B, 7B, and 13B parameters, comparing BF16 mixed-precision against the full FP4 framework (W4A4 with DGE and OCC). The curves for BF16 and FP4 "largely overlap across different model sizes, with the FP4 curve exhibiting a slightly higher training loss compared to the BF16 curve" (Section 4.2).

The specific loss values at 100B tokens (read from Figure 5 and quoted in the text):

The gap is small and does not grow with model size — it's approximately 0.06–0.10 across all three scales, representing about 2–5% relative increase in loss. If FP4 were introducing systematic errors that compound during training, the gap would widen as training progresses and as model size increases. The fact that the curves remain parallel throughout training (from early tokens through 100B) and the gap is roughly constant across scales suggests the quantization error is acting as a small constant noise source rather than a diverging bias.

However, the 7B model shows a slightly larger gap (0.10) than the 1.3B model (0.06) or the 13B model (0.09). This non-monotonic pattern is not explained in the paper. Possible interpretations: (a) it's within run-to-run variance and would disappear with multiple seeds; (b) the 7B model happens to have activation distributions that interact slightly worse with the OCC parameters (α = 0.99 was tuned on the 1.3B ablation); or (c) there's a scale-dependent effect that the paper didn't fully characterize. Without multiple training runs, this remains an unresolved observation.

An important note: the paper reports training loss (loss on the training data being processed), not validation loss. Training loss is expected to be lower than validation loss and can mask overfitting. The downstream evaluations (Tables 2 and 3) are the paper's main check against overfitting — if FP4 models were memorizing the training data at the expense of generalization, this would show up as worse zero-shot performance despite similar training loss. The downstream results (discussed next) suggest this is not happening, but the absence of a held-out validation loss curve is a gap.

Downstream Accuracy: FP4 Models Are Competitive with BF16

Table 2 reports zero-shot accuracy on nine benchmarks for models trained with BF16 and FP4 at all three scales. The headline finding: average accuracy is comparable across precision formats, with FP4 models actually scoring slightly higher on average at 7B and 13B.

Several patterns are notable:

FP4 models sometimes outperform BF16 models. At 7B, the FP4 average (54.42) is 0.55 points above BF16 (53.87). At 13B, the FP4 average (54.95) is 0.51 points above BF16 (54.44). At 1.3B, FP4 is 0.10 points below BF16. These differences are small and likely within the noise floor of single-model evaluation, but the consistent pattern of FP4 not being systematically worse is the key finding. The paper does not claim FP4 is better than BF16 — and it shouldn't, since quantization can only preserve or lose information — but the fact that FP4 models are not systematically degraded across nine diverse benchmarks is strong evidence that the training framework is not introducing harmful biases.

Individual benchmark variation is substantial. Within any model size, FP4 and BF16 differences on individual benchmarks can be as large as 2–5 points (e.g., BoolQ at 13B: 57.40 BF16 vs. 55.90 FP4, a 1.5-point gap; LogiQA at 13B: 29.65 BF16 vs. 30.88 FP4, a 1.23-point gap in the opposite direction). This is typical of zero-shot evaluation — benchmark scores are noisy, and small differences should not be over-interpreted. The consistent trend across nine benchmarks (no systematic degradation pattern) is more informative than any individual comparison.

Scaling trends are preserved. Both BF16 and FP4 models show the expected pattern of improving accuracy with model size. The 13B FP4 model (54.95 average) outperforms the 7B BF16 model (53.87 average) and the 1.3B BF16 model (53.23 average). This confirms that FP4 training preserves the benefits of scaling — a 13B model trained in FP4 is genuinely better than a 7B model trained in BF16 on these benchmarks, at least for this training budget (100B tokens). This is important because it means FP4 does not "cap" the effective model capacity or create a ceiling that larger models can't break through.

The evaluation is zero-shot, not fine-tuned. All results come from evaluating the pretrained base models directly on downstream tasks without any task-specific training. This tests the model's in-context learning ability and general knowledge acquired during pretraining. It does not test whether FP4-trained models fine-tune as well as BF16 models on downstream tasks — this would require additional experiments not conducted in the paper.

Perplexity: FP4 Models Match or Slightly Exceed BF16

Table 3 reports perplexity on four held-out text datasets. Lower perplexity indicates better language modeling. The pattern mirrors the accuracy results: FP4 models achieve comparable or slightly better perplexity than BF16 models.

The FP4 1.3B model has slightly lower average PPL than BF16 (36.86 vs. 37.38), and the 13B FP4 model is essentially tied (33.99 vs. 33.69). The 7B FP4 model is marginally worse (35.62 vs. 35.06). Again, these differences fall within the expected variance of single-model training runs and do not indicate a systematic degradation.

An interesting pattern: Lambada standard shows substantially lower perplexity for FP4 at 1.3B (23.07 vs. 25.10) and 13B (21.62 vs. 22.45), but higher at 7B (24.42 vs. 23.33). Lambada OpenAI shows higher perplexity for FP4 at 1.3B and 13B but slightly lower at 7B. These inconsistencies across scales suggest that individual benchmark PPL differences are dominated by noise rather than systematic FP4 effects, which is exactly what you'd expect if FP4 training is genuinely matching BF16 quality.

As with training loss, perplexity scaling trends are preserved: larger models achieve lower perplexity regardless of training precision (13B FP4 at 33.99 beats 7B BF16 at 35.06), confirming that FP4 training does not fundamentally limit model quality.

Precision Ablation: FP8 Works, Naive FP4 Fails, DGE+OCC Closes the Gap

Figure 6a compares training curves across five precision configurations on the 1.3B model with 10B training tokens:

• BF16 (baseline): Lowest loss throughout training.
• MS-AMP FP8 (Peng et al., 2023) and TE FP8 (Nvidia, 2022): Both closely track BF16, with MS-AMP FP8 showing a slightly wider gap than TE FP8 but both remaining within a small margin. This validates that FP8 training works as expected on this setup.
• Direct-cast FP4 (W4A4): Quantizing both weights and activations to FP4 using standard absmax scaling and STE, without DGE or OCC. The loss curve sits significantly higher than BF16 — by approximately 1–2 points throughout the 10B-token training run. This is the "FP4 doesn't work" baseline that motivates the paper.
• FP4 with DGE+OCC: The proposed full framework. The curve closely tracks the FP8 methods and sits just slightly above BF16, replicating the pattern from the main 100B-token experiments at a smaller scale.

The gap between direct-cast FP4 and FP4 with DGE+OCC is the paper's primary quantitative evidence for the necessity of the proposed techniques. Without DGE and OCC, FP4 training is substantially worse than BF16. With them, it's competitive. This is not a subtle improvement — it's the difference between unusable and usable.

An important note on Figure 6a: the direct-cast FP4 curve is only shown for approximately 4B tokens before it terminates. The figure caption does not explain whether this is because the run diverged (NaN) at that point, was stopped early due to poor performance, or is simply cropped for visual clarity. If the run diverged, this would strengthen the paper's claim that OCC is necessary for training stability (not just accuracy). If it was stopped early due to poor performance, the necessity claim is about final quality rather than stability. The paper's text in Section 4.3 does not clarify this, saying only that "directly-casted FP4 has a significant training loss gap."

Weight-Only Quantization: DGE Provides Clear Improvement at Modest Cost

Figure 6b isolates the weight quantization problem by keeping activations at FP8 (W4A8) and comparing three DGE configurations against direct quantization (no DGE) and the BF16 baseline:

• W4A8 without DGE: Shows a visible loss gap below BF16 and the DGE configurations — weights quantized to FP4 with STE gradient estimation lose information that the optimizer can't recover.
• W4A8 + DGE (k=3): Improves over no-DGE, with the loss curve sitting closer to BF16.
• W4A8 + DGE (k=5): Further improvement, achieving the best final loss among the FP4 weight configurations.
• W4A8 + DGE (k=10): Slightly worse final performance than k=5 — the curve is between k=3 and k=5 at the end of training.

The paper interprets the k=10 result as evidence of an instability-accuracy trade-off: "a larger k can better model the quantization function, but it can also lead to a more unstable correction term for the gradient" (Section 4.3). This is consistent with the mathematical form of the DGE correction: larger k produces sharper approximations to the step function (better modeling of the actual quantization), but also more extreme gradient corrections near interval midpoints (potentially causing training instability). The paper selects k=5 as the best operating point based on this empirical sweep.

A noteworthy observation: "direct quantizing weight into 4-bit doesn't introduce a substantial training loss gap, suggesting that weights are easier to quantize than activations" (Section 4.3). The W4A8 curve without DGE is worse than BF16, but the gap is relatively small — perhaps 0.1–0.2 in loss — and the model trains stably. This contrasts sharply with the activation-only setting discussed next, where naive quantization causes divergence. This asymmetry is the empirical basis for the paper's two-problem diagnostic: weights need better gradients, activations need outlier handling.

The DGE hyperparameter sweep only tests integer values (k=3, 5, 10). A finer sweep (e.g., k=4, 6, 7, 8) might reveal that the optimal k is not exactly 5, or that the performance surface is relatively flat for moderate k values (suggesting robustness to this hyperparameter). The paper does not explore this.

Activation-Only Quantization: OCC Prevents Divergence

Figure 6c isolates the activation quantization problem by keeping weights at FP8 (W8A4) and comparing three OCC configurations against direct quantization and the BF16 baseline:

• W8A4 without OCC (direct quantization of activations to FP4): The loss curve "turn into NaN (Not a Number) after certain training steps" — this is a hard training divergence. The figure shows the curve tracking BF16 for a short period before shooting upward.
• W8A4 + OCC (α=0.999): Clamping at the 99.9th percentile (top 0.1% values clamped, 0.2% sparse matrix density). Training proceeds stably, but the loss gap below BF16 is visible.
• W8A4 + OCC (α=0.99): Clamping at the 99th percentile (top 1% clamped, 2% sparse density). The loss curve closely tracks OCC(0.999) with slightly better stability.
• W8A4 + OCC (α=0.97): Clamping at the 97th percentile (top 3% clamped, 6% sparse density). The loss curve shows the best convergence — closest to BF16 among the FP4 configurations — but at increased computational cost due to the denser sparse compensation matrix.

The key finding from this ablation is that OCC transforms an impossible training problem (divergence) into a solvable one (convergence with small loss gap). The mechanism is clear from Figure 4 and Table 1: without clamping, outlier-induced scaling factor collapse destroys nearly all activation information, causing gradient noise that leads to divergence. With clamping, the majority of activation values are preserved through quantization, and training proceeds normally.

The α sweep reveals the accuracy-efficiency trade-off: "a smaller α implies a stronger compensation, but at an increased computational cost" (Section 4.3). The paper selects α=0.99 as the operating point based on "comprehensive computational performance considerations" — the loss improvement from α=0.97 (denser compensation) doesn't justify the 3× increase in sparse matrix operations (6% vs. 2% non-zero elements).

However, the paper does not provide quantitative runtime measurements for different α values. The theoretical overhead calculation in Appendix B estimates OCC overhead as 2(1−α) × (12bsh²) FLOPs, yielding 5.6% overhead at α=0.99 and approximately 16.8% at α=0.97. But this doesn't account for hardware efficiency factors: sparse matrix multiplication on GPUs is typically much less efficient per non-zero element than dense matrix multiplication, so the actual runtime impact could be larger. The claim that α=0.99 is optimal for "computational performance" is based on theoretical FLOP counting, not measured wall-clock time.

Quantization Granularity: Vector-Wise Scaling Is Necessary for FP4

Figure 6d compares four granularity configurations for FP4 quantization, holding DGE and OCC fixed:

• Coarse/coarse: Tensor-wise scaling for both weights and activations. Loss is highest among all configurations, with a substantial gap below BF16.
• Fine/coarse: Token-wise activations, tensor-wise weights. Loss improves over coarse/coarse but still shows a visible gap.
• Coarse/fine: Tensor-wise activations, channel-wise weights. Loss is in the same range as fine/coarse — both configurations with one coarse and one fine dimension perform similarly in aggregate.
• Fine/fine: Token-wise activations, channel-wise weights. This is the best configuration, with loss closest to BF16.

The paper draws several conclusions from this experiment. First, "tensor-wise scaling in FP4 introduces significant errors" — the coarse/coarse configuration is substantially worse than any configuration with at least one fine-grained dimension. This contrasts with FP8 training, where "coarse-grained tensor-wise quantization achieves sufficient accuracy" (Section 4.3), establishing that FP4's limited dynamic range makes coarse scaling factors inadequate.

Second, "applying coarse-grained quantization to activations alone result in more severe accuracy degradation than applying it to weights alone" — although the difference between fine/coarse and coarse/fine is not dramatic in the figure, the paper explicitly notes this asymmetry, consistent with the broader finding that activation quantization is the harder problem.

Third, the paper aligns the granularity choices with GeMM computation structure: token-wise activation quantization respects the independence of different sequence positions, and channel-wise weight quantization aligns with the matrix multiplication logic where each output channel is a dot product between the activation row and one weight column. This is not an arbitrary design choice but a principled alignment with the mathematics of matrix multiplication.

The granularity ablation is conducted on the full FP4 framework (with DGE and OCC) rather than isolating granularity from the other techniques. This means the results show the effect of granularity in the presence of DGE and OCC — it's possible that without these techniques, granularity would have an even larger effect (if poor granularity and poor gradient estimation compound) or a smaller effect (if other errors dominate). The paper does not explore these interactions.

Ablation Studies and Robustness Checks

Precision comparison (Figure 6a): Five configurations (BF16, MS-AMP FP8, TE FP8, direct-cast FP4, FP4 with DGE+OCC) compared on the 1.3B model with 10B tokens. Both FP8 methods track BF16 closely, confirming that FP8 training works as expected and establishing the baseline that FP4 must match. Direct-cast FP4 shows a large loss gap (roughly 1–2 points above BF16), while FP4 with DGE+OCC closes this gap to within approximately 0.1–0.2 of BF16. This is the experiment that directly validates the paper's central claim: the proposed techniques are necessary and sufficient to make FP4 training viable.

DGE hyperparameter sweep (Figure 6b): Three values of k (3, 5, 10) tested in W4A8 setting. All DGE configurations improve over no-DGE. k=5 achieves the best final performance, with k=10 performing slightly worse despite better modeling the quantization function — evidence for the stability-accuracy trade-off that the paper explicitly discusses. The differences between k values are modest, suggesting the method is not extremely sensitive to this hyperparameter, though the paper does not test values between 5 and 10 or above 10 to confirm this.

OCC hyperparameter sweep (Figure 6c): Three values of α (0.999, 0.99, 0.97) tested in W8A4 setting. All OCC configurations prevent the divergence seen without OCC. Lower α (more aggressive clamping, denser compensation) produces better training loss but at increased computational cost. α=0.99 is selected as the operating point. The paper does not test α values between 0.99 and 0.999 (which might offer a better accuracy-efficiency trade-off) or values below 0.97 (where the compensation matrix might become dense enough to negate the FP4 speedup entirely).

Quantization granularity (Figure 6d): Four combinations of coarse/fine granularity for weights and activations, tested with the full FP4 framework. Fine-grained quantization for both tensors achieves the best results. Coarse activation quantization is more damaging than coarse weight quantization. The paper concludes that vector-wise (token-wise for activations, channel-wise for weights) scaling is necessary for FP4, in contrast to FP8 where tensor-wise scaling suffices.

Cosine similarity, MSE, and SNR analysis (Table 1): Quantitative fidelity metrics between original and quantized activation tensors across five OCC configurations. Clamping alone improves cosine similarity from 92.19% to 98.83% and SNR from 8.31 to 14.25. Adding compensation further improves similarity to 99.61%+ and SNR to 15.31+ depending on α. These metrics provide a mechanistic explanation for why OCC prevents divergence: the information preservation improvement is large and measurable even before training dynamics enter the picture. The metrics are computed on activation tensors from the 1.3B model at 30,000 training iterations, providing a single-snapshot validation of the OCC effect.

Absence of DGE-only and OCC-only full-system evaluations: The paper tests DGE in weight-only (W4A8) and OCC in activation-only (W8A4) settings, but does not report full FP4 training (W4A4) with only DGE or only OCC. In principle, a W4A4 experiment with DGE but without OCC would test whether better weight gradients can compensate for activation collapse — the paper's diagnostic framework predicts this would diverge, but this prediction is not experimentally verified. Similarly, W4A4 with OCC but without DGE would test whether outlier handling alone makes FP4 training viable. These missing ablation points would strengthen the paper's claim that both techniques are necessary for the full W4A4 setting. The current ablation design, while informative, only demonstrates necessity in isolation (DGE helps when weights are the bottleneck; OCC prevents divergence when activations are the bottleneck), not that both are required simultaneously.

No ablation on the interaction between DGE and OCC: The paper applies both techniques simultaneously in the main experiments but does not test whether combining them introduces any unexpected interactions. For instance, if DGE produces larger weight gradients (because of the correction factor near interval boundaries), this could increase activation magnitudes in subsequent layers, making the OCC clamping more aggressive and potentially creating a feedback loop. No evidence of such interactions is reported, but they are not systematically excluded either.

No sensitivity analysis on precision format (E2M1 vs. alternatives): The paper adopts the E2M1 format based on prior work (Rouhani et al., 2023b;a) but does not compare it against E1M2 or E3M0 (listed in Table 4 and Figure 7 of the Appendix). E1M2 offers finer quantization intervals (0.5 spacing) but a smaller dynamic range (±3.5), while E3M0 offers a larger range (±16) but coarser intervals. It's plausible that one of these alternative formats might interact differently with the OCC clamping threshold or the DGE gradient approximation. The paper's format choice is justified by a qualitative trade-off argument ("balanced trade-off between dynamic range and quantization precision"), but no empirical comparison is provided.

No robustness check across different random seeds: All training curves and evaluation results come from single training runs. For the 1.3B ablation experiments (10B tokens), running multiple seeds would be computationally feasible and would provide a measure of whether the observed gaps between configurations (e.g., 0.06 loss gap at 1.3B) exceed run-to-run variance. The paper's argument that three model scales provide implicit replication is reasonable for the main claim (FP4 ≈ BF16), but weaker for the fine-grained comparisons in the ablations.

No evaluation of training dynamics (gradient norm, weight distribution evolution): The paper focuses exclusively on loss curves and downstream metrics. Training dynamics metrics — gradient norm trajectories, weight distribution statistics over the course of training, activation outlier frequency over time — would provide additional evidence for the mechanisms the paper claims. For instance, showing that DGE reduces the variance of weight gradient norms compared to STE would strengthen the claim that DGE provides "more accurate" gradients. Showing that OCC keeps activation distributions within FP4's effective range throughout training (not just at the 30K-iteration snapshot in Table 1) would strengthen the claim that OCC prevents information collapse dynamically.

Critical Assessment

The experiments in this paper demonstrate that FP4 training of LLaMA 2 models up to 13B parameters and 100B tokens is feasible with minimal accuracy degradation when using the proposed DGE and OCC techniques. The evidence is consistent across model scales and evaluation dimensions, and the ablation studies isolate the contribution of each technique to the overall framework. However, several important boundaries on what these experiments actually establish must be noted.

What the experiments demonstrate rigorously:

The central empirical claim — that the proposed FP4 framework achieves training loss and downstream performance comparable to BF16 — is supported by consistent evidence across all reported experiments. Figure 5 shows parallel loss curves across 1.3B, 7B, and 13B models. Table 2 shows zero-shot accuracy within ±0.5 points of BF16 on average. Table 3 shows comparable or slightly better perplexity. Figure 6a shows that the combination of DGE and OCC closes the gap between naive FP4 (large loss gap) and BF16/FP8 (good convergence). These results are internally consistent and collectively paint a clear picture: FP4 training works under these conditions.

The ablation studies (Figures 6b–6d) provide convincing evidence that the proposed techniques address specific failure modes. Figure 6b shows DGE improving weight-only FP4 convergence, with k=5 emerging as the best among tested values. Figure 6c shows OCC preventing activation-only FP4 divergence, with the α parameter controlling the accuracy-efficiency trade-off. Figure 6d shows that vector-wise quantization granularity is necessary at FP4 precision. These ablated experiments are well-designed to isolate individual effects.

The Appendix D distributional analysis (Figures 8–14) provides qualitative mechanistic evidence: weight tensors are approximately normal and narrow, activation tensors are heavy-tailed and wide, and outliers exhibit channel-specific structure. The heatmap in Figure 14 directly visualizes the channel-wise activation outlier phenomenon that OCC is designed to address. This evidence bridges the mathematical motivation (Section 3.2) and the empirical results (Figure 6c), strengthening the paper's mechanistic claims.

What the experiments do not demonstrate:

1. The claim that FP4 training works "with the same hyperparameters" is accurate but potentially misleading about optimality. The paper uses BF16 hyperparameters for FP4 training without tuning. This is a valid fairness choice for comparison, but it does not establish that these are the best hyperparameters for FP4. It's possible that FP4 training would benefit from a different learning rate, warmup schedule, or weight decay — and that with proper tuning, the small remaining loss gap (0.06–0.10) could close further. The paper's framing of "same hyperparameters" as a demonstration of robustness is legitimate, but readers should not interpret this as proof that BF16 hyperparameters are optimal for FP4.

2. The scalability claim is limited to 13B parameters and 100B tokens. While this is a substantial scale relative to typical quantization research, it falls well short of frontier LLM training regimes (hundreds of billions to trillions of parameters, trillions of training tokens). The paper acknowledges this limitation explicitly (Section 6): "due to constraints on computational resources, we have not yet extended our experiments to extremely large-scale models or to datasets comprising trillions of tokens." The possibility of emergent failure modes at larger scales — activation outlier patterns that grow with model depth, gradient estimation errors that compound in deeper networks, interactions with distributed training strategies not tested here — cannot be excluded based on current evidence.

3. The framework is validated on a single model architecture (LLaMA 2). While LLaMA 2 is widely used and representative of modern decoder-only Transformer architectures, different architectural choices (different activation functions, different normalization placements, different attention mechanisms) might exhibit different quantization characteristics. The paper does not claim universality, but the absence of multi-architecture evaluation means the findings could be partially specific to LLaMA 2's design — particularly its use of pre-layer norm (which the paper doesn't explicitly discuss but which affects activation distributions) and SwiGLU activations in the MLP.

4. The experiments simulate FP4 using FP8 hardware, not native FP4 tensor cores. Section 6 acknowledges this as a limitation: "we are unable to directly measure the potential speedup and energy efficiency gains achievable with native FP4 support." The accuracy validation is reliable (FP8 can represent all FP4 values without loss), but the speedup estimates (2.95× for 7B models in Appendix B) are theoretical calculations based on FLOP counting, not measured wall-clock time. Real hardware will introduce additional effects — memory bandwidth limitations, quantization/dequantization latency, sparse matrix multiplication efficiency — that could significantly affect actual performance. The paper is transparent about this, but the distinction between "accuracy validated" and "speedup theoretical" is crucial for assessing the practical significance of the work.

5. The experiments do not evaluate training stability under different initialization schemes or data orderings. All experiments use a fixed random seed (implicitly, since only single runs are performed) and a single data ordering from the DCLM dataset. Training stability at the edge of numerical precision can be sensitive to these factors — an unlucky initialization or a batch with unusual outlier patterns might trigger divergence that a different seed would avoid. The paper demonstrates that FP4 training can be stable; it does not demonstrate that it will be stable under all reasonable conditions. Multiple-seed experiments at the 1.3B scale would address this concern at modest computational cost.

6. The ablation design could be stronger. The paper tests DGE and OCC in isolation (weight-only and activation-only settings respectively) but never tests the full W4A4 configuration with only one of the two techniques — leaving open the question of whether both are strictly necessary for full FP4 training, or whether one technique dominates. The interaction between the two techniques is not explored. Additionally, the absence of downstream evaluation for the ablation experiments (all ablation metrics are training loss only) means we don't know whether the loss improvements from DGE/OCC translate to better downstream task performance or merely better training set fitting.

7. The cost of difficulty estimation is zero in this context (this paper has no difficulty estimation), but the cost of the per-step operations is estimated theoretically, not measured. Appendix B provides a detailed theoretical FLOP analysis estimating 2.95× speedup for 7B models, accounting for DGE (0.1% overhead) and OCC (5.6% overhead). However, theoretical FLOP counting does not capture hardware-specific factors: the sparse matrix multiplication in OCC may be significantly less efficient per FLOP than dense GeMM due to irregular memory access patterns; the DGE correction kernel involves division, power, and absolute value operations that may not achieve peak throughput on GPU tensor cores; and the quantization/dequantization operations (absmax scaling factor computation, lookup table application) add latency that doesn't appear in the FLOP count. Without actual runtime measurements — which the paper cannot provide without native FP4 hardware — the speedup claims should be treated as upper bounds.

Summary assessment: The experiments provide strong and consistent evidence that the proposed FP4 training framework closes the gap between naive 4-bit quantization (which fails) and BF16/FP8 training (which work) for LLaMA 2 models up to 13B parameters and 100B tokens. The ablation studies cleanly isolate the contributions of DGE and OCC. The main limitations are the absence of multi-architecture validation, the theoretical (not measured) speedup estimates, the single-seed nature of all experiments, and the gap between the tested scale (13B/100B tokens) and frontier training regimes where this technology would be most impactful. These limitations do not undermine the paper's core contribution — establishing feasibility — but they define the scope of what "feasible" currently means and point to necessary follow-up work before FP4 training can be considered production-ready.

6. Limitations and Trade-offs

The Difficult, Unaccounted Cost of FP4 Training: Simulation Overhead Obscures True Practicality

The constraint: The paper's central empirical claim — that FP4 training matches BF16 accuracy — relies entirely on FP4 simulation using FP8 tensor cores on NVIDIA H100 GPUs. Native FP4 tensor cores do not exist in available hardware, making direct speedup measurements impossible. The paper is explicit about this in Section 6:

"One primary limitation of this work lies in the absence of dedicated FP4 Tensor Cores in existing hardware. Consequently, we are unable to directly measure the potential speedup and energy efficiency gains achievable with native FP4 support. All current experiments rely on FP4 simulations, which introduce additional computational overhead due to extra precision casting and significantly prolong runtime."

The consequence: The paper's value proposition — training LLMs faster and cheaper — is entirely theoretical. The 2.95× speedup estimate for a 7B model (Appendix B) is a FLOP-counting exercise built on assumptions about how efficiently future hardware will execute FP4 GeMM, sparse compensation, and the DGE correction kernel. Real hardware will introduce effects that FLOP counting cannot capture: memory bandwidth bottlenecks (FP4 tensors are smaller but must be loaded, unpacked, and repacked), quantization/dequantization latency (scaling factor computation, lookup table application), and sparse matrix multiplication inefficiency (irregular memory access patterns in the OCC compensation path dramatically reduce effective throughput per non-zero element compared to dense GeMM).

This means a practitioner reading this paper cannot answer the most important practical question: will FP4 training actually be faster than BF16, and by how much? The 2.95× number could turn out to be 1.5× on real hardware (if memory and dequantization overhead dominate) or 3.5× (if the hardware is perfectly optimized). The paper's core contribution — "FP4 training is feasible" — is validated only for accuracy, not for speed. Since speed is the entire motivation (Section 1: "low-bit arithmetic kernels can save memory and accelerate computations"), this is a significant gap between the paper's claims and what it actually demonstrates.

What evidence exists in the paper: Appendix B provides the only quantitative analysis of speedup, and it is purely theoretical. The FLOP breakdown in Table 5 assumes a perfect 4× reduction in GeMM FLOPs from FP4 (6bsh² vs. 24bsh² for FP32), with the DGE and OCC overheads added as percentages of the theoretical ideal. No runtime measurements — even for the simulated FP4 on H100 GPUs — are reported. The paper does not compare wall-clock training time between BF16 and simulated FP4, which would at least establish whether the simulation overhead makes FP4 training slower (as expected) and provide a baseline for estimating native FP4 speed. Section 6 acknowledges this gap but provides no partial evidence.

Mitigation status: The paper does not attempt to mitigate this limitation beyond acknowledging it and calling for future hardware. This is a genuine chicken-and-egg problem: native FP4 hardware doesn't exist because there was no demonstrated software need, and software speedup can't be demonstrated without hardware. The paper positions itself as breaking this cycle by establishing accuracy feasibility first ("sets a foundation for efficient ultra-low precision training"), which is a reasonable framing. However, for practitioners evaluating whether to invest in FP4 training infrastructure, the lack of any speedup evidence — even simulation-based runtime comparisons — means the decision must be made on faith in theoretical FLOP accounting.

Single Architecture, Single Dataset: The Generalization of FP4 Training Dynamics Is Unknown

The constraint: All experiments use the LLaMA 2 architecture trained on the DCLM dataset exclusively. The paper provides no evidence about whether the proposed techniques — DGE with k=5, OCC with α=0.99, token-wise activation quantization, channel-wise weight quantization — transfer to other architectures or data distributions. Section 4.1 states:

"We adopt the widely recognized LLaMA 2 model (Touvron et al., 2023) as the primary model architecture."

The paper frames LLaMA 2 as "representative," but this is asserted, not demonstrated.

The consequence: The activation outlier patterns that OCC is designed to handle — channel-specific extreme values visible in Figure 14's heatmap — are known in the quantization literature to be architecture-dependent. LLaMA 2 uses pre-layer normalization and SwiGLU activations. Models with post-layer norm, different activation functions (GELU, ReLU), different normalization placements (pre-norm vs. post-norm vs. hybrid), or different attention mechanisms (multi-query, grouped-query) may exhibit qualitatively different outlier structures. If outliers in some architecture are not channel-specific (spread across tokens rather than concentrated in channels), token-wise quantile clamping could be ineffective or counterproductive. If outliers are more frequent (requiring a lower α), the sparse compensation overhead could grow beyond the 5.6% budgeted in Appendix B.

Similarly, the DGE correction factor $f'(x) = (1/k) \cdot |2x/\delta - 1|^{1/k - 1}$ was tuned with k=5 specifically for LLaMA 2 weight distributions. Weight distributions vary across architectures — models with different initialization schemes, weight decay values, or normalization strategies might have weight distributions that interact differently with the DGE correction. The paper's own evidence (Figures 8–10) shows that weight distributions vary across layers even within LLaMA 2 (standard deviations from 0.006 to 0.028), suggesting architecture-level variation could be substantial.

The DCLM dataset, while comprehensive, represents a specific data distribution. Training on code-heavy datasets (e.g., The Stack), multilingual data, or domain-specific corpora might produce activation distributions with different outlier characteristics that challenge the OCC parameters tuned on DCLM. The paper provides no evidence about this.

What evidence exists in the paper: None for architecture generalization. The paper does not test non-LLaMA architectures, alternative activation functions, or different normalization schemes. The distributional analysis in Appendix D (Figures 8–14) is entirely from a single LLaMA 1.3B training run and makes no comparative claims about other architectures. The paper does not vary the dataset — all experiments use DCLM.

Mitigation status: The paper does not claim generalization and does not attempt to address this limitation. The focus on establishing feasibility for one representative architecture is a reasonable scope for a first-of-its-kind demonstration, but readers should not infer that FP4 training "works" in general without architecture-specific validation. This is a standard limitation of systems papers that establish feasibility rather than universality, but it is particularly salient here because both DGE and OCC rely on distributional assumptions (weight distribution shape for DGE, activation outlier patterns for OCC) that are known to vary across architectures.

No Validation Loss Reporting: Overfitting Risk Under FP4 Training Is Uncharacterized

The constraint: All loss curves in the paper (Figures 1, 5, 6) report training loss — the loss computed on the data being used for parameter updates. The paper does not report validation loss (loss on a held-out set not used for training) anywhere. Downstream evaluations (Tables 2 and 3) partially address generalization, but validation loss during training is the standard diagnostic for detecting overfitting, and its absence is a gap.

The consequence: Quantization introduces noise into both the forward pass (through activation quantization error) and the backward pass (through gradient approximation error). This noise can act as a form of implicit regularization, potentially causing the model to underfit the training data (higher training loss) while still achieving comparable or better generalization (as suggested by Tables 2 and 3). However, the converse is also possible: if FP4 quantization systematically biases the optimization trajectory toward sharper minima that happen to fit the training data well but generalize poorly, this could produce reasonable training loss curves while validation loss degrades. Without validation loss curves, we cannot distinguish these scenarios.

The downstream evaluations in Tables 2 and 3 provide a partial check — if FP4 models were overfitting, we would expect worse zero-shot performance despite similar training loss. The results show FP4 models are competitive or slightly better, which argues against severe overfitting. However, downstream task accuracy at the end of training (a single point) is not equivalent to tracking validation loss throughout training. A model could overfit early (validation loss rising while training loss continues dropping) and then partially recover through the regularization effects of continued training on diverse data. Validation loss curves would reveal these dynamics; single-point evaluations do not.

Additionally, the lack of validation loss means there is no check on whether the small but consistent training loss gap (0.06–0.10 across model scales in Figure 5) reflects worse optimization (model hasn't converged as well) or a different optimum (model has converged to a different solution with different generalization properties). These have different implications for whether FP4 training can match BF16 given more training tokens or different hyperparameters.

What evidence exists in the paper: Only indirect. Tables 2 and 3 show comparable downstream performance at the end of training, which is consistent with (but does not prove) comparable generalization. The paper does not report validation loss at any point and does not discuss the training/validation loss relationship.

Mitigation status: Not addressed. The paper focuses exclusively on training loss and final downstream metrics, following a common practice in large-scale training papers where held-out validation is sometimes omitted due to dataset constraints or computational cost. However, for a paper whose central claim is that FP4 training matches BF16, validation loss curves would provide a more rigorous comparison than training loss alone. This is a methodological gap that weakens the strength of the "comparable accuracy" claim, though the downstream evaluations partially compensate.

Scale Ceiling at 13B Parameters and 100B Tokens: Frontier Training Regimes Are Unvalidated

The constraint: The largest model trained in this paper is 13B parameters, trained on 100B tokens. While substantial, this is orders of magnitude below frontier LLM training, which routinely involves models of 70B–405B+ parameters trained on trillions of tokens. Section 6 acknowledges this explicitly:

"Additionally, due to constraints on computational resources, we have not yet extended our experiments to extremely large-scale models or to datasets comprising trillions of tokens. Investigating such scalability remain as critical directions for future research."

The consequence: Several FP4-specific failure modes could emerge at larger scales that are invisible at 13B parameters:

Activation outlier severity may scale with model depth. The paper's Appendix D shows that activation distributions become wider and more heavy-tailed in deeper layers (the standard deviation of post-attention layer norm outputs grows from ~1.0 in early layers to ~1.3 in middle layers, Figure 12). If this trend continues with model depth — and there is evidence from the broader LLM literature that outliers become more extreme in larger models — then the OCC clamping threshold α=0.99 may need to be lowered (clamping more values) to prevent information collapse. This would increase the sparse compensation matrix density and erode the theoretical 2.95× speedup. At some scale, the compensation overhead might become large enough that FP4 training is no longer faster than FP8.

Gradient estimation errors may compound in deeper networks. The DGE correction factor is applied per-layer, independently at each linear layer. In very deep networks (70B+ parameter models can have 80+ layers), small per-layer gradient estimation biases could compound through the chain rule, producing systematic errors in early-layer weight updates. The paper's evidence that the FP4-BF16 loss gap does not grow with model scale (0.06 at 1.3B, 0.10 at 7B, 0.09 at 13B — no monotonic trend) is weakly reassuring, but three data points spanning only a 10× parameter range cannot confidently predict behavior at 100×+ scale.

Distributed training interactions are untested. The paper uses FP8 gradient communication (following Peng et al., 2023) and presumably some form of data parallelism or model parallelism for the larger models, but the specific distributed training configuration is not described. At frontier scales, training is distributed across thousands of GPUs, with complex interactions between quantization, gradient synchronization, and optimizer state sharding. Quantization errors that are negligible in single-GPU or small-cluster training could interact with distributed communication patterns (e.g., gradient all-reduce with FP8 compression) in unexpected ways.

Training duration effects. The 100B-token training runs represent a fixed compute budget. The paper shows that FP4 and BF16 loss curves remain parallel through 100B tokens, but does not establish what happens at 1T+ tokens. If FP4 introduces a small constant bias in the gradient direction (rather than zero-mean noise), the optimization trajectory could systematically diverge from BF16 over very long training runs. The parallel curves through 100B tokens argue against a large bias, but a small bias integrated over 10× more steps could produce a meaningful gap.

What evidence exists in the paper: Only the three-scale comparison in Figure 5 (1.3B, 7B, 13B). The paper does not train any model beyond 13B parameters or 100B tokens. The claim that "the FP4 curve [exhibits] a slightly higher training loss" is empirically supported at these scales, but the paper does not extrapolate or predict behavior at larger scales.

Mitigation status: The paper explicitly acknowledges this as a limitation and frames it as future work. This is a reasonable scope for an initial feasibility demonstration — establishing that the method works at moderate scale before investing in frontier-scale validation. However, for practitioners at organizations training 70B+ models, the paper provides no direct evidence that FP4 training would succeed at their scale. The jump from "works at 13B" to "will work at 405B" requires extrapolation that the paper does not attempt to justify.

The OCC Accuracy-Efficiency Trade-off Is Characterized Only Theoretically, Not Empirically

The constraint: The OCC method introduces a hyperparameter α (the clamping quantile) that directly controls the trade-off between quantization accuracy and computational overhead. The paper selects α = 0.99 based on a theoretical FLOP analysis (Appendix B) and qualitative judgment ("a better choice for comprehensive computational performance considerations"), but provides no runtime measurements for different α values, even on the simulated FP4 hardware.

The consequence: The sparse matrix multiplication in the OCC compensation path (ΔY · W) is inherently less efficient per FLOP than dense GeMM, because sparse operations on GPUs suffer from irregular memory access patterns, load imbalance across warps, and reduced tensor core utilization. The paper's theoretical overhead calculation (5.6% additional FLOPs at α=0.99) assumes sparse operations achieve the same FLOP efficiency as dense operations, which is incorrect in practice. Sparse matrix multiplication on modern GPUs typically achieves 10–30% of dense throughput for unstructured sparsity at moderate densities.

This means the actual runtime overhead of OCC could be 2–5× higher than the theoretical 5.6% FLOP overhead — potentially 10–30% of total training time rather than the paper's estimated 5.6%. If the true overhead at α=0.99 is 25%, the adjusted theoretical speedup drops from 2.95× to approximately 2.4× (rough calculation: 3.12 ideal speedup, now with 25% overhead instead of 5.6%). If α needs to be lowered to 0.97 for larger models (as the activation outlier severity discussion above suggests), the overhead could be 50%+, eliminating most of the advantage over FP8. These numbers are speculative — the point is that the paper provides no empirical basis for choosing α beyond training loss curves (Figure 6c) that ignore runtime.

The paper also provides no evidence that α=0.99 is the right choice at larger scales or for different architectures. The sweep in Figure 6c covers only α ∈ {0.97, 0.99, 0.999} on a single model size (1.3B) with 10B tokens. This is a minimal sweep that doesn't explore the region between 0.99 and 0.999 (which might offer a better accuracy-efficiency trade-off) and doesn't validate the choice at larger scales.

What evidence exists in the paper: The theoretical FLOP analysis in Appendix B (Table 5, speedup formula). The α sweep in Figure 6c showing training loss for three α values, with the paper noting that "a smaller α leads to better model accuracy, which is consistent with the conclusion of Table 1." No runtime measurements for different α values, even on the H100 simulation hardware where such measurements would be straightforward to obtain.

Mitigation status: Not addressed. The paper selects α=0.99 as a design choice and reports sensitivity analysis only for accuracy, not runtime. The distinction between theoretical FLOP overhead and actual runtime overhead is not discussed. For a systems paper whose primary value proposition is training speedup, this is a significant omission — the technique that makes FP4 training accurate (OCC) might also be the technique that makes it slow, and the paper provides no empirical evidence to bound this concern.

The Gradient Correction Method Is Validated Only Through Final Loss, Not Through Gradient Quality Metrics

The constraint: The DGE method claims to provide "more accurate gradient computation" (Section 3.1) by replacing the STE's identity approximation with a differentiable correction factor. However, the paper provides no direct evidence that DGE improves gradient accuracy. All evidence for DGE's effectiveness is indirect: training loss curves with and without DGE (Figure 6b), showing that DGE reduces the final loss gap.

The consequence: The training loss improvement from DGE could arise for reasons other than the claimed mechanism (better gradient estimation). Alternative explanations include:

• Implicit regularization: The DGE correction factor $f'(x)$ is not a true gradient of any function (since it's derived from a differentiable approximation applied only in the backward pass, with hard quantization in the forward pass). The resulting weight updates may not correspond to the gradient of any well-defined objective. This "gradient misalignment" can act as a regularizer that happens to improve convergence on this particular training setup, without actually providing "more accurate" gradients in any meaningful sense.

• Adaptive learning rate effect: The DGE correction factor amplifies gradients for weights near quantization boundaries and attenuates them for weights near interval centers. This is a form of per-weight learning rate scaling that might help convergence independently of gradient accuracy — similar to how Adam's adaptive learning rates help, but applied based on quantization interval position rather than gradient history.

• Noise injection: The k parameter controls the sharpness of the approximation. At k=5, the derivative near interval midpoints is high but bounded (clipped at 3.0). This effectively injects structured noise into the weight gradients that might help escape poor local minima, acting as implicit regularization rather than improving gradient fidelity.

Without direct gradient quality metrics — such as comparing DGE-computed weight updates against finite-difference approximations of the true loss gradient, or measuring the correlation between DGE gradients and the gradients that would be computed in full precision — the paper cannot distinguish between "DGE provides better gradients" and "DGE provides different gradients that happen to work better for this training setup."

This matters for the paper's broader claims. If DGE works because of implicit regularization rather than gradient accuracy, its effectiveness may not generalize to different models, datasets, or training configurations. A regularizer that helps on LLaMA 2 with DCLM and a specific learning rate schedule might not help — or might hurt — on a different setup. If DGE genuinely provides more accurate gradients, generalization is more likely.

What evidence exists in the paper: Only training loss curves (Figure 6b) and the final loss comparison in the full FP4 framework (Figure 5). The paper provides mathematical motivation (Equations 6–8, Figure 3) for why the DGE correction should improve gradient accuracy, but no empirical gradient analysis — no gradient norm trajectories, no cosine similarity between DGE and STE gradients, no comparison against oracle gradients from higher-precision training.

Mitigation status: Not addressed. The paper treats the connection between DGE's mathematical formulation and its empirical benefit as sufficient validation. For a technique whose claimed mechanism is "more accurate gradient computation," the absence of gradient-level evaluation is a methodological gap. This is partially understandable — computing oracle gradients in large-scale training is expensive — but smaller-scale diagnostic experiments (e.g., on a single training step or a small model) could have provided mechanistic evidence without prohibitive cost.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally changes the conversation around low-bit training by establishing that 4-bit floating-point pretraining is not merely a hardware aspiration but a demonstrated software reality. Before this work, the quantization community operated under an implicit assumption: FP8 was the floor for training precision, and 4-bit formats were the domain of inference (via PTQ/QAT) or fine-tuning at best. The paper breaks that assumption with systematic evidence at non-trivial scale.

This is a feasibility demonstration that redefines the frontier, not a paradigm shift in how quantization works. The individual techniques — differentiable gradient estimation and outlier clamping — are not conceptually revolutionary in isolation. What has changed is the field's understanding of where the floor actually is. The paper shows that the transition from FP8 to FP4 is not a smooth degradation treatable with existing methods (the direct-cast FP4 curve in Figure 6a makes this unambiguous — it produces a ~1–2 point loss gap), but that two specific, targeted interventions are sufficient to close the gap. This transforms FP4 training from "probably impossible" to "solved in principle, pending hardware."

The work also resolves a latent tension in the quantization literature between accuracy and hardware compatibility. Prior differentiable quantization methods (Gong et al., 2019; Uhlich et al., 2019; Chen et al., 2019) addressed the gradient problem by replacing the quantization function with a smooth surrogate in both forward and backward passes — a mathematically clean solution that destroys the computational speedup because the forward pass no longer uses low-bit operations. The DGE approach demonstrates that you can have it both ways: hard quantization in the forward pass (preserving hardware efficiency) with a principled gradient correction in the backward pass (preserving accuracy). This decoupling pattern — what the hardware computes vs. what the optimizer sees — is generalizable beyond quantization to other non-differentiable operations in training (sparsification, discretization, routing), and the paper provides a concrete template for how to do it.

The most consequential shift may be in how hardware roadmaps and software research interact. The paper explicitly positions itself as a call for next-generation hardware ("sets a foundation for efficient ultra-low precision training" and "may also serve as a call for next-generation hardware designs"). By demonstrating software feasibility before hardware availability, the paper reverses the typical dependency: hardware vendors usually build capability first, and software catches up. Here, the software validation de-risks the hardware investment. The specific numbers matter: the theoretical 2.95× speedup for 7B models (Appendix B) gives hardware architects a quantified target. The finding that activation quantization is the harder problem (Figure 6c: W8A4 diverges; Figure 6d: coarse activation quantization degrades more than coarse weight quantization) tells hardware designers where to focus silicon optimizations — better sparse matrix multiplication for outlier compensation, not just denser FP4 tensor cores.

The paper also makes certain research directions less attractive by establishing clear failure conditions. The evidence that naive FP4 quantization fails (Figure 6a: large loss gap; Figure 6c: NaN divergence) and that FP8 works with existing methods (Figure 6a: MS-AMP FP8 and TE FP8 both track BF16) suggests that efforts to extend FP8 methods downward to FP4 without addressing the two specific failure modes (gradient estimation quality and activation outlier collapse) are unlikely to succeed. The field now has a validated diagnosis of why FP4 is harder than FP8, and incremental extensions of FP8 techniques without addressing these root causes are a dead end.

Similarly, the paper's finding that vector-wise quantization granularity is necessary for FP4 (Figure 6d: coarse/coarse is substantially worse than fine/fine), while FP8 achieves sufficient accuracy with tensor-wise scaling, establishes that granularity requirements tighten dramatically with precision reduction. Research on automatic granularity selection or learned per-group scaling factors becomes more relevant for FP4 and below, while unified tensor-wise approaches become less tenable.

Finally, the paper's validation-as-contribution framing — the idea that demonstrating feasibility at scale (13B parameters, 100B tokens) is itself a scientific contribution even when the underlying techniques are not novel — provides a template for how systems research can interact with hardware roadmaps. The field has many proposed quantization methods that work on small models or synthetic benchmarks but lack evidence at scale. This paper raises the bar for what constitutes meaningful validation: training curves across multiple model sizes, zero-shot downstream evaluations, and ablations that isolate mechanism. Future FP4 training papers will be expected to match or exceed this validation standard.

Follow-Up Research This Work Enables

Direct measurement of FP4 training speedup on native hardware. The paper's central gap is the absence of speedup measurements — all accuracy validation is on FP8-simulated FP4, and all speedup estimates are theoretical FLOP calculations. The most urgent follow-up is to port the framework to NVIDIA Blackwell GPUs (or whatever hardware first ships with native FP4 tensor cores) and measure actual wall-clock training time per step compared to BF16 and FP8. The key metrics would be: (a) end-to-end training throughput in tokens/second for 7B and 13B models, (b) breakdown of time spent in FP4 GeMM vs. sparse compensation vs. DGE correction vs. quantization/dequantization, and (c) how the measured speedup compares to the paper's theoretical 2.95× estimate. A negative result — measured speedup of 1.5× rather than 2.95× — would not invalidate the paper's accuracy claims but would significantly change the practical calculus for adoption. This experiment requires no algorithmic innovation, only hardware access and careful benchmarking.

Scaling FP4 training to 70B+ parameters and 1T+ tokens. The paper's largest experiment is 13B parameters on 100B tokens. Frontier LLM training operates at 70B–405B parameters on trillions of tokens. Critical open questions: Does the FP4-BF16 loss gap remain constant (~0.06–0.10) at these scales, or does it grow with model depth? Section 5 noted that activation outlier severity appears to increase with depth in the 1.3B model (Figures 11–13), but we don't know if this trend continues or saturates. Does the OCC clamping threshold α=0.99 need to be lowered for larger models, increasing compensation overhead? Does the DGE correction factor's bounded instability (k=5, gradient clipped at 3.0) cause training instability in 80+ layer networks that is invisible at 13B? A strong experiment would train a 70B LLaMA 2 model from scratch with FP4 and BF16 on 500B+ tokens, report both training and validation loss curves, and measure whether the per-step training speed (if native hardware is available) remains within the 2–3× range predicted by the paper's analysis. A negative result — divergence at scale, or a growing loss gap — would establish the practical ceiling for FP4 training.

Cross-architecture validation of the OCC clamping threshold. The paper's OCC method with α=0.99 was tuned exclusively on LLaMA 2's activation distributions. Different architectures — particularly those with different normalization schemes (post-layer norm, DeepNorm), activation functions (GELU vs. SwiGLU vs. ReLU), or attention mechanisms (multi-query, grouped-query, sliding window) — are known to produce different outlier patterns. A systematic study would: (a) characterize activation distributions across 5–10 popular architectures (LLaMA, Mistral, Falcon, MPT, Gemma, etc.) at comparable scales, (b) for each architecture, sweep α ∈ {0.95, 0.97, 0.99, 0.995, 0.999} and measure both training loss and the resulting sparse compensation density, and (c) determine whether a single α works across architectures or architecture-specific tuning is necessary. If α must be lowered substantially (to 0.95 or below) for some architectures, the compensation overhead could erode the FP4 speedup advantage, making some architectures better candidates for FP4 training than others. This experiment would also test the paper's implicit assumption that LLaMA 2 is "representative" — if other architectures show qualitatively different quantization behavior, the paper's feasibility claim would need to be qualified as architecture-specific.

Gradient fidelity experiments for the DGE mechanism. The paper claims DGE provides "more accurate gradient computation" but provides no direct gradient-level evidence — only training loss curves that could arise from implicit regularization rather than improved gradient accuracy. A diagnostic experiment would: (a) on a small model (e.g., 125M parameters) trained for a few thousand steps, compute "oracle" weight gradients by running the forward and backward passes in FP32 (or BF16 with no quantization), (b) compare the cosine similarity between oracle gradients and the gradients produced by three estimators: STE (baseline), DGE with k=5 (paper's method), and DGE with k=3 and k=10 (sensitivity), (c) track how gradient similarity evolves during training (early vs. late steps), and (d) correlate gradient similarity with downstream training loss and validation performance. A finding that DGE gradients have higher cosine similarity to oracle gradients than STE gradients would validate the claimed mechanism. A finding that DGE gradients are different but not more similar to oracle gradients would suggest the method works through regularization rather than accuracy, which has different implications for generalization across architectures and training configurations.

Combining DGE with learned quantization parameters. The paper's DGE uses a fixed quantization function (the E2M1 lookup table) and a fixed differentiable approximation. Prior differentiable quantization work (Gong et al., 2019; Uhlich et al., 2019) explored learning the quantization parameters themselves (step boundaries, scaling factors) through backpropagation, but these methods modified the forward pass and were incompatible with hardware acceleration. The DGE framework — hard quantization in forward, differentiable surrogate in backward — creates an opportunity to learn quantization parameters without modifying the forward pass: the forward pass still uses the hardware-efficient lookup table, but the backward pass uses the differentiable surrogate to compute gradients with respect to quantization parameters (step positions, scaling factors). This could enable per-layer or per-channel learned quantization grids that adapt to the data distribution while preserving hardware compatibility. A proof-of-concept experiment on a 1.3B model would: (a) parameterize the quantization step boundaries as learnable parameters, (b) update them using the DGE surrogate gradient, (c) measure whether learned grids outperform the fixed E2M1 grid, and (d) characterize the computational overhead of the additional gradient computation.

Stress-testing FP4 training under distribution shift. The paper trains and evaluates on DCLM data, which represents a single (though broad) distribution. Real-world LLM training often involves multiple phases — pretraining on web text, continued training on code or math, fine-tuning on instruction data — with significant distribution shifts between phases. A stress test would: (a) pretrain a model in FP4 on DCLM for 100B tokens, (b) continue training on a code-heavy corpus (e.g., The Stack) for 50B tokens, (c) measure whether the activation outlier patterns change during the distribution shift (requiring OCC parameter adjustment) and whether the FP4-BF16 loss gap widens during the transition, and (d) evaluate downstream code generation performance to check for any FP4-specific degradation on the new domain. This experiment addresses the practical question of whether FP4 training is compatible with the multi-phase training pipelines used in production, or whether distribution shifts trigger instability that requires precision-specific intervention.

Practical Applications and Downstream Use Cases

Cost reduction for large-scale pretraining at organizations training models in the 7B–70B range. The most immediate application targets teams that pretrain LLaMA-scale models on budgets of hundreds of billions of tokens. With the paper's theoretical 2.95× speedup for 7B models, a training run that currently takes 30 days on 256 GPUs could theoretically complete in ~10 days on the same hardware, or in 30 days on ~87 GPUs — a 3× reduction in GPU-hours. Even if the actual speedup on native hardware turns out to be 2× (conservative relative to the paper's estimate), the cost savings for a single 70B training run could be in the hundreds of thousands of dollars. The key enabler is that the paper's framework uses the same hyperparameters as BF16 training (Section 4.1), meaning adoption requires minimal tuning — teams can switch precision without redesigning their learning rate schedules, optimizer settings, or data pipelines. The primary adoption barrier is hardware availability (native FP4 tensor cores), not software complexity.

Enabling on-device or edge-cluster model training by reducing memory and compute requirements. While the paper focuses on large-scale pretraining, the techniques apply to any training scenario where GeMM dominates compute. For fine-tuning scenarios — adapting a pretrained model to a specific domain or task — the 4× reduction in weight and activation precision during matrix multiplications means smaller working memory and lower power consumption. A team fine-tuning a 7B model on domain-specific data could potentially run the training on a single workstation GPU rather than requiring a multi-GPU server, because FP4 GeMM operations require 4× less memory bandwidth and compute than FP16. The OCC overhead (5.6% in theory) is a small price to pay for this reduction. This is speculative without native hardware measurements, but the paper's accuracy validation at 1.3B–13B scales provides confidence that the trained model quality would not suffer.

Accelerating iterative research cycles where model quality is validated frequently during training. In research settings — architecture search, hyperparameter optimization, data mixture experiments — researchers train many models to partial convergence (e.g., 10B–50B tokens) to compare configurations, rather than training a single model to completion. The 2–3× theoretical speedup from FP4 directly reduces the wall-clock time for each experimental trial, enabling more experiments per GPU-day and faster iteration. The paper's evidence that FP4 loss curves track BF16 from early in training (Figure 5 shows the gap is established early and remains constant) means that partial-training comparisons are meaningful proxies for final model quality. A research team running 50 experiments at 10B tokens each could theoretically complete their sweep in 17 days instead of 50 days, dramatically accelerating the research cycle. This application does not require the FP4 model to be the final deployed model — it only requires that FP4 training produces faithful rankings of architectural or hyperparameter choices, which the paper's consistent loss gap across scales suggests is plausible.

When to Prefer FP4 Training Over Higher-Precision Alternatives

The paper does not explicitly position FP4 training against a clear set of named alternatives with decision criteria. It frames FP4 as a direct replacement for BF16/FP8 in GeMM operations — same model, same hyperparameters, same training recipe, lower precision. The implicit trade-off is accuracy vs. speed: FP4 offers a theoretical 2.95× speedup (for 7B models) at the cost of a small training loss gap (0.06–0.10 across the tested model sizes) and comparable downstream performance. The paper does not define a threshold beyond which the loss gap becomes unacceptable, or a scenario where FP8 would be preferred over FP4 despite the speed difference.

The only explicit preference articulated is in the granularity discussion (Section 4.3): "tensor-wise scaling in FP4 introduces significant errors," implying that if hardware or implementation constraints force coarse-grained quantization, FP8 with tensor-wise scaling may be preferable to FP4 with tensor-wise scaling. This is a hardware-compatibility consideration rather than a principled accuracy-speed trade-off. The paper provides no guidance on when a practitioner should choose FP4 with DGE+OCC over, say, FP8 with standard STE and tensor-wise scaling — the loss curves in Figure 6a suggest both are very close to BF16, and the speedup difference (FP4's 2.95× vs. FP8's ~2×) is unmeasured.

Since the paper does not articulate a structured trade-off between its method and specific alternatives, a decision matrix would be fabricated rather than derived from the paper's evidence.